What is a Stream Processing Engine?
Stream processing engines are runtime engines and libraries that let programmers develop relatively simple code to process stream-encoded data. These libraries and frameworks allow developers to read and compute data without needing to bother with lower-level stream physics.

Most of these stream libraries don’t have any built-in algorithms for more advanced processing, but Apache Spark (for example) has MLLib that makes it possible to implement machine learning algorithms.

Direct Acyclic Graph (DAG)

This is an essential part of processing engines, and processes in the engines run through DAG cycles. This idea makes it possible to chain multiple functions together in a given sequence, with the effect that the procedure never returns back to some previous state. Check out the graph below for an overview of DAG at work.

The two processing engines that are created by the use of DAG – a declarative engine and a compositional engine – are different in that the user must pass functions through them (i.e., this is what the declarative engine does for the user), and manipulate the flow of data (which the compositional engine must explicitly accomplish by defining and pushing the data through the DAG). Apache Spark and Flink are declarative, functional code; Apache Samza is compositional, low-level code for designing a DAG.

Since you’ve gained a basic understanding of what exactly stream processing engines are doing, let’s take a closer look at two of the most used processing engines in the world — Flink and Kafka. Let’s start with a discussion of the fundamental differences between the two.

Flink vs. Kafka – The Big Differences
If we compare Flink with the Kafka Streams API, the former is what you might say a cluster model based data processing framework, while the Kafka Streams API runs as an embeddable library, no clusters are required.

The two most common streaming providers are Apache Flink and Kafka. Flink vs Kafka reminds me of the famous Sci-Fi vs Fantasy debate. It’s dependent on what we’re looking for, of course, but there are two distinct methods so it’s difficult. Let’s dive into these frameworks and get a clear picture of Flink vs Kafka.

Flink vs Kafka Streams API: What is the difference?
The table below shows the top differences between Kafka and Flink:

Apache Flink Kafka Streams API
Deployment Flink is a clustering framework and deploys your application to a Flink cluster, or on YARN, Mesos, or containers (Docker/Kubernetes). The Streams API can be embedded into any ordinary Java application and, therefore, has no requirements for deployment: you can deploy apps using almost any deployment mechanism – containers (Docker, Kubernetes), resource managers (Mesos, YARN), deployment orchestration (Puppet, Chef, Ansible), or any other company-provided tools.
Life cycle User’s stream processing code is deployed and executed as a job on the Flink cluster User’s stream processing code executes inside their application.
Typically managed by Data infrastructure/BI team Line of business team responsible for the specific application.
Coordination Flink Master (JobManager),
an element of the streaming show Uses the Kafka cluster for coordination, load-balancing, and fault-tolerance.
Source for streaming data Kafka, Files, other message queues Strictly Kafka Connect API in Kafka to solve the data into, data out of Kafka problem.
Sink to outcomes Kafka, other MQs, file system, analytic database, key/value stores, stream processor state and any other external systems Kafka, application state, operational database or any other external system.
Bounded and unbounded data flows Unbounded and Bounded Unbounded.
Semantical guarantees At least, exactly, once for "everything internally significant in Flink state"; end-to-end exactly once when the chosen sources and sinks (Kafka to Flink to HDFS) are selected; at least once, "when Kafka is used as sink here end-to-end exactly-once guarantee for Kafka will probably be in the near future. Exactly-once end-to-end with Kafka
Flink
The purest streaming engine in this class is Berlin TU University's Flink, which uses lambda too and streams batch as a variant of streaming (when the data is constrained).

Apache Flink was designed from the ground up as a distributed, high-throughput, stateful, batch/real-time, streaming dataflow engine and framework. It’s a little bit slick, moving your stream processing code from state to state, and parallelising it so you can leverage a cluster. When we needed such a processing engine to accommodate all the real-time data flows and use cases (and there were hundreds of them) at Soul Machines, Apache had one.

Flink was the first open source system that could achieve such high throughput (we are using Flink for our example of tens of millions of events per second), sub-second millisecond latency (down to 10s of milliseconds) and fault-tolerant output. Streams run standalone on top of an iterative cluster architecture – that can then be distributed via resource managers or simply standalone. The streams can feed data into streams, even databases, with associated APIs and libraries. These enable Flink to be a batch-processing system, as well as an operational system that can function well, even in production. Flink is often used with Apache Kafka serving as the storage, but teams may opt to operate Flink on their own to have the full benefits of both tools.

And by doing this automatically, without any particular understanding of how to use this tuning, Flink makes it convenient to not have to make a mistake with the setting. It was the first true streaming framework. With its event time processing and watermarking capabilities, Flink offers Uber and Alibaba the latest in stream processing technology.

Kafka
Kafka Streams API is a lightweight stream processor library and stream engine for implementing ordinary Java applications built on the functional programming model with full fault tolerance. it can be integrated into reactive stateful apps or event driven systems as part of microservices. It is specifically intended to be embedded and implemented in a java script. it is built over the top of its parent tech apache Kafka and inherits the distributed nature and use of Kafka fault tolerance capability. Instead of having to stack several servers together as part of a cluster to run an application over a messaging framework such as Kafka, Kafka Streams API is embeddable. This means you don’t need to create clusters to add it to your current toolstack. Your developers can remain in their apps without worrying about how their code gets deployed. And all the Kafka failover, scalability and security features are available to teams instantly for free.

Industry Example

You can read more about Kafka vs Flink (or technically Flink vs Kafka Streams) and it’ll tell you that Kafka Streams allows you to embed it anywhere – even in an application of any size and complexity – but it isn’t intended for lifting heavy loads.

The second big disadvantage at a rigorous theoretical level is that Kafka doesn’t need to be run on its own cluster, so once deployed, deployment is quite simple. For Apache Flink vs Kafka: The case studies reveal that heavy workloads will shift enterprises like the ride-sharing firm Uber towards Flink, while enterprises like Bouygues Telecom will adopt Kafka for real-time analytics and event processing.

For example, in Flink vs Kafka Streams, Kafka Streams is the leader for interactive queries (after the feature was removed from Flink due to low demand), but Flink also provides an application mode for quick microservices development, though most people still prefer Kafka Streams.

Kafka Streams and Flink: What’s the Difference?
In choosing a candidate stream processing system, evaluate it across multiple factors – deployment, usability, architecture, performance (thruput, latency, etc.).

Architecture and Deployment
Apache Kafka implements a publish/subscribe message broker architecture with component chains. The API Kafka Streams (which also uses a broker-based distributed computing platform), the broker persists data in a very small embedded database. The Kafka Streams API’s library can be deployed into an existing application or you can push Kafka over clusters as an autonomous deployment, using containers, resource managers, or automated deployment (e.g. Puppet).

At the Heart of Flink: distributed computing architectureAs a single job, the application runs either as a stand-alone cluster or on top of YARN, Mesos or any container service like Docker or Kubernetes. Coordination requires a master node, which makes Flink quite convoluted.

Complexity and Accessibility
Adaptability depends on the user. Kafka Streams and Flink are owned by the developers and the data analysts who use them, therefore, they are somewhat complex.

Unlike R, Kafka Streams is relatively straightforward to start and to maintain in the long term as far as a Java or Scala programmer is concerned. It’s relatively straightforward to deploy common Java and Scala code with Kafka Streams, and Kafka Streams is largely available as a plugin. You don’t need to instrument any cluster manager in order to get started. So Kafka Streams are actually pretty simple. Even if you’re not a developer this is quite the learning curve.

Your team has maybe only a few stream processing features. Big deal, right? But what about the future? If you’re about to begin using more events, more aggregation and more streams, Flink simply provides a high-level streaming framework. After you’ve bought and implemented a technology, you don’t usually get a second chance. It’s also expensive.

https://upstaff.com/blog/engineering/a-quick-guide-to-stream-processing-engines-flink-vs-kafka/