Stream processing engines are tools that consume a series of data and take action on it in real-time or pseudo-real-time. Possible actions include aggregations (sum), analytics (predicting future actions), transformations (converting to a date format), or enrichment of data (combining data to enhance context). Stream processing has become more popular with increases in data collection and the need to decrease application response time, which is achieved with synchronous stream processing. Technologies for implementing stream processing can simplify data architectures, react appropriately to time-sensitive data, and provide real-time insights and analytics.
Many tools are available for implementing stream processing, each providing stream processing with a unique feature set. These include open source tools such as Apache Spark and Kafka as well as proprietary public cloud services like AWS Kinesis and Google Dataflow.
Types of Stream Processing Engines
Stream processing engines are divided into three major types. When choosing a stream processing engine for your software architecture, consider which type will work best for your offering to filter down your options. The difference between these technologies mainly relates to the directed acyclic graph (DAG) control, which defines the processing functions applied to data as it flows through the stream.
Compositional Engines
Software developers must define the DAG and how processing will proceed in advance with compositional engines. Planning must be done in advance to ensure efficient and effective processing. Processing is controlled entirely by the developer.
Declarative Engines
In declarative engines, the developers implement functions in the stream engine, but the engine itself determines the correct DAG and will direct streamed data accordingly. There may be options for developers to define the DAG in some tools, but the stream engine will also optimize the DAG as needed.
Fully Managed Self-Service Engines
These engines are based on declarative engines, but they further reduce the work required of developers. The DAG is determined by the stream engine, as in declarative engines, but a complete end-to-end solution is provided, including data storage and streaming analytics.
Streaming Tools
There are many current and retired stream engines available for use. The tools chosen for this list are all being actively enhanced and monitored for bugs. Each is popular for its own feature set, making each useful for specific use cases. A variety of self-managed and managed options are also provided.
Open Source
Apache is an open-source software foundation producing various projects for community use. All Apache projects listed are available for use without paying licensing fees when running as self-managed software. Some tools are also available as managed software through different cloud providers. While all Apache projects are open-source, other open-source streaming tools are available.
1. Apache Spark
Apache Spark is an open-source distributed streaming engine designed for large-scale data processing. Spark can distribute data processing tasks across multiple nodes, making it a good choice for big data and machine learning. Spark is also flexible in handling both batch and streaming data using a variety of language options such as Java, Python, R, and more. Spark supports SQL querying, machine learning, and graph processing.
Spark can run on a standalone cluster or a cluster using container orchestration tools like Kubernetes or Docker Swarm. The latter option offers automated deployments, resilience, and security. Managed versions are also available through Amazon, Google, and Microsoft.
Spark is a popular engine for scalable computing and is utilized by thousands of companies in various fields. It has a highly active community contributing to the codebase, with contributions from thousands of individuals.
2. Apache Storm
Apache Storm is an open-source compositional streaming engine designed for large-scale data processing. Storm uses parallel task computation to operate on one record at a time, making it a real-time streaming technology. Storm can be used for online machine learning, continuous computation, and more. However, it doesn't store data locally, so it's not used for analyzing historical data.
Developers can create Apache Storm clusters natively, on virtual machines, or using container technologies like Kubernetes and Docker. There are no options for managed versions of Storm.
Storm is a popular engine for real-time data integrations since it can handle high throughputs of data that require complex processing. When less complex processing is required, other stream processing engines can provide simpler integrations.
3. Apache Flink
Apache Flink is an open-source, distributed streaming engine that supports both batch and stream processing. It excels at processing both unbounded streams (which have a start but no end) and bounded streams (with a defined start and end). It's designed to run stateful streaming applications with low latency and at large scales.
Flink supports running as a standalone cluster or can run on all common cluster environments. It's a distributed system, so it does require compute resources to execute applications.
Flink, with its fault-tolerant design, is used for critical event-driven applications like fraud detection, rule-based alerting, and anomaly detection. It's also used for data analytics applications like quality monitoring and data pipeline applications that feed real-time search indices.
4. Apache Flume
Apache Flume is an open-source distributed streaming engine designed to handle log data. Flume can accept data from a variety of external sources, including web servers, Twitter, and Facebook. It's mainly used to aggregate and load data into the Apache Hadoop Distributed File System (HDFS).
Flume is installed and configured as part of the Hadoop repository. It's generally run on the Hadoop cluster using a cluster management tool such as Hadoop YARN.
Since Apache Flume is designed for log aggregations, it's commonly used to collect and move large data volumes within Hadoop. Flume is also used by e-commerce companies to analyze buying behavior and for sentiment analysis.
5. Apache Kafka Streams
Apache Kafka Streams is an open-source distributed streaming engine and storage system. It allows high throughput while delivering especially low latencies. It's able to scale up and down both storage and processing as needed, making it a suitable option for large and small requirements.
Kafka Streams can be deployed to containers, run on virtual machines, and also run on bare metal. Cloud providers like AWS also provide managed Kafka services, including support for Kafka Streams.
Kafka Streams is popular across many different industries. It can be used to process payments and financial transactions, track shipments and other geospatial data, capture and analyze sensor data, or drive event-driven software architectures.
6. RisingWave
RisingWave is an open-source cloud-native streaming database used to support real-time applications. It's a stateful system that can perform complex incremental computations such as aggregation, joins, and windows. It uses the PostgreSQL syntax to store the results at each step to decrease the latency of future calculations. It accepts data from various sources such as Kafka and Kinesis and supports outputting data to Kafka streams.
RisingWave was built to provide a real-time streaming database that's easy to use. It can support real-time applications without significant operational overhead for software engineering and maintenance.
Managed Stream Processing
Public cloud providers allow users to run stream processing tools on their servers. They include different guarantees on the availability of the service, freeing up developers to work on other tasks besides maintaining the stability of the stream. Some open-source tools are available as managed services in the public cloud.
7. Google Cloud Dataflow
Google Cloud Dataflow is a fully managed, cloud-based data processing and streaming service. It's capable of handling stream processing as well as batch processing, and it allows for scheduling batch processing executions. Dataflow also provides auto-scaling to handle spikes in streaming data sizes.
Dataflow comes with extra processing features built in, such as real-time AI patterns that allow the system to react to events within the data flow, including anomaly detection.
8. Amazon Kinesis Data Streams
Amazon Kinesis Data Streams is a fully managed streaming service provided by Amazon's cloud platform. Data is captured from one or more consumers, and the data can then trigger processing (or consumer) functions with batched stream data.
Kinesis Data Streams is designed to handle high data throughput, handling gigabytes of data per second. The cost of the stream is based on what is used and can be either provisioned for use cases with predictable throughput or set to on-demand when the throughput should scale with the data.
This is popular for big data processing in use cases like event data logging, running real-time analytics (using consumer functions), and powering event-driven applications.
9. Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose is a fully managed streaming service on AWS. This service is designed to collect, transform, and load data streams to permanent storage and analytics services.
Like Kinesis Data Streams, Firehose can handle high data throughput. When data storage is required, Firehose is an effective tool since it formats and stores directly rather than requiring a consumer to provide that functionality.
Firehose is popular for storing stream data in data lakes or data warehouses. It can also provide a large data set for machine learning applications and security analytics.
CONCLUSION
Stream processing tools continuously obtain real-time data from external sources and perform operations. This processing supports event-driven actions, analysis of behavior, and machine learning. Available engines may be self-managed or managed by a cloud provider, which eases the load on in-house development teams. When choosing a stream processing tool, determine which is best for your use case by considering design aspects like throughput, latency, deployment options, and the complexity of building it.
About RisingWave Labs
RisingWave is an open-source distributed SQL database for stream processing. It is designed to reduce the complexity and cost of building real-time applications. RisingWave offers users a PostgreSQL-like experience specifically tailored for distributed stream processing.
Official Website: https://www.risingwave.com/
GitHub：https://github.com/risingwavelabs/risingwave
LinkedIn：linkedin.com/company/risingwave-labs