Evolutionary Data Infrastructure

Workflow Automation Stream Processing Big Data

Kafka Streams Landing Page

1

Kafka Streams

Apache Kafka: A Distributed Streaming Platform.

Therefore, I still recommend using a streaming framework such as Apache Flink or Apache Kafka Streams.

#Stream Processing #Analytics #Workflow Automation 14 social mentions
Apache Flink Landing Page
2

Apache Flink

Flink is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations.
Pricing:
- Open Source
Therefore, I still recommend using a streaming framework such as Apache Flink or Apache Kafka Streams.

#Stream Processing #Big Data #Developer Tools 27 social mentions
Google BigQuery Landing Page
3

Google BigQuery

A fully managed data warehouse for large-scale data analytics.
Pricing:
- Open Source
In addition, batch tasks require knowledge of the data schema of each service in order to get the data correctly and save it to the corresponding warehouse table. Assuming our data warehouse is GCP BigQuery, the schema in the warehouse table also needs to be created and modified manually.

#Data Management #Data Warehousing #Data Dashboard 35 social mentions
Apache Airflow Landing Page
4

Apache Airflow

Airflow is a platform to programmaticaly author, schedule and monitor data pipelines.
Pricing:
- Open Source
For batch processing, I recommend using Apache Airflow, which is easy to manage and easy to script for various DAGs to address the needs of multiple batch processing scenarios.

#Workflows #Workflow Automation #Data Pipelines 65 social mentions