Ask HN: What is the correct way to deal with pipelines?

Workflow Automation Workflows Automation

Huginn Landing Page
1

Huginn

Build agents that monitor and act on your behalf. Your agents are standing by!
Pricing:
- Open Source
"correct" is a value judgement that depends on lots of different things. Only you can decide which tool is correct. Here are some ideas: - https://camel.apache.org/ - https://www.windmill.dev/ Your idea about a queue (in redis, or postgres, or sqlite, etc) is also totally valid. These off-the-shelf tools I listed probably wouldn't give you a huge advantage IMO.

#Automation #Web Service Automation #API Tools 65 social mentions
Apache Camel Landing Page
2

Apache Camel

Apache Camel is a versatile open-source integration framework based on known enterprise integration patterns.
Pricing:
- Open Source
"correct" is a value judgement that depends on lots of different things. Only you can decide which tool is correct. Here are some ideas: - https://camel.apache.org/ - https://www.windmill.dev/ Your idea about a queue (in redis, or postgres, or sqlite, etc) is also totally valid. These off-the-shelf tools I listed probably wouldn't give you a huge advantage IMO.

#Data Integration #ETL #Open Source 12 social mentions
Luigi Landing Page

3

Luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs.

I agree there are many options in this space. Two others to consider: - https://airflow.apache.org/ - <a href="https://github.com/spotify/luigi">https://github.com/spotify/luigi</a> There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file showing up in a directory….

#DevOps Tools #Workflow Automation #Workflows 9 social mentions
Apache Airflow Landing Page
4

Apache Airflow

Airflow is a platform to programmaticaly author, schedule and monitor data pipelines.
Pricing:
- Open Source
I agree there are many options in this space. Two others to consider: - https://airflow.apache.org/ - <a href="https://github.com/spotify/luigi">https://github.com/spotify/luigi</a> There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file showing up in a directory….

#Workflows #Workflow Automation #Data Pipelines 65 social mentions