Apache Airflow Reviews and details

Social recommendations and mentions

• The 2024 Web Hosting Report

  For the third, examples here might be analytics plugins in specialized databases like Clickhouse, data-transformations in places like your ETL pipeline using Airflow or Fivetran, or special integrations in your authentication workflow with Auth0 hooks and rules. - Source: dev.to / 3 months ago

• Best ETL Tools And Why To Choose

  Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. The platform features a web-based user interface and a command-line interface for managing and triggering workflows. Source: 6 months ago

• Simplifying Data Transformation in Redshift: An Approach with DBT and Airflow

  Airflow is the most widely used and well-known tool for orchestrating data workflows. It allows for efficient pipeline construction, scheduling, and monitoring. - Source: dev.to / 6 months ago

• Share Your favorite python related software!

  AIRFLOW This is more of a library in my opinion, but Airflow has become an essential tool for scheduling in my work. All our ML training pipelines are ordered and scheduled with Airflow and it works seamlessly. The dashboard provided is also fantastic! Source: 7 months ago

• Ask HN: What is the correct way to deal with pipelines?

  I agree there are many options in this space. Two others to consider: - https://airflow.apache.org/ - https://github.com/spotify/luigi There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file... - Source: Hacker News / 8 months ago

• PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows

  Folks who have used Python-based orchestration tools such as Apache Airflow, Luigi and Mage will be familiar with the concepts and the API if PyJaws. Source: 12 months ago

• My experience with optimizing machine learning workflow

  There are a range of solutions available to help make this process easier. Some of these options include automation tools such as Apache Airflow and Azure Data Factory, specialized libraries that focus on fine-tuning deep learning models like FinetunerPlus, or machine learning platforms that provide end-to-end solutions like Amazon SageMaker and Google Cloud ML Engine. Source: about 1 year ago

• Python task scheduler with a web UI

  Looks interesting as a light-weight alternative to https://www.prefect.io/ (which itself is a lighter-weight / more modern alternative to https://airflow.apache.org/ ). Source: about 1 year ago

• .NET Modern Task Scheduler

  A few years ago, I opened a GitHub issue with Microsoft telling them that I think the .NET ecosystem needs its own equivalent of Apache Airflow or Prefect. Fast forward 'til now, and I still don't think we have anything close to these frameworks. Source: about 1 year ago

• How do you backup running systems?

  If you have the spare capacity Apache Airflow is great for this. Source: about 1 year ago

• How do you manage scheduled tasks?

  Its a bit overkill but I use Airflow with local executor. Source: over 1 year ago

• Twitter Data Pipeline with Apache Airflow + MinIO (S3 compatible Object Storage)

  To learn more about it, I built a Data Pipeline that uses Apache Airflow to pull Elon Musk tweets using the Twitter API and store the result in a CSV stored in a MinIO (OSS alternative to AWS s3) Object Storage bucket. - Source: dev.to / over 1 year ago

• What's the best and easiest to use GUI-based CI tool? No Jenkins suggestions, please.

  Airflow, that's it https://airflow.apache.org/. Source: over 1 year ago

• First data lake pipeline advice - multitenancy

  If you are fixed on developing a solution in house, you may have options that don't require many additional tools. Aurora Postgres already supports exporting data to S3. So use an orchestration tool like AWS ECS Scheduled Tasks, Airflow, Prefect, etc to run a script (probably Python). That script can ask for all the distinct tenant ids "SELECT distinct tenant_id FROM...". Then iterate through them and run a query... Source: over 1 year ago

• ETL tool

  Airflow is really popular, started at Airbnb. Pros: huge community, super mature. Cons: generic workflow orchestration, not the best for handling only data, hard to scale and maintain. Source: over 1 year ago

• is there a kind of software for describing and searching data about batch jobs (their structure, relationship between them, constraints, date/time details etc)?

  Seems like you want documentation. Look into plantuml or something like that? For reference tools like airflow provide this stuff too. Source: over 1 year ago

• How to do distributed cronjobs with worker queues?

  Airflow might also be a good option for you. Essentially DAGs of cronjobs. We like it a lot. Source: over 1 year ago

• Airflow :: Deploy Apache Airflow on Rancher K3s

  $ helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace Release "airflow" does not exist. Installing it now. NAME: airflow LAST DEPLOYED: Sun Nov 6 02:06:55 2022 NAMESPACE: airflow STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Thank you for installing Apache Airflow 2.4.1! Your release is named airflow. You can now access your dashboard(s) by executing the following... - Source: dev.to / over 1 year ago

• Need help finding container automation solution (logic flows)

  Https://airflow.apache.org/ might be worth looking into. Source: over 1 year ago

• Duct Size vs. Airflow (2012)

  I gotta admit, my first thought was "Duct Size" is a weird name for a distributed work-flow tool[1]. [1] https://airflow.apache.org/. - Source: Hacker News / over 1 year ago

• Built and automated a complete end-to-end ELT pipeline using AWS, Airflow, dbt, Terraform, Metabase and more as a beginner project!

  Infrastructure provisioning through Terraform, containerized through Docker and orchestrated through Airflow. Created dashboard through Metabase. Source: over 1 year ago