Delta Lake VS Apache Airflow

Delta Lake Reviews

Apache Airflow Reviews

Delta Lake mentions (31)

• Delta Lake vs. Parquet: A Comparison

  Delta is pretty great, let's you do upserts into tables in DataBricks much easier than without it. I think the website is here: https://delta.io. - Source: Hacker News / 3 months ago

• Getting Started with Flink SQL, Apache Iceberg and DynamoDB Catalog

  Apache Iceberg is one of the three types of lakehouse, the other two are Apache Hudi and Delta Lake. - Source: dev.to / 4 months ago

• [D] Is there other better data format for LLM to generate structured data?

  The Apache Spark / Databricks community prefers Apache parquet or Linux Fundation's delta.io over json. Source: 5 months ago

• Databricks Strikes $1.3B Deal for Generative AI Startup MosaicML

  Databricks provides Jupyter lab like notebooks for analysis and ETL pipelines using spark through pyspark, sparkql or scala. I think R is supported as well but it doesn't interop as well with their newer features as well as python and SQL do. It interfaces with cloud storage backend like S3 and offers some improvements to the parquet format of data querying that allows for updating, ordering and merged through... - Source: Hacker News / 10 months ago

• The "Big Three's" Data Storage Offerings

  Structured, Semi-structured and Unstructured can be stored in one single format, a lakehouse storage format like Delta, Iceberg or Hudi (assuming those don't require low-latency SLAs like subsecond). Source: 11 months ago

Apache Airflow mentions (65)

• The 2024 Web Hosting Report

  For the third, examples here might be analytics plugins in specialized databases like Clickhouse, data-transformations in places like your ETL pipeline using Airflow or Fivetran, or special integrations in your authentication workflow with Auth0 hooks and rules. - Source: dev.to / 2 months ago

• Best ETL Tools And Why To Choose

  Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. The platform features a web-based user interface and a command-line interface for managing and triggering workflows. Source: 6 months ago

• Simplifying Data Transformation in Redshift: An Approach with DBT and Airflow

  Airflow is the most widely used and well-known tool for orchestrating data workflows. It allows for efficient pipeline construction, scheduling, and monitoring. - Source: dev.to / 6 months ago

• Share Your favorite python related software!

  AIRFLOW This is more of a library in my opinion, but Airflow has become an essential tool for scheduling in my work. All our ML training pipelines are ordered and scheduled with Airflow and it works seamlessly. The dashboard provided is also fantastic! Source: 7 months ago

• Ask HN: What is the correct way to deal with pipelines?

  I agree there are many options in this space. Two others to consider: - https://airflow.apache.org/ - https://github.com/spotify/luigi There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file... - Source: Hacker News / 7 months ago