Delta Lake Reviews and Details

Is Delta Lake good?

Social recommendations and mentions

• From Postgres to Iceberg

  A common solution is using open table formats like Apache Iceberg(others are Delta lake and Apache Hudi). With these tools you get the benefits of traditional database functionality on your data lake i.e ACID guarantees, transactions. The Iceberg specification defines an open table format that enables accessing related data stored in separate files in a distributed storage system, as one table. - Source: dev.to / 8 months ago

• Twitter's 600-Tweet Daily Limit Crisis: Soaring GCP Costs and the Open Source Fix Elon Musk Ignored

  Delta Lake: Delta Lake is an open-source storage layer that provides ACID transactions, scalable metadata management, and data versioning on top of existing data lakes. It aims to bring reliability and performance optimizations to big data workloads while ensuring data integrity and consistency. - Source: dev.to / about 1 year ago

• Stream Processing Systems in 2025: RisingWave, Flink, Spark Streaming, and What's Ahead

  When it comes to stream processing systems, Iceberg support varies across vendors. Databricks, which oversees Spark Streaming, focuses on Delta Lake. Apache Flink, heavily influenced by Alibaba’s contributions, promotes Paimon, an alternative to Iceberg. RisingWave, on the other hand, fully embraces Iceberg. Rather than focusing solely on one table format, RisingWave aims to support various catalog services,... - Source: dev.to / over 1 year ago

• 25 Open Source AI Tools to Cut Your Development Time in Half

  Delta Lake is a storage layer framework that provides reliability to data lakes. It addresses the challenges of managing large-scale data in lakehouse architectures, where data is stored in an open format and used for various purposes, like machine learning (ML). Data engineers can build real-time pipelines or ML applications using Delta Lake because it supports both batch and streaming data processing. It also... - Source: dev.to / almost 2 years ago

• Make Rust Object Oriented with the dual-trait pattern

  There is a neat example, of how a third party project belonging to the Linux Foundation, is implementing UserDefinedLogicalNodeCore: MetricObserver in delta-rs. The developer had to use only #[derive(Debug, Hash, Eq, PartialEq)] to get dyn_eq and dyn_hash implemented. - Source: dev.to / almost 2 years ago

• Delta Lake vs. Parquet: A Comparison

  Delta is pretty great, let's you do upserts into tables in DataBricks much easier than without it. I think the website is here: https://delta.io. - Source: Hacker News / over 2 years ago

• Getting Started with Flink SQL, Apache Iceberg and DynamoDB Catalog

  Apache Iceberg is one of the three types of lakehouse, the other two are Apache Hudi and Delta Lake. - Source: dev.to / over 2 years ago

• [D] Is there other better data format for LLM to generate structured data?

  The Apache Spark / Databricks community prefers Apache parquet or Linux Fundation's delta.io over json. Source: over 2 years ago

• Databricks Strikes $1.3B Deal for Generative AI Startup MosaicML

  Databricks provides Jupyter lab like notebooks for analysis and ETL pipelines using spark through pyspark, sparkql or scala. I think R is supported as well but it doesn't interop as well with their newer features as well as python and SQL do. It interfaces with cloud storage backend like S3 and offers some improvements to the parquet format of data querying that allows for updating, ordering and merged through... - Source: Hacker News / almost 3 years ago

• The "Big Three's" Data Storage Offerings

  Structured, Semi-structured and Unstructured can be stored in one single format, a lakehouse storage format like Delta, Iceberg or Hudi (assuming those don't require low-latency SLAs like subsecond). Source: almost 3 years ago

• Medallion/lakehouse architecture data modelling

  Take a look at Delta Lake https://delta.io, it enables a lot of database-like actions on files. Source: about 3 years ago

• How to build a data pipeline using Delta Lake

  This sounds like a new trending destination to take selfies in front of, but it’s even better than that. Delta Lake is an “open-source storage layer designed to run on top of an existing data lake and improve its reliability, security, and performance.” (source). It let’s you interact with an object storage system like you would with a database. - Source: dev.to / about 3 years ago

• Delta.io/deltalake self hosting

  You are right, delta.io is just a framework. Sorry for the unclear question. Another try: when you host spark on your own with delta as table format compared to usage of Databricks, what are the differences? Source: about 3 years ago

• Delta.io/deltalake self hosting

  I mean the different between using the delta.io framework to let it run on your own machines/ vms vs using databricks and have clusters defined. Source: about 3 years ago

• Delta.io/deltalake self hosting

  Is there actually any company implementing delta.io self hosted beside microsoft/synapse and databricks? Would it be worth the effort compared to the features microsoft/databricks bring to the table? Source: about 3 years ago

• Lightweight HTTP API for Big Data on S3

  We are happy to announce our third opensource project - Delta Fetch. Delta Fetch is a configurable HTTP API service for accessing Delta Lake tables. Service is highly configurable, with possibility to filter your Delta tables by selected columns. - Source: dev.to / about 3 years ago

• Question about lambda architecture

  I’d suggest looking at the open table formats. Delta lake does an excellent job at providing batch and streaming APIs for Spark. This would unify your workloads. It would follow the medallion architecture which is a bit more popular lately. Aspects of the lamda architecture can still be present in the medallion model, especially when real-time requirements are present. Source: over 3 years ago

• HDFS/Spark + Delta: Is this warning dangerous?

  I've installed the stack (Hadoop, Hive, Spark) into a Centos VM, built everything from sources to make sure it fits together. Then added Delta Lake (delta.io) from their maven repo. Source: over 3 years ago

• Confusion with Hadoop/Hive/Spark

  (I configured delta.io in $SPARK_HOME/conf/spark-defaults.xml so it's loaded & available). Source: over 3 years ago

• 5 Reasons Your Data Lakehouse should Embrace Dremio Cloud

  You can query data organized in many open table formats like Apache Iceberg and Delta Lake. (Here is a good article on what is a table format and the differences between different ones). - Source: dev.to / almost 4 years ago

• How do we bridge SQL and Python.

  Bit more specific: https://delta.io/. Source: almost 4 years ago