Apache Parquet Reviews and Details

Is Apache Parquet good?

Social recommendations and mentions

• Can you build observability ingestion on S3 alone — no Kafka, no disks, no coordination layer?

  Apache Iceberg fits these requirements well. Iceberg stores data as immutable Apache Parquet files and adds them through atomic commits, so readers always see a consistent snapshot. A separate metadata layer prunes files by their statistics before the data itself is ever read, and those statistics can be extended to match an observability filtering profile. - Source: dev.to / 20 days ago

• Zeroserve: A zero-config web server you can script with eBPF

  Depends on the domain. There's a bunch of sciences using large datasets served up efficiently using static file formats, e.g., https://zarr.dev/ and https://parquet.apache.org/. - Source: Hacker News / about 1 month ago

• What Are Table Formats and Why Were They Needed?

  The data files themselves are still standard Parquet or ORC. The table format adds a metadata layer on top that gives those files the properties of a database table. - Source: dev.to / 3 months ago

• So, you know what? I just wasted 3 months of my life

  The dataset is huge - in parquet conversion - it is total 9gb. And in raw PNG image nested folders - it is 67 gigabytes. Huge... - Source: dev.to / 4 months ago

• Fix Slow Query: A Developer's Guide to Data Warehouse Performance

  The solution is to standardize on columnar formats like Apache Parquet. Parquet stores data in columns, not rows, which immediately enables column pruning. If a query is SELECT avg(price) FROM sales, the engine reads only the price column and ignores all others. This can reduce storage footprints by up to 75% compared to raw formats and is a cornerstone of modern analytics performance. - Source: dev.to / 8 months ago

• From Postgres to Iceberg

  Iceberg is able to efficiently manage large amounts of data stored in the data lake. The data layer supports storing data in open formats like Apache parquet or Avro. Apache Parquet is an open columnar data format for efficient data storage and retrieval. With this, you automatically get the benefits of column storage for your analytical workloads. Engines like Apache Spark, Apache Flink, Presto, Trino etc can be... - Source: dev.to / 9 months ago

• 🔥 Simulating Course Schedules 600x Faster with Web Workers in CourseCast

  If there was a way to package and compress the Excel spreadsheet in a web-friendly format, then there's nothing stopping us from loading the entire dataset in the browser!1 Sure enough, the Parquet file format was specifically designed for efficient portability. - Source: dev.to / 11 months ago

• How to Pitch Your Boss to Adopt Apache Iceberg?

  Iceberg decouples storage from compute. That means your data isn’t trapped inside one proprietary system. Instead, it lives in open file formats (like Apache Parquet) and is managed by an open, vendor-neutral metadata layer (Apache Iceberg). - Source: dev.to / over 1 year ago

• Processing data with “Data Prep Kit” (part 2)

  Data prep kit github repository: https://github.com/data-prep-kit/data-prep-kit?tab=readme-ov-file Quick start guide: https://github.com/data-prep-kit/data-prep-kit/blob/dev/doc/quick-start/contribute-your-own-transform.md Provided samples and examples: https://github.com/data-prep-kit/data-prep-kit/tree/dev/examples Parquet: https://parquet.apache.org/. - Source: dev.to / over 1 year ago

• 🔬Public docker images Trivy scans as duckdb datas on Kaggle

  Deliver nice ready-to-use data as duckdb, parquet and csv. - Source: dev.to / over 1 year ago

• Introducing Promptwright: Synthetic Dataset Generation with Local LLMs

  Push the dataset to hugging face in parquet format. - Source: dev.to / over 1 year ago

• Shades of Open Source - Understanding The Many Meanings of "Open"

  It's this kind of certainty that underscores the vital role of the Apache Software Foundation (ASF). Many first encounter Apache through its pioneering project, the open-source web server framework that remains ubiquitous in web operations today. The ASF was initially created to hold the intellectual property and assets of the Apache project, and it has since evolved into a cornerstone for open-source projects... - Source: dev.to / about 2 years ago

• [D] Is there other better data format for LLM to generate structured data?

  The Apache Spark / Databricks community prefers Apache parquet or Linux Fundation's delta.io over json. Source: over 2 years ago

• Demystifying Apache Arrow

  Apache Parquet (Parquet for short), which nowadays is an industry standard to store columnar data on disk. It compress the data with high efficiency and provides fast read and write speeds. As written in the Arrow documentation, "Arrow is an ideal in-memory transport layer for data that is being read or written with Parquet files". - Source: dev.to / about 3 years ago

• Parquet: more than just "Turbo CSV"

  Googling that suggests this page: https://parquet.apache.org/. Source: over 3 years ago

• Beginner question about transformation

  You should also consider distribution of data because in a company that has machine learning workflows, the same data may need to go through different workflows using different technologies and stored in something other than a data warehouse, e.g. Feature engineering in Spark and loaded/stored in binary format such as Parquet in a data lake/object store. Source: over 3 years ago

• Pandas Free Online Tutorial In Python — Learn Pandas Basics In 5 Lessons!

  This section will teach you how to read and write data to and from a variety of file types, including CSV, Excel, SQL, HTML, Parquet, JSON etc. You’ll also learn how to manipulate data from other sources, such as databases and web sites. Source: over 3 years ago

• What pandas can do and polars can’t?

  Loading or writing Parquet files is lightning fast. Pandas uses PyArrow -Python bindings exposed by Arrow- to load Parquet files into memory, but it has to copy that data into Pandas memory. With Polars there is no extra cost due to copying as we read Parquet directly into Arrow memory and keep it there. Source: over 3 years ago

• Help me figure out ETL and storage for user search and click logs. So lost in all the DB alternatives.

  Write a python script to convert the JSON data you receive to Parquet. Put the parquet data in the S3 bucket. Parquet is an open source data serialization format. Almost all the major tooling (including pandas) supports it so even if you move solutions (to Snowflake, etc) your data will already be setup. It also compresses the data and makes it faster to do aggregate queries like counts and sums. Source: over 3 years ago

• Transition to cloud. What Warehouse to choose?

  Why are you trying to work with CSVs anyways? Compared to other options you have in Azure, it's more effective to pull data into your Azure environment and save it as a compressed parquet. You'd be saving quite a bit of space most likely. Working with CSVs isn't really something people do on a large-scale unless the source format is CSVs - there are way more efficient means of storing and working with data files. Source: over 3 years ago

• What is the separation of storage and compute in data platforms and why does it matter?

  Apache Parquet is a file format that is used to store data in a columnar format. This allows for faster reads and reduced disk space. Not only is the Parquet format open source, but it also has an entire ecosystem of tools to help you create, read and transform data. - Source: dev.to / over 3 years ago