Apache Arrow Reviews and details

Social recommendations and mentions

• How moving from Pandas to Polars made me write better code without writing better code

  In comes Polars: a brand new dataframe library, or how the author Ritchie Vink describes it... a query engine with a dataframe frontend. Polars is built on top of the Arrow memory format and is written in Rust, which is a modern performant and memory-safe systems programming language similar to C/C++. - Source: dev.to / about 2 months ago

• Time Series Analysis with Polars

  One is related to the heritage of being built around the NumPy library, which is great for processing numerical data, but becomes an issue as soon as the data is anything else. Pandas 2.0 has started to bring in Arrow, but it's not yet the standard (you have to opt-in and according to the developers it's going to stay that way for the foreseeable future). Also, pandas's Arrow-based features are not yet entirely on... - Source: dev.to / 5 months ago

• TXR Lisp

  IMO a good first step would be to use the txr FFI to write a library for Apache arrow: https://arrow.apache.org/. - Source: Hacker News / 5 months ago

• A Polars exploration into Kedro

  Polars is an open-source library for Python, Rust, and NodeJS that provides in-memory dataframes, out-of-core processing capabilities, and more. It is based on the Rust implementation of the Apache Arrow columnar data format (you can read more about Arrow on my earlier blog post “Demystifying Apache Arrow”), and it is optimised to be blazing fast. - Source: dev.to / 12 months ago

• Demystifying Apache Arrow

  Apache Arrow (Arrow for short) is an open source project that defines itself as "a language-independent columnar memory format" (more on that later). It is part of the Apache Software Foundation, and as such is governed by a community of several stakeholders. It has implementations in several languages (C++ and also Rust, Julia, Go, and even JavaScript) and bindings for Python, R and others that wrap the C++... - Source: dev.to / 12 months ago

• GPU vendor-agnostic fluid dynamics solver in Julia

  Are you talking about Apache Arrow? Interesting! Don't think I've seen this one. https://arrow.apache.org/. - Source: Hacker News / 12 months ago

• Making Python 100x faster with less than 100 lines of Rust

  Apache Arrow (https://arrow.apache.org/) is built exactly around this idea: it's a library for managing the in-memory representation of large datasets. - Source: Hacker News / about 1 year ago

• Show HN: Up to 100x Faster FastAPI with simdjson and io_uring on Linux 5.19

  If anything you'd probably want to send it in Arrow[1] format. CSV's don't even preserve data types. [1]: https://arrow.apache.org/. - Source: Hacker News / about 1 year ago

• IPC communication between rust, c++, and python

  In that case, why not use polars, which supports apache arrow format which supports C, C++, Rust, Python and supports zero-copy read. Source: over 1 year ago

• Introducing ArrowJS • Reactivity without the framework

  I think the naming will likely cause some confusion with apache arrow. My initial thoughts when reading "Introducing ArrowJS" was a new port of the apache arrow spec. Source: over 1 year ago

• Java Serialization with Protocol Buffers

  The information can be stored in a database or as files, serialized in a standard format and with a schema agreed with your Data Engineering team. Depending on your information and requirements, it can be as simple as CSV, XML or JSON, or Big Data formats such as Parquet, Avro, ORC, Arrow, or message serialization formats like Protocol Buffers, FlatBuffers, MessagePack, Thrift, or Cap'n Proto. - Source: dev.to / over 1 year ago

• GlueSQL: A SQL database engine written as a library in Rust

  Just another embedded SQL engine. There are SQLite(OLTP), DuckDB(OLAP) and some engine-based project like mentioned Apache Arrow(https://arrow.apache.org/)(OLAP): Apache Arrow has many language implementations, some do not include the query engine(for example, Rust implementation, which depends on the DataFusion for more SQL-like analytics) in its own repo, but other do include(for example, C++). There is a... - Source: Hacker News / over 1 year ago

• New Pandas-for-Haskell data frame library: Name suggestions

  This is a meta-request for the library, but imo it would be really awesome if it used a data structure compatible with Arrow: https://arrow.apache.org/. Source: over 1 year ago

• How to Deploy ML Models Using Gravity AI and Meadowrun

  As a bit of an aside, you could imagine a way to get the best of both worlds with an extension to Docker that would allow you to publish a container that exposes a Python API, so that someone could call sentiment = call_container_api(image="huggingface/transformers", "my input text") directly from their python code. This would effectively be a remote procedure call into a container that is not running as a service... - Source: dev.to / over 1 year ago

• Scala needs a good, dependency-free DataFrame library

  I assume you mean to use Apache arrow rather than scala Arrow? Source: over 1 year ago

• Dragonflydb – A modern replacement for Redis and Memcached

  I've used Apache Arrow before[1]; in-memory columnar storage. We did some AI/ML stuff with data gathered from social network APIs, but you can probably do a ton of things. [1] https://arrow.apache.org/. - Source: Hacker News / almost 2 years ago

• How to use Spark and Pandas to prepare big data

  Pandas user-defined function (UDF) is built on top of Apache Arrow. Pandas UDF improves data performance by allowing developers to scale their workloads and leverage Panda’s APIs in Apache Spark. Pandas UDF works with Pandas APIs inside the function, and works with Apache Arrow to exchange data. - Source: dev.to / over 2 years ago

• Spice.ai v0.6-alpha is now available!

  Building upon the Apache Arrow support in v0.6-alpha, Spice.ai now includes new Apache Arrow data processor and Apache Arrow Flight data connector components! Together, these create a high-performance bulk-data transport directly into the Spice.ai ML engine. Coupled with big data systems from the Apache Arrow ecosystem like Hive, Drill, Spark, Snowflake, and BigQuery, it's now easier than ever to combine big data... Source: about 2 years ago

• Arrowdantic 0.1.0 released

  Arrowdantic is a small Python library backed by a Mature Rust implementation of Apache Arrow that can interoperate with * Parquet * Apache Arrow and * ODBC (databases). Source: about 2 years ago

• Introducing Spice.xyz - Data and AI infrastructure for web3

  🔥 Some cool things for eth/finance. We have per-block pool reserve data for Uniswap and Sushiswap and a Python SDK which lets you get data into Pandas, NumPy in 4 lines of code so you can use all the Python ecosystem of finance libraries you are used to. It uses Apache Arrow as the transport, so much faster than JSON. Here's an example Kaggle notebook: https://www.kaggle.com/code/spiceluke/spice-xyz-ethereum-blocks. Source: about 2 years ago

• What are the differences between feather and parquet?

  Both are columnar (disk-)storage formats for use in data analysis systems. Both are integrated within Apache Arrow (pyarrow package for python) and aredesigned to correspond with Arrow as a columnar in-memory analytics layer. Source: about 2 years ago