Apache Arrow Reviews and details

Social recommendations and mentions

• Unlocking DuckDB from Anywhere - A Guide to Remote Access with Apache Arrow and Flight RPC (gRPC)

  Apache Arrow : It contains a set of technologies that enable big data systems to process and move data fast. - Source: dev.to / 5 months ago

• Using Polars in Rust for high-performance data analysis

  One of the main selling points of Polars over similar solutions such as Pandas is performance. Polars is written in highly optimized Rust and uses the Apache Arrow container format. - Source: dev.to / 6 months ago

• Kotlin DataFrame ❤️ Arrow

  Kotlin DataFrame v0.14 comes with improvements for reading Apache Arrow format, especially loading a DataFrame from any ArrowReader. This improvement can be used to easily load results from analytical databases (such as DuckDB, ClickHouse) directly into Kotlin DataFrame. - Source: dev.to / about 1 year ago

• Shades of Open Source - Understanding The Many Meanings of "Open"

  It's this kind of certainty that underscores the vital role of the Apache Software Foundation (ASF). Many first encounter Apache through its pioneering project, the open-source web server framework that remains ubiquitous in web operations today. The ASF was initially created to hold the intellectual property and assets of the Apache project, and it has since evolved into a cornerstone for open-source projects... - Source: dev.to / 11 months ago

• Arrow Flight SQL in Apache Doris for 10X faster data transfer

  Apache Doris 2.1 has a data transmission channel built on Arrow Flight SQL. (Apache Arrow is a software development platform designed for high data movement efficiency across systems and languages, and the Arrow format aims for high-performance, lossless data exchange.) It allows high-speed, large-scale data reading from Doris via SQL in various mainstream programming languages. For target clients that also... - Source: dev.to / 12 months ago

• How moving from Pandas to Polars made me write better code without writing better code

  In comes Polars: a brand new dataframe library, or how the author Ritchie Vink describes it... a query engine with a dataframe frontend. Polars is built on top of the Arrow memory format and is written in Rust, which is a modern performant and memory-safe systems programming language similar to C/C++. - Source: dev.to / about 1 year ago

• Time Series Analysis with Polars

  One is related to the heritage of being built around the NumPy library, which is great for processing numerical data, but becomes an issue as soon as the data is anything else. Pandas 2.0 has started to bring in Arrow, but it's not yet the standard (you have to opt-in and according to the developers it's going to stay that way for the foreseeable future). Also, pandas's Arrow-based features are not yet entirely on... - Source: dev.to / over 1 year ago

• TXR Lisp

  IMO a good first step would be to use the txr FFI to write a library for Apache arrow: https://arrow.apache.org/. - Source: Hacker News / over 1 year ago

• A Polars exploration into Kedro

  Polars is an open-source library for Python, Rust, and NodeJS that provides in-memory dataframes, out-of-core processing capabilities, and more. It is based on the Rust implementation of the Apache Arrow columnar data format (you can read more about Arrow on my earlier blog post “Demystifying Apache Arrow”), and it is optimised to be blazing fast. - Source: dev.to / almost 2 years ago

• Demystifying Apache Arrow

  Apache Arrow (Arrow for short) is an open source project that defines itself as "a language-independent columnar memory format" (more on that later). It is part of the Apache Software Foundation, and as such is governed by a community of several stakeholders. It has implementations in several languages (C++ and also Rust, Julia, Go, and even JavaScript) and bindings for Python, R and others that wrap the C++... - Source: dev.to / almost 2 years ago

• GPU vendor-agnostic fluid dynamics solver in Julia

  Are you talking about Apache Arrow? Interesting! Don't think I've seen this one. https://arrow.apache.org/. - Source: Hacker News / almost 2 years ago

• Making Python 100x faster with less than 100 lines of Rust

  Apache Arrow (https://arrow.apache.org/) is built exactly around this idea: it's a library for managing the in-memory representation of large datasets. - Source: Hacker News / about 2 years ago

• Show HN: Up to 100x Faster FastAPI with simdjson and io_uring on Linux 5.19

  If anything you'd probably want to send it in Arrow[1] format. CSV's don't even preserve data types. [1]: https://arrow.apache.org/. - Source: Hacker News / about 2 years ago

• IPC communication between rust, c++, and python

  In that case, why not use polars, which supports apache arrow format which supports C, C++, Rust, Python and supports zero-copy read. Source: over 2 years ago

• Introducing ArrowJS • Reactivity without the framework

  I think the naming will likely cause some confusion with apache arrow. My initial thoughts when reading "Introducing ArrowJS" was a new port of the apache arrow spec. Source: over 2 years ago

• Java Serialization with Protocol Buffers

  The information can be stored in a database or as files, serialized in a standard format and with a schema agreed with your Data Engineering team. Depending on your information and requirements, it can be as simple as CSV, XML or JSON, or Big Data formats such as Parquet, Avro, ORC, Arrow, or message serialization formats like Protocol Buffers, FlatBuffers, MessagePack, Thrift, or Cap'n Proto. - Source: dev.to / over 2 years ago

• GlueSQL: A SQL database engine written as a library in Rust

  Just another embedded SQL engine. There are SQLite(OLTP), DuckDB(OLAP) and some engine-based project like mentioned Apache Arrow(https://arrow.apache.org/)(OLAP): Apache Arrow has many language implementations, some do not include the query engine(for example, Rust implementation, which depends on the DataFusion for more SQL-like analytics) in its own repo, but other do include(for example, C++). There is a... - Source: Hacker News / over 2 years ago

• New Pandas-for-Haskell data frame library: Name suggestions

  This is a meta-request for the library, but imo it would be really awesome if it used a data structure compatible with Arrow: https://arrow.apache.org/. Source: over 2 years ago

• How to Deploy ML Models Using Gravity AI and Meadowrun

  As a bit of an aside, you could imagine a way to get the best of both worlds with an extension to Docker that would allow you to publish a container that exposes a Python API, so that someone could call sentiment = call_container_api(image="huggingface/transformers", "my input text") directly from their python code. This would effectively be a remote procedure call into a container that is not running as a service... - Source: dev.to / over 2 years ago

• Scala needs a good, dependency-free DataFrame library

  I assume you mean to use Apache arrow rather than scala Arrow? Source: over 2 years ago

• Dragonflydb – A modern replacement for Redis and Memcached

  I've used Apache Arrow before[1]; in-memory columnar storage. We did some AI/ML stuff with data gathered from social network APIs, but you can probably do a ton of things. [1] https://arrow.apache.org/. - Source: Hacker News / almost 3 years ago