Your task could be trivially done with EasyMorph (https://easymorph.com). We've designed it exactly for such use cases. Data needs of non-technical people have long been neglected. It was believed that any data wrangling should be done by IT people. So all non-technical people had was Excel. Luckily, the no-code movement finally started addressing that issue with a varying degree of success. - Source: Hacker News / almost 2 years ago

EasyMorph | F# Developer | Full-time, Remote | https://easymorph.com EasyMorph is a powerful data wrangling and ETL application that makes life easier for thousands of knowledge workers around the world. Your job would be to keep developing our high-performance best-in-class columnar data transformation engine that lies at the heart of the application. It's a challenging task that requires an experienced F#... - Source: Hacker News / almost 2 years ago

>PRQL is intended to be a modern, simple, declarative language for transforming data It's not declarative. It's functional. I believe that the approach that is followed by PRQL is more practical than SQL. We've implemented a similar approach in our visual ETL tool for non-technical people (https://easymorph.com) and it works wonderfully. Other cool things you can do with this approach (and can't with SQL): *... - Source: Hacker News / over 2 years ago

I started a company that built, I believe, the most advanced no-code data automation desktop application (https://easymorph.com). It's entirely bootstrapped, profitable and pays salaries to 10+ people (and we keep hiring). Many our customers specifically looked for an on-prem solution as they don't want to deal with SaaS and its privacy issues and use-based pricing. With an on-prem tool like ours they get... - Source: Hacker News / over 2 years ago

May I suggest your non-tech users to try EasyMorph[1] (I'm the founder)? It's easier than PowerQuery (it's completely no-code) and at the same time more powerful. 80% of PowerQuery capabilities require coding. Our tool is 100% visual (and has a free edition). [1] https://easymorph.com. - Source: Hacker News / over 2 years ago

Apache Doris 2.1 has a data transmission channel built on Arrow Flight SQL. (Apache Arrow is a software development platform designed for high data movement efficiency across systems and languages, and the Arrow format aims for high-performance, lossless data exchange.) It allows high-speed, large-scale data reading from Doris via SQL in various mainstream programming languages. For target clients that also... - Source: dev.to / 11 days ago

In comes Polars: a brand new dataframe library, or how the author Ritchie Vink describes it... a query engine with a dataframe frontend. Polars is built on top of the Arrow memory format and is written in Rust, which is a modern performant and memory-safe systems programming language similar to C/C++. - Source: dev.to / 3 months ago

One is related to the heritage of being built around the NumPy library, which is great for processing numerical data, but becomes an issue as soon as the data is anything else. Pandas 2.0 has started to bring in Arrow, but it's not yet the standard (you have to opt-in and according to the developers it's going to stay that way for the foreseeable future). Also, pandas's Arrow-based features are not yet entirely on... - Source: dev.to / 6 months ago

IMO a good first step would be to use the txr FFI to write a library for Apache arrow: https://arrow.apache.org/. - Source: Hacker News / 6 months ago

Polars is an open-source library for Python, Rust, and NodeJS that provides in-memory dataframes, out-of-core processing capabilities, and more. It is based on the Rust implementation of the Apache Arrow columnar data format (you can read more about Arrow on my earlier blog post “Demystifying Apache Arrow”), and it is optimised to be blazing fast. - Source: dev.to / about 1 year ago