For the data loaded from the raw and unstructured data sources (RESTful APIs, files, feeds) I'd recommend checking tablum.io. The data first gets into the staging SQL DB where you can apply transformations in SQL (ClickHouse dialect with hundreds of ready-to-use functions for aggregations, transformations, string and array processing, etc). After that, you can set up a pipeline to external DWH or DB. Or just... Source: about 1 year ago

Fairly easy to parse it with Python (or any other scripting languages that support regular expressions). Besides, there're no-code parsers that can do that. E.g. tablum.io, it has a regexp parser that turns poorly formatted text into an SQL table. All you need it is to provide the regular expression (regexp) to split the text (per-line basis or the entire text) into chunks that become SQL rows and columns, after... Source: about 1 year ago

You're right. There should be some way to safely pass the authentication token without storing it in the query. BTW, to keep all data within the company network perimeter and no share any keys with the cloud service, one can install tablum.io as a self-hosted version on a Linux server (it runs in Docker). Source: about 1 year ago

For anyone interested in exploring the data yourself, here are a few tools https://dadroit.com/: desktop tool for processing large JSON files.There’s a free version so you can get pretty far with it if you’re patient https://www.dolthub.com/repositories/dolthub/quest: they’re running a bounty program for some healthcare providers to process the machine readable files, and have some useful code snippets in case... - Source: Hacker News / over 1 year ago

Note for people who don't know much about FreePascal. It is a full-featured and very fast compiler. The resulting program is a rival for the best output of C/CPP compilers. It can be used in the style of simpler languages like Go and is almost as safe as Rust in a much faster manner. It has a great but old-looking IDE, Lazarus. It has been under active development for decades and is used for proper projects like:... - Source: Hacker News / over 1 year ago

The files are insanely large. Eventually the only way I've been able to open them is using the DADROIT large JSON viewer: http://dadroit.com (but even this only worked when I got an M1 Mac). Source: over 1 year ago

For those looking for the ability to (locally) open and query very large JSON files, Dadroit is great: https://dadroit.com. It's been a while since I used it last, but it was a life saver for me when working with JSON lines files in S3 and Kafka log dumps. A side tidbit, IIRC, it was written in Pascal. - Source: Hacker News / over 1 year ago

You can review all the contents of it, but generally, there is one big 5GB JSON file — yelp_academic_dataset_review.json it contains 6,9 🍋 JSON records. Use Dadroid to review massive JSON files. - Source: dev.to / over 1 year ago