I have lived through the hype of Big data it was a time of HDFS+HTable I guess and Hapoop etc. One can't go wrong with DuckDB+SQLite+Open/Elasticsearch either with 6 to 8 even 10 TB of data. [0]. https://duckdb.org/. - Source: Hacker News / 23 days ago

More than once, I have been in a situation where I needed to query CloudTrail logs but was working in a customer environment where they weren’t aggregated to a search interface. Another similar situation is when CloudTrail data events are disabled for cost reasons but need to be temporarily turned on for troubleshooting/audit purposes. While the CloudTrail console offers some (very) limited lookups (for management... - Source: dev.to / 29 days ago

DuckDB: An in-process SQL OLAP database management system. While not a traditional OLAP database, DuckDB is designed to execute analytical queries efficiently, making it suitable for analytical workloads within data-intensive applications. - Source: dev.to / 4 months ago

Easiest way to practically use SIMD table scan database is try out DuckDB: https://duckdb.org/. - Source: Hacker News / 4 months ago

Duckdb so we can make OLAP like queries on the data. - Source: dev.to / 7 months ago

A Kafka-based data integration platform will be a good fit here. The services can add events to different topics in a broker whenever there is a data update. Kafka consumers corresponding to each of the services can monitor these topics and make updates to the data in real-time. It is also possible to create a unified data store through the same integration platform. Developers can implement a unified store either... - Source: dev.to / almost 2 years ago