Google Cloud Dataflow
Google BigQuery
Amazon EMR
Qubole
Snowflake
Databricks
Confluent
Google Cloud Dataproc
OctoSQL
Materialize
LNAV
Steampipe
DuckDB
RisingWave
Frame
DSQ
OctoSQLNo OctoSQL videos yet. You could help us improve this page by suggesting one.
Based on our record, OctoSQL should be more popular than Google Cloud Dataflow. It has been mentiond 23 times since March 2021. We are tracking product recommendations and mentions on various public social media platforms and blogs. They can help you identify which product is more popular and what people think of it.
Imo if you are using the cloud and not doing anything particularly fancy the native tooling is good enough. For AWS that is DMS (for RDBMS) and Kinesis/Lamba (for streams). Google has Data Fusion and Dataflow . Azure hasData Factory if you are unfortunate enough to have to use SQL Server or Azure. Imo the vendored tools and open source tools are more useful when you need to ingest data from SaaS platforms, and... Source: over 2 years ago
This sub is for Apache Beam and Google Cloud Dataflow as the sidebar suggests. Source: almost 3 years ago
I am pretty sure they are using pub/sub with probably a Dataflow pipeline to process all that data. Source: about 3 years ago
You can run a Dataflow job that copies the data directly from BQ into S3, though you'll have to run a job per table. This can be somewhat expensive to do. Source: about 3 years ago
It was clear we needed something that was built specifically for our big-data SaaS requirements. Dataflow was our first idea, as the service is fully managed, highly scalable, fairly reliable and has a unified model for streaming & batch workloads. Sadly, the cost of this service was quite large. Secondly, at that moment in time, the service only accepted Java implementations, of which we had little knowledge... - Source: dev.to / over 3 years ago
This looks extremely cool. This is basically incremental view maintenance in databases, a problem that almost everybody (I think) has when using SQL databases and wanting to do some derived views for more performant access patterns. Importantly, they seem to support a wide breath of SQL operators, and it's open-source! There's already a bunch of tools in this area: 1. Materialize[0], which afaik is more... - Source: Hacker News / about 1 year ago
OctoSQL[0] or DuckDB[1] will most likely be much simpler, while going through 10 GB of JSON in a couple seconds at most. Disclaimer: author of OctoSQL [0]: https://github.com/cube2222/octosql. - Source: Hacker News / over 2 years ago
This is really cool! With their Postgres scanner[0] you can now easily query multiple datasources using SQL and join between them (i.e. Postgres table with JSON file). Something I strived to build with OctoSQL[1] before. It's amazing to see how quickly DuckDB is adding new features. Not a huge fan of C++, which is right now used for authoring extensions, it'd be really cool if somebody implemented a Rust extension... - Source: Hacker News / over 2 years ago
Congrats on the Show HN! It's great to see more tools in this area (querying data from various sources in-place) and the Lambda use case is a really cool idea! I've recently done a bunch of benchmarking, including ClickHouse Local and the usage was straightforward, with everything working as it's supposed to. Just to comment on the performance area though, one area I think ClickHouse could still possibly improve... - Source: Hacker News / over 2 years ago
SPyQL is really cool and its design is very smart, with it being able to leverage normal Python functions! As far as similar tools go, I recommend taking a look at DataFusion[0], dsq[1], and OctoSQL[2]. DataFusion is a very (very very) fast command-line SQL engine but with limited support for data formats. Dsq is based on SQLite which means it has to load data into SQLite first, but then gives you the whole breath... - Source: Hacker News / almost 3 years ago
Google BigQuery - A fully managed data warehouse for large-scale data analytics.
Materialize - A Streaming Database for Real-Time Applications
Amazon EMR - Amazon Elastic MapReduce is a web service that makes it easy to quickly process vast amounts of data.
LNAV - The Log File Navigator (lnav) is an advanced log file viewer for the console.
Qubole - Qubole delivers a self-service platform for big aata analytics built on Amazon, Microsoft and Google Clouds.
Steampipe - Steampipe: select * from cloud; The extensible SQL interface to your favorite cloud APIs select * from AWS, Azure, GCP, Github, Slack etc.
