Estuary Flow Reviews and details

Social recommendations and mentions

• Ask HN: Freelancer? Seeking freelancer? (April 2024)

  SEEKING FREELANCER | Python Developer | Remote (Within 3 hours of EST) Estuary is a dynamic company focused on developing cutting-edge real-time data integration solutions. Our platform is powered by an open-source repository of pre-built data connectors, making data exchange between systems seamless. https://estuary.dev/ We are seeking a passionate and talented Software Engineer to help expand our catalog of data... - Source: Hacker News / about 1 month ago

• How do Streaming Aggregation Pipelines work?

  I work at Estuary, which is itself a streaming data pipeline. We actually use that approach to power all of the data processing statistics we show in our UI. Lately we've been processing ~200-300 transactions per second (each transaction produces a stats event), and the stats queries in the dashboard are quite snappy. We actually pre-aggregate by minute, hour, and day in order to serve queries of larger time... Source: 5 months ago

• All the ways to capture changes in Postgres

  Estuary (https://estuary.dev ; I'm CTO) gives you a real time data lake'd change log of all the changes happening in your database in your cloud storage -- complete with log sequence number, database time, and even before/after states if you use REPLICA IDENTITY FULL -- with no extra setup in your production DB. By default, if you then go on to materialize your collections somewhere else (like Snowflake), you get... - Source: Hacker News / 8 months ago

• sharing kafka streams externally

  Disclaimer: I work for a streaming ETL startup (estuary.dev) with a connector for Kafka and ability to share data. I'm wondering if Confluent's currently functionality is missing features by not more easily enabling to push shared streams into the consumer.... Or just generally other things that are on the 'wish list' of those sharing / receiving topics. Source: 8 months ago

• Launch HN: Artie (YC S23) – Real time data replication to data warehouses

  Hi, I'm Estuary's CTO (https://estuary.dev). Mind speaking a bit more about what didn't work? We put quite a bit of effort into our CDC connectors, as it's a core competency. We have numerous customers using them at scale successfully, but they can be a bit nuanced to get configured. We're constantly trying to make our onboarding experience more intuitive and seamless... it's a hard problem. - Source: Hacker News / 10 months ago

• Lakehouse - event data

  Gazette is actually what forms the beating heart of our streaming data platform. We went with gzipped newline-delimited json files, and I have no regrets there. It's easy and cheap to read and write, and it compresses remarkably well with gzip. Source: 10 months ago

• Data-Oriented Design Principles

  In contrast to jebarker's comment, I actually think it's really interesting that a concept coming from game engine development actually seems quite applicable in some very different domains. We (https://estuary.dev/) ended up arriving at a very similar design for transformations in streaming analytics pipelines: https://docs.estuary.dev/concepts/derivations/ To paraphrase, each derivation produces a collection of... - Source: Hacker News / 10 months ago

• ETL vs Reverse ETL - why do we have 2 different sets of tools?

  One random thought since you mentioned databases on each side - is estuary.dev. I'm pretty early in the evaluation phase and I'm working on getting a trial slotted on my roadmap, so I don't have any "real-world thoughts", but they support having one data source being materialized to multiple destinations, which starts to look like reverse etl (for datastores). Their perspective on saas extraction sounds... Source: 10 months ago

• Fivetran vs Estuary.dev

  Can anyone speak to estuary.dev and their thoughts on it? Source: 10 months ago

• Data Engineering Frameworks

  You might call Estuary a data engineering framework (also where I work). Same goes for things like Spark, Flink, and even Airflow. We've tended to use slightly more specific terminology, like "orchestration platform" for airflow, so Data Engineering Framework isn't really a term you hear very often. But there's definitely lots of things out there that fit the (vague) definition of "framework" and are commonly used... Source: 11 months ago

• Streaming ETL tools for SaaS products?

  We do exactly this at Estuary. Theres a useful free tier, too! And once you capture the data, we can do streaming materializations of the data into any number of destinations. I look forward to hearing what you think. Source: 11 months ago

• How to handle data from source systems that will be used both for analytics and IT services?

  I can tell you how we'd handle this with [Estuary](https://estuary.dev). A lot of this could also be applied to other streaming platforms or bespoke solutions, too. Source: 11 months ago

• Ask HN: AI Pipelines have a “Customer 360” problem?

  Thesis: I suspect that AI pipelines are the killer application for incremental, deeply stateful stream processing. But I’m very curious for the community’s perspective. Thanks. [0]: Co-founder of streaming data ops company https://estuary.dev. - Source: Hacker News / 12 months ago

• Rust in the world of data

  We're building Estuary Flow mostly in Rust. It's a realtime data ops platform. We don't (yet) support Rust as a transformation language, but we plan to support transforms as arbitrary docker images, which would allow you to write transforms in Rust. Source: about 1 year ago