Estuary Flow Reviews

Apache Flink Reviews

Estuary Flow mentions (14)

• Ask HN: Freelancer? Seeking freelancer? (April 2024)

  SEEKING FREELANCER | Python Developer | Remote (Within 3 hours of EST) Estuary is a dynamic company focused on developing cutting-edge real-time data integration solutions. Our platform is powered by an open-source repository of pre-built data connectors, making data exchange between systems seamless. https://estuary.dev/ We are seeking a passionate and talented Software Engineer to help expand our catalog of data... - Source: Hacker News / about 2 months ago

• How do Streaming Aggregation Pipelines work?

  I work at Estuary, which is itself a streaming data pipeline. We actually use that approach to power all of the data processing statistics we show in our UI. Lately we've been processing ~200-300 transactions per second (each transaction produces a stats event), and the stats queries in the dashboard are quite snappy. We actually pre-aggregate by minute, hour, and day in order to serve queries of larger time... Source: 6 months ago

• All the ways to capture changes in Postgres

  Estuary (https://estuary.dev ; I'm CTO) gives you a real time data lake'd change log of all the changes happening in your database in your cloud storage -- complete with log sequence number, database time, and even before/after states if you use REPLICA IDENTITY FULL -- with no extra setup in your production DB. By default, if you then go on to materialize your collections somewhere else (like Snowflake), you get... - Source: Hacker News / 8 months ago

• sharing kafka streams externally

  Disclaimer: I work for a streaming ETL startup (estuary.dev) with a connector for Kafka and ability to share data. I'm wondering if Confluent's currently functionality is missing features by not more easily enabling to push shared streams into the consumer.... Or just generally other things that are on the 'wish list' of those sharing / receiving topics. Source: 9 months ago

• Launch HN: Artie (YC S23) – Real time data replication to data warehouses

  Hi, I'm Estuary's CTO (https://estuary.dev). Mind speaking a bit more about what didn't work? We put quite a bit of effort into our CDC connectors, as it's a core competency. We have numerous customers using them at scale successfully, but they can be a bit nuanced to get configured. We're constantly trying to make our onboarding experience more intuitive and seamless... it's a hard problem. - Source: Hacker News / 10 months ago

Apache Flink mentions (29)

• Array Expansion in Flink SQL

  I’ve recently started my journey with Apache Flink. As I learn certain concepts, I’d like to share them. One such "learning" is the expansion of array type columns in Flink SQL. Having used ksqlDB in a previous life, I was looking for functionality similar to the EXPLODE function to "flatten" a collection type column into a row per element of the collection. Because Flink SQL is ANSI compliant, it’s no surprise... - Source: dev.to / 5 days ago

• Show HN: An SQS Alternative on Postgres

  You should let the Apache Flink team know, they mention exactly-once processing on their home page (under "correctness guarantees") and in their list of features. [0] https://flink.apache.org/ [1] https://flink.apache.org/what-is-flink/flink-applications/#building-blocks-for-streaming-applications. - Source: Hacker News / 19 days ago

• Top 10 Common Data Engineers and Scientists Pain Points in 2024

  Data scientists often prefer Python for its simplicity and powerful libraries like Pandas or SciPy. However, many real-time data processing tools are Java-based. Take the example of Kafka, Flink, or Spark streaming. While these tools have their Python API/wrapper libraries, they introduce increased latency, and data scientists need to manage dependencies for both Python and JVM environments. For example,... - Source: dev.to / about 2 months ago

• Choosing Between a Streaming Database and a Stream Processing Framework in Python

  Other stream processing engines (such as Flink and Spark Streaming) provide SQL interfaces too, but the key difference is a streaming database has its storage. Stream processing engines require a dedicated database to store input and output data. On the other hand, streaming databases utilize cloud-native storage to maintain materialized views and states, allowing data replication and independent storage scaling. - Source: dev.to / 4 months ago

• Go concurrency simplified. Part 4: Post office as a data pipeline

  Also, this knowledge applies to learning more about data engineering, as this field of software engineering relies heavily on the event-driven approach via tools like Spark, Flink, Kafka, etc. - Source: dev.to / 5 months ago