Apache Kafka Reviews

PGLoader Reviews

Apache Kafka mentions (155)

• Building Kafka Producer-Consumer Using Go and Docker

  Kafka is a distributed streaming platform used to build real-time data pipelines and streaming applications. It allows producers to send messages to topics, which are then consumed by various consumers, making it ideal for event-driven architectures. - Source: dev.to / about 1 month ago

• 7 Free Tools for Data Pipeline Reconciliation and Cross-Source Validation

  Apache Kafka is the most widely used distributed event streaming platform and the standard transport layer for event-driven reconciliation architectures. - Source: dev.to / 2 months ago

• How to Build a Dead Letter Queue System for Reliable Data Processing

  For message-queue-based pipelines: RabbitMQ has native DLQ support through dead letter exchanges. Messages that exceed their retry count or their time-to-live are automatically routed to a designated DLQ exchange. Apache Kafka does not have native DLQ semantics, but the standard pattern is to write failed records to a dedicated topic (-dlq by convention) and include the failure metadata in the record headers. - Source: dev.to / 2 months ago

• Idempotency in Data Pipelines: How to Prevent Duplicate Records

  Upsert with timestamp tracking. Keep the upsert approach but track which time windows have been fully processed. On retry, skip windows that are marked complete and reprocess only windows that failed mid-run. The Kafka documentation covers offset management patterns that implement this for stream-based pipelines. - Source: dev.to / 2 months ago

• Real-Time Fraud Detection in Java with Kafka Streams and Vector Similarity

  Apache Kafka allows the payment service to publish a transaction event to a topic, without knowing who will consume it. The fraud service, the notification service, and any other interested component can subscribe to that topic independently:. - Source: dev.to / 3 months ago

PGLoader mentions (12)

• Migrating from MySQL to PostgreSQL

  Pg Loader is a tool that can be used to move your data to PostgreSQL, however, it's not perfect, but can work well in some cases. It's worth looking at to see if it's the direction you want to go. - Source: dev.to / about 2 years ago

• We need to talk about parentheses

  Examples (for Common Lisp, so not citing Emacs): reddit v1, Google's ITA Software that powers airfare search engines (Kayak, Orbitz…), Postgres' pgloader (http://pgloader.io/), which was re-written from Python to Common Lisp, Opus Modus for music composition, the Maxima CAS, PTC 3D designer CAD software (used by big brands worldwide), Grammarly, Mirai, the 3D editor that designed Gollum's face, the ScoreCloud app... - Source: Hacker News / over 2 years ago

• We migrated our PostgreSQL database with 11 seconds downtime

  I worked on migrating our MySQL system to PostgreSQL using pgloader ( https://pgloader.io/ ). There were some hiccups, things that needed clarification in documentation, and some additional processes that needed to be done outside of the system to get everything we need in place, it was a amazing help. Not sure the project would've been possible without it. Data mapping from PostgreSQL to PostgreSQL as in the... - Source: Hacker News / over 2 years ago

• Time For Me To Fly… To Render

  Initially, I started down the pgloader path, which seemed to be a common approach for the database conversion. However, using my M1-chip MacBook Pro led to some unexpected issues. Instead, I opted to use NMIG to convert MySQL to PostgreSQL. For more information, please check out the “Highlights From the Database Conversion” section below. - Source: dev.to / over 3 years ago

• What ETL tool you use with Postgres ?

  I would warmly recommend https://pgloader.io but, unfortunately, the Oracle support is still in need of a sponsor :-). Source: over 3 years ago