Ingesting Data into OpenSearch using Apache Kafka and Go

Programming Language Stream Processing Custom Search Engine

OpenSearch Landing Page

1

OpenSearch

OpenSearch is a community-driven, open source search and analytics suite derived from Apache 2.0 licensed Elasticsearch 7.10.2 & Kibana 7.10.2. It consists of a search engine daemon, and a visualization and user interface, OpenSearch Dashboards.

Scalable data ingestion is a key aspect for a large-scale distributed search and analytics engine like OpenSearch. One of the ways to build a real-time data ingestion pipeline is to use Apache Kafka. It's an open-source event streaming platform used to handle high data volume (and velocity) and integrates with a variety of sources including relational and NoSQL databases. For example, one of the canonical use cases is real-time synchronization of data between heterogeneous systems (source components) to ensure that OpenSearch indexes are fresh and can be used for analytics or consumed downstream applications via dashboards and visualizations.

#Custom Search Engine #Search Engine #Custom Search 26 social mentions
Apache Kafka Landing Page
2

Apache Kafka

Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala.
Pricing:
- Open Source
Scalable data ingestion is a key aspect for a large-scale distributed search and analytics engine like OpenSearch. One of the ways to build a real-time data ingestion pipeline is to use Apache Kafka. It's an open-source event streaming platform used to handle high data volume (and velocity) and integrates with a variety of sources including relational and NoSQL databases. For example, one of the canonical use cases is real-time synchronization of data between heterogeneous systems (source components) to ensure that OpenSearch indexes are fresh and can be used for analytics or consumed downstream applications via dashboards and visualizations.

#Stream Processing #Data Integration #ETL 144 social mentions
Go Programming Language Landing Page
3

Go Programming Language

Go, also called golang, is a programming language initially developed at Google in 2007 by Robert...
Pricing:
- Open Source
Thankfully, Kafka and OpenSearch provide client libraries for a variety of programming languages which make it possible to write your own integration layer. This is exactly what's covered in this blog! We will make use of a custom Go application to ingest data using Go clients for Kafka and OpenSearch.

#Programming Language #OOP #Generic Programming Language 322 social mentions