Redis Streams in Action: Part 1 (Intro and overview)

1

RediSearch

Available as a Redis module, RediSearch provides flexible search capabilities, thanks to a first-class secondary indexing engine. Some of its key features include full-text search, auto completion, and geographical indexing. There are a bunch of other features whose detailed exploration is out of scope of this blog series. I would highly recommend you to go through the documentation to explore further. For now, here is a quick overview of some of the RediSearch commands. You will see them in action in subsequent blog posts.

#Custom Search Engine #Custom Search #Search Engine 1 social mentions

2

NATS

I had written a blog post RediSearch in Action that covered the same use case i.e. How to implement a set of applications for consuming tweets in real-time, index them in RediSearch and query them using a REST API. However, the solution presented here has been implemented with the help of Redis Streams along with other components in order to make the architecture scalable and fault-tolerant. In this specific example, it's the ability to process large volume of tweets, but the same idea can be extended/applied to other use-cases which deal with high velocity data e.g. IoT, log analytics, etc. Such problems benefit from an architecture where you can horizontally scale out your applications to handle increasing data volumes. Typically, this involves introducing a Messaging system to act a buffer between producers and consumers. Since this is a common requirement and the problem space is well understood, there are lot of established solutions in the distributed messaging world ranging from JMS (Java Messaging Service), Apache Kafka, RabbitMQ, NATS, and of course Redis.

#Developer Tools #Queueing, Messaging And Background Processing #App Development 73 social mentions

3

Apache Kafka

I had written a blog post RediSearch in Action that covered the same use case i.e. How to implement a set of applications for consuming tweets in real-time, index them in RediSearch and query them using a REST API. However, the solution presented here has been implemented with the help of Redis Streams along with other components in order to make the architecture scalable and fault-tolerant. In this specific example, it's the ability to process large volume of tweets, but the same idea can be extended/applied to other use-cases which deal with high velocity data e.g. IoT, log analytics, etc. Such problems benefit from an architecture where you can horizontally scale out your applications to handle increasing data volumes. Typically, this involves introducing a Messaging system to act a buffer between producers and consumers. Since this is a common requirement and the problem space is well understood, there are lot of established solutions in the distributed messaging world ranging from JMS (Java Messaging Service), Apache Kafka, RabbitMQ, NATS, and of course Redis.

#Stream Processing #Data Integration #ETL 142 social mentions

4

Go Programming Language

Monitoring service: The last part is a Go application to monitor the progress of the tweets processor service and ensure that any failed records are re-processed. This is a Serverless component which will be deployed to Azure Functions where you can run it based on a Timer trigger and only pay for the duration it runs for.

#Programming Language #OOP #Generic Programming Language 322 social mentions