EasyMorph VS Amazon Kinesis Firehose

Your task could be trivially done with EasyMorph (https://easymorph.com). We've designed it exactly for such use cases. Data needs of non-technical people have long been neglected. It was believed that any data wrangling should be done by IT people. So all non-technical people had was Excel. Luckily, the no-code movement finally started addressing that issue with a varying degree of success. - Source: Hacker News / over 1 year ago

EasyMorph | F# Developer | Full-time, Remote | https://easymorph.com EasyMorph is a powerful data wrangling and ETL application that makes life easier for thousands of knowledge workers around the world. Your job would be to keep developing our high-performance best-in-class columnar data transformation engine that lies at the heart of the application. It's a challenging task that requires an experienced F#... - Source: Hacker News / almost 2 years ago

>PRQL is intended to be a modern, simple, declarative language for transforming data It's not declarative. It's functional. I believe that the approach that is followed by PRQL is more practical than SQL. We've implemented a similar approach in our visual ETL tool for non-technical people (https://easymorph.com) and it works wonderfully. Other cool things you can do with this approach (and can't with SQL): *... - Source: Hacker News / over 2 years ago

I started a company that built, I believe, the most advanced no-code data automation desktop application (https://easymorph.com). It's entirely bootstrapped, profitable and pays salaries to 10+ people (and we keep hiring). Many our customers specifically looked for an on-prem solution as they don't want to deal with SaaS and its privacy issues and use-based pricing. With an on-prem tool like ours they get... - Source: Hacker News / over 2 years ago

May I suggest your non-tech users to try EasyMorph[1] (I'm the founder)? It's easier than PowerQuery (it's completely no-code) and at the same time more powerful. 80% of PowerQuery capabilities require coding. Our tool is 100% visual (and has a free edition). [1] https://easymorph.com. - Source: Hacker News / over 2 years ago

First, you may not know the Kinesis Data Firehose service. Here's the AWS definition: Amazon Kinesis Data Firehose is an Extract, Transform, and Load (ETL) service that captures, transforms, and reliably delivers streaming data to data lakes, data stores, and analytics services. (https://aws.amazon.com/kinesis/data-firehose/). - Source: dev.to / about 1 year ago

As you can see in the diagram, we are feeding all events from Event Bus via a catch-all rule into Kinesis Data Firehose. Firehose is a fully managed service that streams into specific destinations like Data Warehouses or Data Lakes. Unlike it's bigger brother of using Kinesis Data Streams directly, there are no setting up of shards and it's mostly configuration free. We are only defining a buffer interval which is... - Source: dev.to / over 1 year ago

When using EventBridge I always log all events to an S3 bucket for auditing, analytics and debugging purposes. A super easy method to do this is to create a Kinesis Data Firehose stream and create a rule that captures all events that points to the Firehose stream. The Firehose stream can then flush the events on S3 in an interval/size of choice based on configuration. - Source: dev.to / over 1 year ago

Have you looked at Kinesis Firehose? It was pretty much build for this use case although you will still need to see if you can define a partitioning scheme probably in combination with an S3 Select query to meet your query requirements. https://aws.amazon.com/kinesis/data-firehose/?nc=sn&loc=0. - Source: Hacker News / almost 2 years ago

Is continuous backup important ? e.g. If the stuff fails for one day and you lose that day's upload is that ok? Do you want it to push updates more frequently than once a day? If you want to continuously push updates then Kinesis Firehose might be worth looking into. Source: over 2 years ago