Serverless Data Processor using AWS Lambda, Step Functions and Fargate on ECS (with Rust 🦀🦀)

1

AWS Step Functions

There are a few ways to solve this of course but one solution I wanted to explore is using AWS Step Functions (https://aws.amazon.com/step-functions/) to drive the whole process. Step Functions is a serverless workflow orchestration system. One part of it is support for a distributed map mode where you can run many parallel operations over a set of data. There are different approaches you can use to get the list of work items but the one that makes the most sense for this problem is iterating over all the objects in an S3 bucket based on a prefix.

#Project Management #Workflow Automation #Web Service Automation 57 social mentions

2

AWS Fargate

One great option in the serverless world for something like this is to run containers using AWS Fargate (https://aws.amazon.com/fargate/). Fargate is a service from AWS where you don't need to spin up or manage EC2 VMs to get access to compute. Also you don't need to pay for a container orchestration layer. You just provide a docker image and the specs of what you need to run it (cpu, ram, disk, etc) and AWS spins up the computer and runs the container.

#Developer Tools #Containers As A Service #DevOps Tools 44 social mentions

3

Amazon ECS

You can use this in a couple of ways on AWS including with the Elastic Container Service (ECS) https://aws.amazon.com/ecs/ or within an Elastic Kuberneters Service (EKS) cluster. When used with EKS you will need to have an always on EKS control plane which will cost you money.

#Developer Tools #Containers As A Service #Cloud Computing 46 social mentions