Processing large payloads with Amazon API Gateway asynchronously

Cloud Hosting Object Storage Cloud Storage

AWS Step Functions Landing Page

1

AWS Step Functions

AWS Step Functions makes it easy to coordinate the components of distributed applications and microservices using visual workflows.

But what if that 15-minute timeout limitation is a thorn at your side? What if the process within the compute layer of the solution is complex and would benefit from splitting it into multiple chunks? If that is the case, keep on reading, we are going to be talking about AWS Step Functions next.

#Workflow Automation #Workflows #Automation 67 social mentions
Amazon S3 Landing Page

2

Amazon S3

Amazon S3 is an object storage where users can store data from their business on a safe, cloud-based platform. Amazon S3 operates in 54 availability zones within 18 graphic regions and 1 local region.

Suppose you were to ask the community about the potential solution to this problem. In that case, I wager that the most common answer would be to use Amazon S3 as the data storage layer and utilize presigned URLs to push the data to Amazon S3 storage.

#Cloud Hosting #Object Storage #Cloud Storage 196 social mentions
AWS Lambda Landing Page
3

AWS Lambda

Automatic, event-driven compute service
Pricing:
- Open Source
The AWS Lambda service is often called a "swiss knife" of serverless. In most cases, processing the data in-memory within the AWS Lambda is sufficient.

#Cloud Computing #Cloud Hosting #Business & Commerce 273 social mentions
Amazon Elastic File System Landing Page

4

Amazon Elastic File System

Amazon Elastic File System is a cloud-native and serverless data store that provides simple saleable elastic shared file storage for Linux-based business applications.

Depending on the requirements and constraints, I like to use Amazon EFS integration with AWS Lambda in such situations. Amazon EFS allows me to have a storage layer shared by all AWS Lambda functions that partake in the workflow. Let us look into that next.

#File Management #Work Collaboration #Project Management 25 social mentions
DynamoDB Landing Page

5

DynamoDB

Amazon DynamoDB is a fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed cloud database and supports both document and key-value store models.

In my humble opinion, the key:value nature of the "work progress" data makes the Amazon DynamoDB a perfect choice for the database component (nothing is stopping you from using RDS, which also supports CDC events, which themselves are optional here).

#Databases #NoSQL Databases #Relational Databases 119 social mentions
AWS CloudFormation Landing Page
6

AWS CloudFormation

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a...
Pricing:
- Open Source
On the IaC side of things, know that, in creating the AWS CloudFormation template, you might end up with circular dependency problems. Deployment frameworks like AWS CDK will make sure that should never happen. However, I still think you should be aware of this potential problem, even if you use deployment frameworks.

#Continuous Integration #DevOps Tools #Continuous Deployment 123 social mentions
Amazon API Gateway Landing Page

7

Amazon API Gateway

Create, publish, maintain, monitor, and secure APIs at any scale

As stated in the blog post, the ultimate solution for a "big payload" problem is making the architecture asynchronous. Let us then zoom in on the aspect of asynchronous communication in the Amazon API Gateway service context and build a serverless architecture based on the notion of "pending work".

#API Tools #APIs #Web Service Automation 107 social mentions