AWS Step Functions VS Amazon EMR

AWS Step Functions Reviews

Amazon EMR Reviews

AWS Step Functions mentions (57)

• Serverless Data Processor using AWS Lambda, Step Functions and Fargate on ECS (with Rust 🦀🦀)

  There are a few ways to solve this of course but one solution I wanted to explore is using AWS Step Functions (https://aws.amazon.com/step-functions/) to drive the whole process. Step Functions is a serverless workflow orchestration system. One part of it is support for a distributed map mode where you can run many parallel operations over a set of data. There are different approaches you can use to get the list... - Source: dev.to / 3 months ago

• The Energy Drink Episodes 3: The Step Function Awakens

  If you have ever spoken to me, read anything I've written or listened to any talks I’ve done in relation to Serverless or infrastructure as code, there is a high likelihood that I have confessed my love for Step Functions. Even when unprompted. Putting my biases aside, however, there are some legitimate reasons we can consider using them in our app. If you are new to Step Functions or just fancy a refresher, have... - Source: dev.to / 5 months ago

• Testing Serverless Applications on AWS

  For context; the web application is built with React and TypeScript which makes calls to an AppSync API that makes use of the Lambda and DynamoDB datasources. We use Step Functions to orchestrate the flow of events for complex processing like purchasing and renewing policies, and we use S3 and SQS to process document workloads. - Source: dev.to / 6 months ago

• Customizing error handling in Step Functions

  If we have to coordinate multiple function calls, we can use AWS Step Functions to orchestrate the workflow. Step Functions integrates with many other AWS services, but here I'll focus on Lambda functions. - Source: dev.to / 12 months ago

• Amazon Ditches Microservices for Monolith: Decoding Prime Video's Architectural Shift

  Amazon's tool was broken into three main components: the media converter, defect detectors, and orchestration. The media converter ran as an AWS Lambda function, converted audio and video streams, and stored the data in an S3 bucket. The defect detectors, also running as an AWS Lambda function, would pull the parsed data from the S3 bucket and analyze the frames and audio for any issues. Finally, the orchestration... - Source: dev.to / 12 months ago

Amazon EMR mentions (10)

• 5 Best Practices For Data Integration To Boost ROI And Efficiency

  There are different ways to implement parallel dataflows, such as using parallel data processing frameworks like Apache Hadoop, Apache Spark, and Apache Flink, or using cloud-based services like Amazon EMR and Google Cloud Dataflow. It is also possible to use parallel dataflow frameworks to handle big data and distributed computing, like Apache Nifi and Apache Kafka. Source: about 1 year ago

• What compute service i should use? Advice for a duck-tape kind of guy

  I'm going to guess you want something like EMR. Which can take large data sets segment it across multiple executors and coalesce the data back into a final dataset. Source: almost 2 years ago

• Processing a large text file containing millions of records.

  This is exactly the kind of workload EMR was made for, you can even run it serverless nowadays. Athena might be a viable option as well. Source: almost 2 years ago

• How to use Spark and Pandas to prepare big data

  Apache Spark is one of the most actively developed open-source projects in big data. The following code examples require that you have Spark set up and can execute Python code using the PySpark library. The examples also require that you have your data in Amazon S3 (Simple Storage Service). All this is set up on AWS EMR (Elastic MapReduce). - Source: dev.to / over 2 years ago

• Beginner building a Hadoop cluster

  Check out https://aws.amazon.com/emr/. Source: about 2 years ago