AWS Batch Reviews and details

Social recommendations and mentions

• Looking for a decent (self hostable) program to orchestrate scripts, notify on failures, etc

  After moving off Jenkins, I moved everything to AWS Batch with Fargate. This works quite well, but it is proving to be a little expensive, as I have to pay for:. Source: 10 months ago

• Hosting strategy suggestions

  If you're looking for more control over your infrastructure and want to run a full computing environment, EC2 might be the right choice for you. With EC2, you have complete control over the operating system, network, and storage, which can be useful if you need to install custom software or use specific hardware configurations. Additionally, EC2 + Batch processing provide a wider range of instance types, including... Source: about 1 year ago

• Questions for bioinformatics researchers that use AWS

  AWS Batch is the equivalent of a university cluster you submit to with slurm/sge/lsf/etc. But does not use those schedulers as AWS has their own. Source: about 1 year ago

• Scheduling "Fetch & Run" Batch Jobs with AWS Batch and CloudWatch Rules

  Developers frequently use batch computing to access significant amounts of processing power. You may perform batch computing workloads in the AWS Cloud with the aid of AWS Batch, a fully managed service provided by AWS. It is a powerful solution that can plan, schedule, and execute containerized batch or machine learning workloads across the entire spectrum of AWS compute capabilities, including Amazon ECS, Amazon... - Source: dev.to / about 1 year ago

• can you run OS applications in lambda layers?

  As others mentioned, you *can*. It might be easier with AWS Batch (https://aws.amazon.com/batch/) depending on what you're trying to do. Source: over 1 year ago

• A serverless architecture for high performance financial modelling

  I remember as part of the AWS certification exams this use was explicitly mentioned. https://aws.amazon.com/batch/. - Source: Hacker News / over 1 year ago

• Alternatives to EKS

  Https://aws.amazon.com/batch/ is the go-to for HPC, or maybe I'm misunderstanding your requirements. Source: almost 2 years ago

• Tips for scalable workflows on AWS

  Instead, break up the workflow - run each step on its own compute instance with resources right sized for each step. There are a lot of AWS instance types to choose from, and if you connect a workflow engine to AWS Batch, AWS Batch will manage picking the right one for you based on CPU and memory (and GPU) requirements. - Source: dev.to / almost 2 years ago

• Rust: CSV processing

  Another alternative is to use AWS Batch with spot instances in conjunction with AWS Step Functions leveraging the service integration Run a Job (.sync) pattern. After calling AWS Batch submitJob, the workflow pauses. When the job is complete, Step Functions progresses to the next state. - Source: dev.to / about 2 years ago

• Orchestration of processing in the cloud and locally

  Check out AWS Batch, it will provide the instances for you and remove them once the job is done. And if you're looking for a framework that allows you to quickly move back and forth between local and cloud, check out Ploomber. Source: over 2 years ago

• Does anyone know of good AWS resources or tutorials for bioinformatics

  I would suggest the AWS genomics CLI and AWS Batch as good features to learn. Both benefit from knowledge of nextflow so I would start there and work towards to the cloud. There's already a website with loads of common bioinformatics workflows already implemented. Source: over 2 years ago

• Launching VM instances only when needed

  If you're looking for something a bit more managed, check out Batch. It's basically a managed AWS service that does a lot of what I describe, kind of the same, but kind of differently. Batch has its own workflow peculiarities, but you may prefer dealing with those rather than dealing with something custom hacked together to behave like it. Source: over 2 years ago

• running python scripts parallelly on aws

  I think AWS batch is what you're looking for. Source: over 2 years ago

• Auto Scaling EC2?

  Sounds like Batch could be what you are after; if you can package your software into a docker container and separate the parts of the pipeline needing different resources into different jobs and run them in separate queues with appropriately sized instances in their compute environments. I’d explore wrapping up the jobs in a Nextflow script too. Source: about 3 years ago