Python & ETL 2020: A List and Comparison of the Top Python ETL Tools

1

Apache Airflow

When does Apache Airflow make sense? If you're performing long ETL jobs or your ETL has multiple steps, Airflow will let you restart from any point during the ETL process. That being said, Apache Airflows IS NOT a library, so it has to be deployed and may make less sense on small ETL jobs.

#Workflows #Workflow Automation #Data Pipelines 65 social mentions

2

Luigi

When does Luigi make sense? If you need to automate simple ETL processes (like logs) Luigi can handle them rapidly and without much setup. When it comes to complex tasks, Luigi is limited by its strict pipeline-like structure.

#DevOps Tools #Workflow Automation #Workflows 9 social mentions

3

Pandas

When it comes to ETL, you can do almost anything with Pandas if you're willing to put in the time. Plus, pandas is extraordinarily easy to run. You can set up a simple script to load data from a Postgre table, transform and clean that data, and then write that data to another Postgre table.

#Data Science And Machine Learning #Data Science Tools #Python Tools 196 social mentions

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

6

Bubbles

Bubbles is another Python framework that you can use to run ETL. Unlike some other ETL frameworks, Bubbles uses metadata to describe pipelines as opposed to script-based. While Bubbles is written for Python, the author claims that it's not meant to be Python-exclusive in nature. A driving theme of Bubbles is that it's technologically agnostic, so you don't have to worry about working with or accessing data — just the transformation.

#Chrome Extensions #Screen Recording #Video Maker 1 social mentions

BeautifulSoup: This Python tool pulls data out of webpages (XML, HTML). Has integrations with tons of ETL tools like petl.

#Web Scraping #Data Extraction #Data

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

9

Blaze

Blaze: This is an interface that queries data. Also, this is part of the "Blaze Ecosystem" which is a framework for an ETL process using Blaze, Dask, Datashape, DyND, and Odo.

#Apple Watch #Design Tools #Sales

10

Dask

Dask: You can use Dask for Parallel computing via task scheduling. It can also process continuous data streams. Again, this is part of the "Blaze Ecosystem."

#Workflows #Databases #Software Development 16 social mentions

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Odo: This lets you move data between multiple containers. Odo lets you use the native CSV loading capabilities of SQL databases, which is faster than trying to load with Python.

#Productivity #AI #Writing Tools

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

18

Xplenty

Customer Story Keith connected multiple data sources with Amazon Redshift to transform, organize and analyze their customer data. Amazon Redshift Keith Slater Senior Developer at Creative Anvil Before we started with Xplenty, we were trying to move data from many different data sources into Redshift. Xplenty has helped us do that quickly and easily. The best feature of the platform is having the ability to manipulate data as needed without the process being overly complex. Also, the support is great - they’re always responsive and willing to help. FIND OUT IF WE CAN INTEGRATE YOUR DATA TRUSTED BY COMPANIES WORLDWIDE

#ETL #Data Integration #Monitoring Tools

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

20

AWS Glue

#Data Integration #ETL #Data Workflow 13 social mentions

21

AWS Batch

AWS Batch: This is used for batch computing jobs on AWS resources. It has insane scalability and is well-suited for engineers look to do large compute jobs.

#Cloud Computing #Cloud Hosting #Development 14 social mentions

22

Google Cloud Dataflow

#Big Data #Data Dashboard #Data Warehousing 14 social mentions

23

Azure Data Factory

#ETL #Data Integration #Workflow Automation 3 social mentions

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

25

Pachyderm

Pachyderm: This is another great alternative to tools like Airflow. Here's a great GitHub writeup about some of the simple differences between Airflow and Pachyderm. Note: Paychyderm has an open-source edition on their website.

#Data Science And Machine Learning #Data Science Notebooks #Machine Learning Tools 1 social mentions

26

Mara

Mara: This is another ETL framework for Python. It's a middle-ground between pure Python and Apache Airflow, so it's fast and simple to set up.

#ETL #Data Integration #Data Pipelines

27

Pinball

Pinball: This is Pinterest's workflow manager. It has auto-retries, priorities, overrun policies, and tons of horizontal scalability.

#Tech #ETL #Web Service Automation

28

Azkaban

#Workflow Automation #DevOps Tools #Automation 3 social mentions

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

30

Spark Mail

Spark: This is a full-blown toolkit that has tons of helpful tools. With Spark Streaming you can set up your entire batch streaming ETL.

#Email #Email Clients #Calendar 30 social mentions

31

Kettle Pentaho

#Data Integration #Workflows #ETL

#Data Integration #ETL #Stream Processing

33

apatar

#Data Integration #ETL #Monitoring Tools

34

Jaspersoft

#Data Dashboard #Business Intelligence #Data Visualization

35

Spring Batch

#Workflow Automation #Databases #Data Dashboard 2 social mentions

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

39

Crunch

#Health & Wellness #Health And Fitness #Gym And Fitness Studio Management

40

NoFlo

#Business Intelligence #Automation #Data Dashboard 2 social mentions

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.

Dray.it: This is a Docker workflow engine that helps tremendously with resource management.