Automate your data processing pipeline in 9 steps ⚙️

1

Google Cloud Platform

Before configuring the node, you have to sign up on the Google Cloud Platform to enable the API and get the necessary credentials (Client ID and Client Secret). Follow the instructions in our reference docs to set up your account and the node credentials.

#Cloud Computing #Backend As A Service #Cloud Infrastructure 168 social mentions

2

Apache Airflow

I was really excited, though also a bit overwhelmed by all the things I had to set up for this project. In total, I spent five days learning the tools, debugging, and building this pipeline with Python (including libraries like Tweepy, TextBlob, VADER, and SQLAlchemy), Postgres, MongoDB, Docker, and Airflow (most frustrating part...). If you're interested to see how I did this, you can check out the project on GitHub and read this blog post.

#Workflows #Workflow Automation #Data Pipelines 65 social mentions