Mage AI Reviews and details

Social recommendations and mentions

• Looking for an open-source project

  Want to work on Mage, a modern replacement for Airflow? Source: almost 2 years ago

• Is there an alternative for Airflow for running thousands of dynamic tasks?

  You could also check mage. https://github.com/mage-ai/mage-ai It is developed by former engineers of AirBnb too. Source: almost 2 years ago

• ETL tool

  Disclaimer: I worked at Airbnb for 5+ years working on data tools like Airflow and I helped start Mage almost 2 years ago. Source: about 2 years ago

• Can we take a moment to appreciate how much of dataengineering is open source?

  Here is an easy to use data pipeline tool (free) with a user friendly UI: https://github.com/mage-ai/mage-ai. Source: about 2 years ago

• Show HN: Mage, Fivetran alternative for ELT and data integrations

  You can also check out the repo here: https://github.com/mage-ai/mage-ai. - Source: Hacker News / about 2 years ago

• El tool for a growing startup Dec 22 edition!

  Mage (singer spec, OSS, modern Airflow replacement). Source: about 2 years ago

• How to deploy my pipelines…

  If you have your files written in Python, SQL, or R, I’d suggest using an easy to use orchestration tool like Mage; it’s a modern replacement for Airflow. Source: about 2 years ago

• After Airflow. Where next for DE?

  If you think Dagster is complicated, try checking out Mage; you build your pipeline’s dependencies by pressing buttons, but you write the body of your code for each step in Python, SQL, or R. Source: about 2 years ago

• After Airflow. Where next for DE?

  Thanks for giving a shout out to Mage.ai. We believe in high-code in the right places. Source: about 2 years ago

• Should i use airflow

  If you want something more modern and has an easier developer experience, try Mage (https://github.com/mage-ai/mage-ai). Source: about 2 years ago

• What is a personality type of a Data Engineer?

  Okay, I will explain what I am doing and how I see the "fun" in the project. I work with an open-source framework for data engineers. The community members are developers and people who use the tool - DEs. Indeed, I am facilitating a monthly community meeting for everyone to meet and discuss important topics, but that's the only part that takes their direct time, and it's totally voluntary, so DEs usually don't... Source: about 2 years ago

• Is it possible to process more than 1000 interfaces with airflow?

  If you just started and aren’t locked in, I’d recommend trying out a more modern/better data pipeline tool. Something like Mage (https://github.com/mage-ai/mage-ai). Source: about 2 years ago

• What are the essential components of a data architecture?

  I would do this: - Create a data pipeline for each report - Schedule each pipeline according to your desired schedule (every 2 weeks, every month) - Each data pipeline will have steps in it that load data from your database, another step to transform or aggregate the data, then a final step to export the data to a data warehouse for reading - Spin up a data warehouse in Snowflake so that your data pipeline can... Source: about 2 years ago

• Data pipeline CSV + RDBMS

  You can try out Mage (https://github.com/mage-ai/mage-ai). You can build the data pipeline you mention above with all those steps. Each step can depend on other steps. Within each of your step, you can write multiple data validation tests inline so that whenever each step runs, the data is validated and will throw an error and alert you. The UI has really good logging... Source: about 2 years ago

• How could I build an ETL pipeline, to trigger worker to transform data from data lake to data warehouse?

  With Mage (https://github.com/mage-ai/mage-ai), you can build data pipelines that can read from a Kafka topic. This is what I would do with Mage: 1. Data collected from API 2. Kafka publish to topic 3. Mage data pipeline constantly consumes events from the topic 4. Mage data pipeline has a block of code to write to NoSQL 5. Mage data pipeline has another block to transform that data 6. Once data is... Source: about 2 years ago

• [P] Are you having trouble building and managing complex data pipelines?

  After 5+ years of working on data at Airbnb and managing thousands of pipelines for Mage users, we open-sourced our data pipeline tool: https://github.com/mage-ai/mage-ai. Source: about 2 years ago

• Scheduling tool for API data collection and cleaning

  Try out this more modern data pipeline tool to schedule and orchestrate your data loading, transformation, and building training set: https://github.com/mage-ai/mage-ai. Source: about 2 years ago

• What aspects of Python should I learn that are most important for Data Engineering?

  Airflow: you can live without. Check out this newer data pipeline tool: https://github.com/mage-ai/mage-ai. Source: about 2 years ago

• When following this article https://analyticsmayhem.com/dbt/apache-airflow-dbt-docker-compose/ I can set up an image for dbt , but I am trying to scale this infrastructure for a team of 4 (including GitHub and circle ci)

  If you aren’t dead set on using Airflow and DBT, there is a dead simpler way to do what you are trying to accomplish with a single tool: https://github.com/mage-ai/mage-ai. Source: about 2 years ago

• What's the best way to ingest data from ftp in AWS?

  Or, use a data pipeline orchestration tool like Mage (https://github.com/mage-ai/mage-ai) to set up scripts to run every N minutes or trigger base off any event. Source: about 2 years ago

• What would be the best order to learn data engineering skills and technology for a beginner?

  If you want to try out building data pipelines on cloud resources, you can try out this tool: http://demo.mage.ai/. Here is the project: https://github.com/mage-ai/mage-ai. Source: about 2 years ago