Mage AI Reviews and details

Social recommendations and mentions

• Looking for an open-source project

  Want to work on Mage, a modern replacement for Airflow? Source: about 1 year ago

• Is there an alternative for Airflow for running thousands of dynamic tasks?

  You could also check mage. https://github.com/mage-ai/mage-ai It is developed by former engineers of AirBnb too. Source: over 1 year ago

• ETL tool

  Disclaimer: I worked at Airbnb for 5+ years working on data tools like Airflow and I helped start Mage almost 2 years ago. Source: over 1 year ago

• Can we take a moment to appreciate how much of dataengineering is open source?

  Here is an easy to use data pipeline tool (free) with a user friendly UI: https://github.com/mage-ai/mage-ai. Source: over 1 year ago

• Show HN: Mage, Fivetran alternative for ELT and data integrations

  You can also check out the repo here: https://github.com/mage-ai/mage-ai. - Source: Hacker News / over 1 year ago

• El tool for a growing startup Dec 22 edition!

  Mage (singer spec, OSS, modern Airflow replacement). Source: over 1 year ago

• How to deploy my pipelines…

  If you have your files written in Python, SQL, or R, I’d suggest using an easy to use orchestration tool like Mage; it’s a modern replacement for Airflow. Source: over 1 year ago

• After Airflow. Where next for DE?

  If you think Dagster is complicated, try checking out Mage; you build your pipeline’s dependencies by pressing buttons, but you write the body of your code for each step in Python, SQL, or R. Source: over 1 year ago

• After Airflow. Where next for DE?

  Thanks for giving a shout out to Mage.ai. We believe in high-code in the right places. Source: over 1 year ago

• Should i use airflow

  If you want something more modern and has an easier developer experience, try Mage (https://github.com/mage-ai/mage-ai). Source: over 1 year ago

• What is a personality type of a Data Engineer?

  Okay, I will explain what I am doing and how I see the "fun" in the project. I work with an open-source framework for data engineers. The community members are developers and people who use the tool - DEs. Indeed, I am facilitating a monthly community meeting for everyone to meet and discuss important topics, but that's the only part that takes their direct time, and it's totally voluntary, so DEs usually don't... Source: over 1 year ago

• Is it possible to process more than 1000 interfaces with airflow?

  If you just started and aren’t locked in, I’d recommend trying out a more modern/better data pipeline tool. Something like Mage (https://github.com/mage-ai/mage-ai). Source: over 1 year ago

• What are the essential components of a data architecture?

  I would do this: - Create a data pipeline for each report - Schedule each pipeline according to your desired schedule (every 2 weeks, every month) - Each data pipeline will have steps in it that load data from your database, another step to transform or aggregate the data, then a final step to export the data to a data warehouse for reading - Spin up a data warehouse in Snowflake so that your data pipeline can... Source: over 1 year ago

• Data pipeline CSV + RDBMS

  You can try out Mage (https://github.com/mage-ai/mage-ai). You can build the data pipeline you mention above with all those steps. Each step can depend on other steps. Within each of your step, you can write multiple data validation tests inline so that whenever each step runs, the data is validated and will throw an error and alert you. The UI has really good logging... Source: over 1 year ago

• How could I build an ETL pipeline, to trigger worker to transform data from data lake to data warehouse?

  With Mage (https://github.com/mage-ai/mage-ai), you can build data pipelines that can read from a Kafka topic. This is what I would do with Mage: 1. Data collected from API 2. Kafka publish to topic 3. Mage data pipeline constantly consumes events from the topic 4. Mage data pipeline has a block of code to write to NoSQL 5. Mage data pipeline has another block to transform that data 6. Once data is... Source: over 1 year ago

• [P] Are you having trouble building and managing complex data pipelines?

  After 5+ years of working on data at Airbnb and managing thousands of pipelines for Mage users, we open-sourced our data pipeline tool: https://github.com/mage-ai/mage-ai. Source: over 1 year ago

• Scheduling tool for API data collection and cleaning

  Try out this more modern data pipeline tool to schedule and orchestrate your data loading, transformation, and building training set: https://github.com/mage-ai/mage-ai. Source: over 1 year ago

• What aspects of Python should I learn that are most important for Data Engineering?

  Airflow: you can live without. Check out this newer data pipeline tool: https://github.com/mage-ai/mage-ai. Source: over 1 year ago

• When following this article https://analyticsmayhem.com/dbt/apache-airflow-dbt-docker-compose/ I can set up an image for dbt , but I am trying to scale this infrastructure for a team of 4 (including GitHub and circle ci)

  If you aren’t dead set on using Airflow and DBT, there is a dead simpler way to do what you are trying to accomplish with a single tool: https://github.com/mage-ai/mage-ai. Source: over 1 year ago

• What's the best way to ingest data from ftp in AWS?

  Or, use a data pipeline orchestration tool like Mage (https://github.com/mage-ai/mage-ai) to set up scripts to run every N minutes or trigger base off any event. Source: over 1 year ago

• What would be the best order to learn data engineering skills and technology for a beginner?

  If you want to try out building data pipelines on cloud resources, you can try out this tool: http://demo.mage.ai/. Here is the project: https://github.com/mage-ai/mage-ai. Source: over 1 year ago