how to get into data eng pt.2

1

Apache Superset

There are other ways for people to consume data, one of the major ones (maybe the major one?) is dashboards. You're going to have to give people some dashboarding solution because they want to fill the world with pie charts and you don't have the power to stop them. I've not worked with it much, but Superset seems alright.

#Data Analysis And Visualization #Business Intelligence #Data Dashboard 51 social mentions

2

Apache Kafka

Now you have a choice, either do something with the data (ie move on to transformation) or make this a better solution. If you want to make this a better solution you need to understand what's bad about it currently, two things spring to mind, the first being the potential for data loss, and the second being the poor output format (single record per file). So really we're talking about getting a queue involved here, probably Kafka (which you'll have to figure out how to deploy).

#Stream Processing #Data Integration #ETL 120 social mentions

3

Apache Hive

How to flesh this idea out more? Start ingesting different types of events and putting them in different tables. Definitely try an OLAP solution (Hive? Most of my experience sits with closed source projects that only a company can afford) and think about column optimised file formats. Get the raw data into an OLAP solution and do aggregations into another systems based off of the OLAP solution.

#Databases #Big Data #Data Warehousing 8 social mentions

4

aviationstack

2- if you want real time data you could use https://aviationstack.com/.

#Aviation #Flight API #Flights API 4 social mentions