Pig, a platform/programming language for authoring parallelizable jobs. - Source: dev.to / over 1 year ago

In the early days of the Big Data era when K8s hasn't even been born yet, the common open source go-to solution was the Hadoop stack. We have written several old-fashioned Map-Reduce jobs, scripts using Pig until we came across Spark. Since then Spark has became one of the most popular data processing engines. It is very easy to start using Lighter on YARN deployments. Just run a docker with proper configuration... - Source: dev.to / over 2 years ago

As far as keeping track of domain changes you can store DDL files in version control like you mention or use tools like Flyway (https://flywaydb.org) or Liquidbase (https://liquibase.org) which takes care of database migrations. Source: about 2 years ago

I just use SQL directly (or something like JOOQ). For database migrations I use Liquibase. Source: about 2 years ago

Regarding the migrations, there are tools such as https://liquibase.org/ or FlyAway that handle this. Heck, you can even use an ORM that has a migration baked-in but that defeats the purpose of having the migrations in a separate project. Source: about 2 years ago

I've trialled schemachange and liquibase which are change script based tools. I've ruled out a whole load of other tools that are either change script based tools or don't support Snowflake, including the following:. Source: over 2 years ago

Nowadays I prefer to automate database updates and deployment, using Liquibase and its relational database vendor agnostic syntax for that. Especially on production systems. But on local dev environments, I can still use the occasional SQL in a pinch. Source: over 2 years ago