Apache Spark, Hive, and Spring Boot — Testing Guide

Databases Big Data Big Data Analytics

Apache Spark Landing Page
1

Apache Spark

Apache Spark is an engine for big data processing, with built-in modules for streaming, SQL, machine learning and graph processing.
Pricing:
- Open Source
In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

#Databases #Big Data #Big Data Analytics 56 social mentions
Apache Hive Landing Page
2

Apache Hive

Apache Hive data warehouse software facilitates querying and managing large datasets residing in distributed storage.
Pricing:
- Open Source
In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

#Databases #Big Data #Data Warehousing 8 social mentions
Apache Derby Landing Page
3

Apache Derby

Relational Databases
Pricing:
- Open Source
And finally, here comes test scoped artefacts. Apache Spark ones are included as testImplementation. Because integration tests will start the local Spark node. So, they are required during the runtime. The slf4j-api is also the runtime dependency. Testcontainers will be used to run the Aerospike instance. The janino is required by Apache Spark during the job execution. And we need Apache Derby to tune Apache Hive for local running. We'll get to this point soon.

#Databases #Relational Databases #Tool 3 social mentions
Aerospike Landing Page
4

Aerospike

Aerospike is a high-performing NoSQL database supporting high transaction volumes with low latency.
Pricing:
- Open Source
In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

#Databases #NoSQL Databases #Key-Value Database 8 social mentions