Apache Spark Reviews and details

Social recommendations and mentions

• Groovy 🎷 Cheat Sheet - 01 Say "Hello" from Groovy

  Recently I had to revisit the "JVM languages universe" again. Yes, language(s), plural! Java isn't the only language that uses the JVM. I previously used Scala, which is a JVM language, to use Apache Spark for Data Engineering workloads, but this is for another post 😉. - Source: dev.to / 2 months ago

• 🦿🛴Smarcity garbage reporting automation w/ ollama

  Consume data into third party software (then let Open Search or Apache Spark or Apache Pinot) for analysis/datascience, GIS systems (so you can put reports on a map) or any ticket management system. - Source: dev.to / 3 months ago

• Go concurrency simplified. Part 4: Post office as a data pipeline

  Also, this knowledge applies to learning more about data engineering, as this field of software engineering relies heavily on the event-driven approach via tools like Spark, Flink, Kafka, etc. - Source: dev.to / 5 months ago

• Five Apache projects you probably didn't know about

  Apache SeaTunnel is a data integration platform that offers the three pillars of data pipelines: sources, transforms, and sinks. It offers an abstract API over three possible engines: the Zeta engine from SeaTunnel or a wrapper around Apache Spark or Apache Flink. Be careful, as each engine comes with its own set of features. - Source: dev.to / 5 months ago

• Spark – A micro framework for creating web applications in Kotlin and Java

  A JVM based framework named "Spark", when https://spark.apache.org exists? - Source: Hacker News / 11 months ago

• Rest in Peas: The Unrecognized Death of Speech Recognition (2010)

  You could of course search for yourself, but it's a python library[1] for interfacing with "Spark"[2], the Apache large scale data processing framework. [1] https://pypi.org/project/pyspark/ [2] https://spark.apache.org/. - Source: Hacker News / about 1 year ago

• Integrate Apache Spark and QuestDB for Time-Series Analytics

  Spark is an analytics engine for large-scale data engineering. Despite its long history, it still has its well-deserved place in the big data landscape. QuestDB, on the other hand, is a time-series database with a very high data ingestion rate. This means that Spark desperately needs data, a lot of it! ...and QuestDB has it, a match made in heaven. - Source: dev.to / about 1 year ago

• Query Real Time Data in Kafka Using SQL

  Additionally, one of the challenges of working with Kafka is how to efficiently analyze and extract insights from the large volumes of data stored in Kafka topics. Traditional batch processing approaches, such as Hadoop MapReduce or Apache Spark, can be slow and expensive, and may not be suitable for real-time analytics. To address this challenge, you can use SQL queries with Kafka to analyze and extract insights... - Source: dev.to / about 1 year ago

• Apache Iceberg as storage for on-premise data store (cluster)

  Spark for your transformation compute engine. Get Spark to talk to Nessie. Source: about 1 year ago

• 5 Best Practices For Data Integration To Boost ROI And Efficiency

  There are different ways to implement parallel dataflows, such as using parallel data processing frameworks like Apache Hadoop, Apache Spark, and Apache Flink, or using cloud-based services like Amazon EMR and Google Cloud Dataflow. It is also possible to use parallel dataflow frameworks to handle big data and distributed computing, like Apache Nifi and Apache Kafka. Source: about 1 year ago

• Beginner question about transformation

  You should also consider distribution of data because in a company that has machine learning workflows, the same data may need to go through different workflows using different technologies and stored in something other than a data warehouse, e.g. Feature engineering in Spark and loaded/stored in binary format such as Parquet in a data lake/object store. Source: about 1 year ago

• Patterns for work across computers

  Because I could talk about things like Apache Spark, but you can't properly understand what it's doing until you have the right foundation. Source: about 1 year ago

• Forward Compatible Enum Values in API with Java Jackson

  We’re not discussing the technical details behind the deduplication process. It could be Apache Flink, Apache Spark, or Kafka Streams. Anyway, it’s out of the scope of this article. - Source: dev.to / over 1 year ago

• Databricks explained for busy engineers | Databricks quick start | Databricks Data Security

  **Databricks **is built on top of Apache Spark, which provides a fast and general-purpose cluster-computing framework for big data processing. - Source: dev.to / over 1 year ago

• DataOps 101: An Introduction to the Essential Approach of Data Management Operations and Observability

  DataOps is a collaborative effort within an organization, with many different teams of people working together to ensure that DataOps functions properly and delivers data value [3]. So, before the data is delivered to end users, it is subjected to a number of treatments and refinements from multiple teams. Data scientists first use their data science techniques, such as machine learning and deep learning to build... - Source: dev.to / over 1 year ago

• Apache Pulsar vs Apache Kafka - How to choose a data streaming platform

  Both Kafka and Pulsar provide some kind of stream processing capability, but Kafka is much further along in that regard. Pulsar stream processing relies on the Pulsar Functions interface which is only suited for simple callbacks. On the other hand, Kafka Streams and ksqlDB are more complete solutions that could be considered replacements for Apache Spark or Apache Flink, state-of-the-art stream-processing... - Source: dev.to / over 1 year ago

• What is the separation of storage and compute in data platforms and why does it matter?

  However, once your data reaches a certain size or you reach the limits of vertical scaling, it may be necessary to distribute your queries across a cluster, or scale horizontally. This is where distributed query engines like Trino and Spark come in. Distributed query engines make use of a coordinator to plan the query and multiple worker nodes to execute them in parallel. - Source: dev.to / over 1 year ago

• Deequ for generating data quality reports

  Aws documentation — Deequ allows you to calculate data quality metrics on your dataset, define and verify data quality constraints, and be informed about changes in the data distribution. Instead of implementing checks and verification algorithms on your own, you can focus on describing how your data should look. Deequ supports you by suggesting checks for you. Deequ is implemented on top of Apache Spark and is... - Source: dev.to / over 1 year ago

• In One Minute : Hadoop

  Spark, a fast and general engine for large-scale data processing. - Source: dev.to / over 1 year ago

• Machine Learning Pipelines with Spark: Introductory Guide (Part 1)

  Apache Spark is a fast and general open-source engine for large-scale, distributed data processing. Its flexible in-memory framework allows it to handle batch and real-time analytics alongside distributed data processing. - Source: dev.to / over 1 year ago

• A peek into Location Data Science at Ola

  This requires the use of distributed computation tools such as Spark and Hadoop, Flink and Kafka are used. But for occasional experimentation, Pandas, Geopandas and Dask are some of the commonly used tools. - Source: dev.to / over 1 year ago