Apache Spark Reviews and details

Social recommendations and mentions

• Every Database Will Support Iceberg — Here's Why

  Apache Iceberg defines a table format that separates how data is stored from how data is queried. Any engine that implements the Iceberg integration — Spark, Flink, Trino, DuckDB, Snowflake, RisingWave — can read and/or write Iceberg data directly. - Source: dev.to / 10 days ago

• How to Reduce Big Data Analytics Costs by 90% with Karpenter and Spark

  Apache Spark powers large-scale data analytics and machine learning, but as workloads grow exponentially, traditional static resource allocation leads to 30–50% resource waste due to idle Executors and suboptimal instance selection. - Source: dev.to / 11 days ago

• Unveiling the Apache License 2.0: A Deep Dive into Open Source Freedom

  One of the key attributes of Apache License 2.0 is its flexible nature. Permitting use in both proprietary and open source environments, it has become the go-to choice for innovative projects ranging from the Apache HTTP Server to large-scale initiatives like Apache Spark and Hadoop. This flexibility is not solely legal; it is also philosophical. The license is designed to encourage transparency and maintain a... - Source: dev.to / about 2 months ago

• The Application of Java Programming In Data Analysis and Artificial Intelligence

  [1] S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Pearson, 2020. [2] F. Chollet, Deep Learning with Python. Manning Publications, 2018. [3] C. C. Aggarwal, Data Mining: The Textbook. Springer, 2015. [4] J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Communications of the ACM, vol. 51, no. 1, pp. 107-113, 2008. [5] Apache Software Foundation, "Apache... - Source: dev.to / about 2 months ago

• Automating Enhanced Due Diligence in Regulated Applications

  If you're designing an event-based pipeline, you can use a data streaming tool like Kafka to process data as it's collected by the pipeline. For a setup that already has data stored, you can use tools like Apache Spark to batch process and clean it before moving ahead with the pipeline. - Source: dev.to / 3 months ago

• Run PySpark Local Python Windows Notebook

  PySpark is the Python API for Apache Spark, an open-source distributed computing system that enables fast, scalable data processing. PySpark allows Python developers to leverage the powerful capabilities of Spark for big data analytics, machine learning, and data engineering tasks without needing to delve into the complexities of Java or Scala. - Source: dev.to / 3 months ago

• How to Install PySpark on Your Local Machine

  If you’re stepping into the world of Big Data, you have likely heard of Apache Spark, a powerful distributed computing system. PySpark, the Python library for Apache Spark, is a favorite among data enthusiasts for its combination of speed, scalability, and ease of use. But setting it up on your local machine can feel a bit intimidating at first. - Source: dev.to / 5 months ago

• How to Use PySpark for Machine Learning

  According to the Apache Spark official website, PySpark lets you utilize the combined strengths of ApacheSpark (simplicity, speed, scalability, versatility) and Python (rich ecosystem, matured libraries, simplicity) for “data engineering, data science, and machine learning on single-node machines or clusters.”. - Source: dev.to / 5 months ago

• Why Apache Spark RDD is immutable?

  Apache Spark is a powerful and widely used framework for distributed data processing, beloved for its efficiency and scalability. At the heart of Spark’s magic lies the RDD, an abstraction that’s more than just a mere data collection. In this blog post, we’ll explore why RDDs are immutable and the benefits this immutability provides in the context of Apache Spark. - Source: dev.to / 7 months ago

• Intro to Ray on GKE

  The Python Library components of Ray could be considered analogous to solutions like numpy, scipy, and pandas (which is most analogous to the Ray Data library specifically). As a framework and distributed computing solution, Ray could be used in place of a tool like Apache Spark or Python Dask. It’s also worthwhile to note that Ray Clusters can be used as a distributed computing solution within Kubernetes, as... - Source: dev.to / 8 months ago

• Avoid These Top 10 Mistakes When Using Apache Spark

  We all know how easy it is to overlook small parts of our code, especially when we have powerful tools like Apache Spark to handle the heavy lifting. Spark's core engine is great at optimizing our messy, complex code into a sleek, efficient physical plan. But here's the catch: Spark isn't flawless. It's on a journey to perfection, sure, but it still has its limits. And Spark is upfront about those limitations,... - Source: dev.to / 8 months ago

• IaaS vs PaaS vs SaaS: The Key Differences

  One specific use case of the IaaS model is for deploying software that would have otherwise been bought as a SaaS. There are many such software from email servers to databases. You can choose to deploy MySQL in your infrastructure rather than buying from a MySQL SaaS provider. Other things you can deploy using the IaaS model include Mattermost for team collaboration, Apache Spark for data analytics, and SAP for... - Source: dev.to / 10 months ago

• How I've implemented the Medallion architecture using Apache Spark and Apache Hdoop

  In this project, I'm exploring the Medallion Architecture which is a data design pattern that organizes data into different layers based on structure and/or quality. I'm creating a fictional scenario where a large enterprise that has several branches across the country. Each branch receives purchase orders from an app and deliver the goods to their customers. The enterprise wants to identify the branch that... - Source: dev.to / 11 months ago

• Shades of Open Source - Understanding The Many Meanings of "Open"

  In contrast, Databricks maintains internal forks of Spark, Delta Lake, and Unity Catalog, using the same names for both the open-source versions and the features specific to the Databricks platform. While they do provide separate documentation, online discussions often reflect confusion about how to use features in the open-source versions that only exist on the Databricks platform. This creates a "muddying of the... - Source: dev.to / 11 months ago

• Groovy 🎷 Cheat Sheet - 01 Say "Hello" from Groovy

  Recently I had to revisit the "JVM languages universe" again. Yes, language(s), plural! Java isn't the only language that uses the JVM. I previously used Scala, which is a JVM language, to use Apache Spark for Data Engineering workloads, but this is for another post 😉. - Source: dev.to / about 1 year ago

• 🦿🛴Smarcity garbage reporting automation w/ ollama

  Consume data into third party software (then let Open Search or Apache Spark or Apache Pinot) for analysis/datascience, GIS systems (so you can put reports on a map) or any ticket management system. - Source: dev.to / over 1 year ago

• Go concurrency simplified. Part 4: Post office as a data pipeline

  Also, this knowledge applies to learning more about data engineering, as this field of software engineering relies heavily on the event-driven approach via tools like Spark, Flink, Kafka, etc. - Source: dev.to / over 1 year ago

• Five Apache projects you probably didn't know about

  Apache SeaTunnel is a data integration platform that offers the three pillars of data pipelines: sources, transforms, and sinks. It offers an abstract API over three possible engines: the Zeta engine from SeaTunnel or a wrapper around Apache Spark or Apache Flink. Be careful, as each engine comes with its own set of features. - Source: dev.to / over 1 year ago

• Spark – A micro framework for creating web applications in Kotlin and Java

  A JVM based framework named "Spark", when https://spark.apache.org exists? - Source: Hacker News / almost 2 years ago

• Rest in Peas: The Unrecognized Death of Speech Recognition (2010)

  You could of course search for yourself, but it's a python library[1] for interfacing with "Spark"[2], the Apache large scale data processing framework. [1] https://pypi.org/project/pyspark/ [2] https://spark.apache.org/. - Source: Hacker News / almost 2 years ago

• Integrate Apache Spark and QuestDB for Time-Series Analytics

  Spark is an analytics engine for large-scale data engineering. Despite its long history, it still has its well-deserved place in the big data landscape. QuestDB, on the other hand, is a time-series database with a very high data ingestion rate. This means that Spark desperately needs data, a lot of it! ...and QuestDB has it, a match made in heaven. - Source: dev.to / about 2 years ago