Apache Spark Reviews

DuckDB Reviews

Apache Spark mentions (58)

• How I've implemented the Medallion architecture using Apache Spark and Apache Hdoop

  In this project, I'm exploring the Medallion Architecture which is a data design pattern that organizes data into different layers based on structure and/or quality. I'm creating a fictional scenario where a large enterprise that has several branches across the country. Each branch receives purchase orders from an app and deliver the goods to their customers. The enterprise wants to identify the branch that... - Source: dev.to / 4 days ago

• Shades of Open Source - Understanding The Many Meanings of "Open"

  In contrast, Databricks maintains internal forks of Spark, Delta Lake, and Unity Catalog, using the same names for both the open-source versions and the features specific to the Databricks platform. While they do provide separate documentation, online discussions often reflect confusion about how to use features in the open-source versions that only exist on the Databricks platform. This creates a "muddying of the... - Source: dev.to / 6 days ago

• Groovy 🎷 Cheat Sheet - 01 Say "Hello" from Groovy

  Recently I had to revisit the "JVM languages universe" again. Yes, language(s), plural! Java isn't the only language that uses the JVM. I previously used Scala, which is a JVM language, to use Apache Spark for Data Engineering workloads, but this is for another post 😉. - Source: dev.to / 4 months ago

• 🦿🛴Smarcity garbage reporting automation w/ ollama

  Consume data into third party software (then let Open Search or Apache Spark or Apache Pinot) for analysis/datascience, GIS systems (so you can put reports on a map) or any ticket management system. - Source: dev.to / 5 months ago

• Go concurrency simplified. Part 4: Post office as a data pipeline

  Also, this knowledge applies to learning more about data engineering, as this field of software engineering relies heavily on the event-driven approach via tools like Spark, Flink, Kafka, etc. - Source: dev.to / 6 months ago

DuckDB mentions (15)

• Big Data Is Dead

  I have lived through the hype of Big data it was a time of HDFS+HTable I guess and Hapoop etc. One can't go wrong with DuckDB+SQLite+Open/Elasticsearch either with 6 to 8 even 10 TB of data. [0]. https://duckdb.org/. - Source: Hacker News / 25 days ago

• Tactical Cloud Audit Log Analysis with DuckDB - AWS CloudTrail

  More than once, I have been in a situation where I needed to query CloudTrail logs but was working in a customer environment where they weren’t aggregated to a search interface. Another similar situation is when CloudTrail data events are disabled for cost reasons but need to be temporarily turned on for troubleshooting/audit purposes. While the CloudTrail console offers some (very) limited lookups (for management... - Source: dev.to / about 1 month ago

• How to choose the right type of database

  DuckDB: An in-process SQL OLAP database management system. While not a traditional OLAP database, DuckDB is designed to execute analytical queries efficiently, making it suitable for analytical workloads within data-intensive applications. - Source: dev.to / 4 months ago

• SIMD-Scan: Ultra Fast In-Memory Table Scan [pdf]

  Easiest way to practically use SIMD table scan database is try out DuckDB: https://duckdb.org/. - Source: Hacker News / 4 months ago

• 🎀 Domaine.nc data as Jupyter on Kaggle 📊

  Duckdb so we can make OLAP like queries on the data. - Source: dev.to / 7 months ago