Materialize Reviews

Apache Spark Reviews

Materialize mentions (72)

• Category Theory in Programming

  It's hard to write something that is both accessible and well-motivated. The best uses of category theory is when the morphisms are far more exotic than "regular functions". E.g. It would be nice to describe a circuit of live queries (like https://materialize.com/ stuff) with proper caching, joins, etc. Figuring this out is a bit of an open problem. Haskell's standard library's Monad and stuff are watered down to... - Source: Hacker News / 5 months ago

• Building Databases over a Weekend

  > [...] `https://materialize.com/` to solve their memory issues [...] Disclaimer: I work at Materialize Recently there have been major improvements in Materialize's memory usage as well as using disk to swap out some data. I find it pretty easy to hook up to Postgres/MySQL/Kafka instances: https://materialize.com/blog/materialize-emulator/. - Source: Hacker News / 5 months ago

• Building Databases over a Weekend

  I agree. So many disparate solutions. The streaming sql primitives are by themselves good enough (e.g. `tumble`, `hop` or `session` windows), but the infrastructural components are always rough in real life use cases. Crossing fingers for solutions like `https://github.com/feldera/feldera` to solve their memory issues, or `https://clickhouse.com/docs/en/materialized-view` to solve reliable streaming consumption.... - Source: Hacker News / 5 months ago

• Drasi: Microsoft's open source data processing platform for event-driven systems

  Or the related Materialize stuff https://materialize.com/. - Source: Hacker News / 7 months ago

• Rama on Clojure's terms, and the magic of continuation-passing style

  The original post makes so much more sense in this context! One of the "holy grails" in my mind is making CQRS and dataflow programming as easy to learn and maintain as existing imperative programming languages - and easy to weave into real-time UX. There are so many backend endpoints in the wild that do a bunch of things in a loop, many of which will require I/O or calls to slow external endpoints, transform the... - Source: Hacker News / 7 months ago

Apache Spark mentions (70)

• Every Database Will Support Iceberg — Here's Why

  Apache Iceberg defines a table format that separates how data is stored from how data is queried. Any engine that implements the Iceberg integration — Spark, Flink, Trino, DuckDB, Snowflake, RisingWave — can read and/or write Iceberg data directly. - Source: dev.to / 11 days ago

• How to Reduce Big Data Analytics Costs by 90% with Karpenter and Spark

  Apache Spark powers large-scale data analytics and machine learning, but as workloads grow exponentially, traditional static resource allocation leads to 30–50% resource waste due to idle Executors and suboptimal instance selection. - Source: dev.to / 13 days ago

• Unveiling the Apache License 2.0: A Deep Dive into Open Source Freedom

  One of the key attributes of Apache License 2.0 is its flexible nature. Permitting use in both proprietary and open source environments, it has become the go-to choice for innovative projects ranging from the Apache HTTP Server to large-scale initiatives like Apache Spark and Hadoop. This flexibility is not solely legal; it is also philosophical. The license is designed to encourage transparency and maintain a... - Source: dev.to / about 2 months ago

• The Application of Java Programming In Data Analysis and Artificial Intelligence

  [1] S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Pearson, 2020. [2] F. Chollet, Deep Learning with Python. Manning Publications, 2018. [3] C. C. Aggarwal, Data Mining: The Textbook. Springer, 2015. [4] J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Communications of the ACM, vol. 51, no. 1, pp. 107-113, 2008. [5] Apache Software Foundation, "Apache... - Source: dev.to / about 2 months ago

• Automating Enhanced Due Diligence in Regulated Applications

  If you're designing an event-based pipeline, you can use a data streaming tool like Kafka to process data as it's collected by the pipeline. For a setup that already has data stored, you can use tools like Apache Spark to batch process and clean it before moving ahead with the pipeline. - Source: dev.to / 3 months ago