Materialize Reviews and Details

Is Materialize good?

Social recommendations and mentions

• Materialized views are obviously useful

  Did I miss in the article where OP reveals the magic database that actually does this? 3rd party solutions like https://readyset.io/ and https://materialize.com/ exist specifically because databases don’t actually have what we all want materialized views to be. - Source: Hacker News / 11 months ago

• The Missing Manual for Signals: State Management for Python Developers

  This triggered some associations for me. Strongest was Cells[0], a library for Common Lisp CLOS. The earliest reference I can find is 2002[1], making it over 20 years old. Second is incremental view maintenance systems like Feldera[2] or Materialize[3]. These use sophisticated theories (z-sets and differential dataflow) to apply efficient updates over sets of data, which generalizes the case of single variables.... - Source: Hacker News / about 1 year ago

• Category Theory in Programming

  It's hard to write something that is both accessible and well-motivated. The best uses of category theory is when the morphisms are far more exotic than "regular functions". E.g. It would be nice to describe a circuit of live queries (like https://materialize.com/ stuff) with proper caching, joins, etc. Figuring this out is a bit of an open problem. Haskell's standard library's Monad and stuff are watered down to... - Source: Hacker News / over 1 year ago

• Building Databases over a Weekend

  > [...] `https://materialize.com/` to solve their memory issues [...] Disclaimer: I work at Materialize Recently there have been major improvements in Materialize's memory usage as well as using disk to swap out some data. I find it pretty easy to hook up to Postgres/MySQL/Kafka instances: https://materialize.com/blog/materialize-emulator/. - Source: Hacker News / over 1 year ago

• Building Databases over a Weekend

  I agree. So many disparate solutions. The streaming sql primitives are by themselves good enough (e.g. `tumble`, `hop` or `session` windows), but the infrastructural components are always rough in real life use cases. Crossing fingers for solutions like `https://github.com/feldera/feldera` to solve their memory issues, or `https://clickhouse.com/docs/en/materialized-view` to solve reliable streaming consumption.... - Source: Hacker News / over 1 year ago

• Drasi: Microsoft's open source data processing platform for event-driven systems

  Or the related Materialize stuff https://materialize.com/. - Source: Hacker News / over 1 year ago

• Rama on Clojure's terms, and the magic of continuation-passing style

  The original post makes so much more sense in this context! One of the "holy grails" in my mind is making CQRS and dataflow programming as easy to learn and maintain as existing imperative programming languages - and easy to weave into real-time UX. There are so many backend endpoints in the wild that do a bunch of things in a loop, many of which will require I/O or calls to slow external endpoints, transform the... - Source: Hacker News / almost 2 years ago

• Feldera Incremental Compute Engine

  This looks extremely cool. This is basically incremental view maintenance in databases, a problem that almost everybody (I think) has when using SQL databases and wanting to do some derived views for more performant access patterns. Importantly, they seem to support a wide breath of SQL operators, and it's open-source! There's already a bunch of tools in this area: 1. Materialize[0], which afaik is more... - Source: Hacker News / almost 2 years ago

• Ask HN: Who is hiring? (July 2024)

  Materialize | https://materialize.com/ | Staff Security Engineer 200k | $200- 230k NYC (HQ) or United States We are looking for a Staff Security Engineer on our Cloud team who will own the security of our infrastructure and product. (5+) years of experience as a security-focused engineer. https://boards.greenhouse.io/materialize/jobs/5220351004. - Source: Hacker News / about 2 years ago

• Choosing Between a Streaming Database and a Stream Processing Framework in Python

  To fully leverage the data is the new oil concept, companies require a special database designed to manage vast amounts of data instantly. This need has led to different database forms, including NoSQL databases, vector databases, time-series databases, graph databases, in-memory databases, and in-memory data grids. Recent years have seen the rise of cloud-based streaming databases such as RisingWave, Materialize,... - Source: dev.to / over 2 years ago

• We Built a Streaming SQL Engine

  Some recent solutions to this problem include Differential Dataflow and Materialize. It would be neat if postgres adopted something similar for live-updating materialized views. https://github.com/timelydataflow/differential-dataflow. - Source: Hacker News / over 2 years ago

• Ask HN: Who is hiring? (October 2023)

  Materialize | Full-Time | NYC Office or Remote | https://materialize.com Materialize is an Operational Data Warehouse: A cloud data warehouse with streaming internals, built for work that needs action on what’s happening right now. Keep the familiar SQL, keep the proven architecture of cloud warehouses but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that... - Source: Hacker News / almost 3 years ago

• Ask HN: Who is hiring? (June 2023)

  Materialize | EM (Compute), Senior PM | New York, New York | https://materialize.com/ You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. That is Materialize, the only true SQL... - Source: Hacker News / about 3 years ago

• Ask HN: Who is hiring? (April 2023)

  Materialize | NY, NY | https://materialize.com/ The Cloud Database for Fast-Changing Data. We put a streaming engine in a database, so your team can build real-time data products without the cost, complexity, and development time of stream processing. Cloud team openings: https://grnh.se/0ad6ab6b4us Senior PM openings: https://grnh.se/415c267f4us. - Source: Hacker News / over 3 years ago

• Query Real Time Data in Kafka Using SQL

  Most streaming database technologies use SQL for these reasons: RisingWave, Materialize, KsqlDB, Apache Flink, and so on offering SQL interfaces. This post explains how to choose the right streaming database. - Source: dev.to / over 3 years ago

• Why Are There No Relational DBMSs? [pdf]

  The relational model (and generally working at the level of sets/collections, instead of the level of individual values/objects) actually makes it easier to have this kind of incremental computation in a consistent way, I think. There's a bunch of work being done on making relational systems work this way. Some interesting reading: -... - Source: Hacker News / over 3 years ago

• Kafka Stream Processing in Java or Scala

  If you want to keep in your Python/SQL area of expertise and by all means I don't mean to promote not learning a new language, but just as an FYI. There are some non-Java/Scala tools between streaming databases like risingwave and materialize, streaming platforms like fluvio and redpanda, and stream processors like bytewax and faust. Source: over 3 years ago

• What makes a time series oriented database (ex: QuestDB) more efficient for OLAP on time series than an OLAP "only" oriented database (ex: DuckDB) technically?

  AFAIK there is a lot of overlap between OLAP databases and time series databases. Timescale](https://legacy-docs.timescale.com/v1.7/introduction/architecture) gains a lot of its performance via the "Hypertable" abstraction which is fairly similar to something like Parquet partitioning/bucketing. In terms of performance I don't know if there is a huge gap either for non optimized use cases. The [Clickhouse] team... Source: over 3 years ago

• How to handle partial updates and bulk updates in the source systems

  Imo this is a matter of schema design. You shouldn't have to send the entire object in the event, just the delta. If you are using an event based schema, ideally you should be able to generate the current state by iterating over all the events and combining the deltas. An OLAP database/warehouse/lakehouse can be very efficient at this depending on how the data is partitioned. You could consider a Materialized... Source: over 3 years ago

• Materialized View: SQL Queries on Steroids

  Projects like https://readyset.io/, https://materialize.com/, https://github.com/mit-pdos/noria can keep your materialized views up to date as the underlying base tables change. - Source: Hacker News / over 3 years ago

• Real Time Data Infra Stack

  Even though these technical stacks are listed by category, some fields actually overlap. For example, although Materialize is classified as a stream processor, it makes sense to treat as a serving layer because it is essentially a streaming database, and the same is true for ksqlDB. - Source: dev.to / over 3 years ago