Materialize Reviews and details

Social recommendations and mentions

• Choosing Between a Streaming Database and a Stream Processing Framework in Python

  To fully leverage the data is the new oil concept, companies require a special database designed to manage vast amounts of data instantly. This need has led to different database forms, including NoSQL databases, vector databases, time-series databases, graph databases, in-memory databases, and in-memory data grids. Recent years have seen the rise of cloud-based streaming databases such as RisingWave, Materialize,... - Source: dev.to / 3 months ago

• We Built a Streaming SQL Engine

  Some recent solutions to this problem include Differential Dataflow and Materialize. It would be neat if postgres adopted something similar for live-updating materialized views. https://github.com/timelydataflow/differential-dataflow. - Source: Hacker News / 6 months ago

• Ask HN: Who is hiring? (October 2023)

  Materialize | Full-Time | NYC Office or Remote | https://materialize.com Materialize is an Operational Data Warehouse: A cloud data warehouse with streaming internals, built for work that needs action on what’s happening right now. Keep the familiar SQL, keep the proven architecture of cloud warehouses but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that... - Source: Hacker News / 7 months ago

• Ask HN: Who is hiring? (June 2023)

  Materialize | EM (Compute), Senior PM | New York, New York | https://materialize.com/ You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. That is Materialize, the only true SQL... - Source: Hacker News / 11 months ago

• Ask HN: Who is hiring? (April 2023)

  Materialize | NY, NY | https://materialize.com/ The Cloud Database for Fast-Changing Data. We put a streaming engine in a database, so your team can build real-time data products without the cost, complexity, and development time of stream processing. Cloud team openings: https://grnh.se/0ad6ab6b4us Senior PM openings: https://grnh.se/415c267f4us. - Source: Hacker News / about 1 year ago

• Query Real Time Data in Kafka Using SQL

  Most streaming database technologies use SQL for these reasons: RisingWave, Materialize, KsqlDB, Apache Flink, and so on offering SQL interfaces. This post explains how to choose the right streaming database. - Source: dev.to / about 1 year ago

• Why Are There No Relational DBMSs? [pdf]

  The relational model (and generally working at the level of sets/collections, instead of the level of individual values/objects) actually makes it easier to have this kind of incremental computation in a consistent way, I think. There's a bunch of work being done on making relational systems work this way. Some interesting reading: -... - Source: Hacker News / about 1 year ago

• Kafka Stream Processing in Java or Scala

  If you want to keep in your Python/SQL area of expertise and by all means I don't mean to promote not learning a new language, but just as an FYI. There are some non-Java/Scala tools between streaming databases like risingwave and materialize, streaming platforms like fluvio and redpanda, and stream processors like bytewax and faust. Source: about 1 year ago

• What makes a time series oriented database (ex: QuestDB) more efficient for OLAP on time series than an OLAP "only" oriented database (ex: DuckDB) technically?

  AFAIK there is a lot of overlap between OLAP databases and time series databases. Timescale](https://legacy-docs.timescale.com/v1.7/introduction/architecture) gains a lot of its performance via the "Hypertable" abstraction which is fairly similar to something like Parquet partitioning/bucketing. In terms of performance I don't know if there is a huge gap either for non optimized use cases. The [Clickhouse] team... Source: over 1 year ago

• How to handle partial updates and bulk updates in the source systems

  Imo this is a matter of schema design. You shouldn't have to send the entire object in the event, just the delta. If you are using an event based schema, ideally you should be able to generate the current state by iterating over all the events and combining the deltas. An OLAP database/warehouse/lakehouse can be very efficient at this depending on how the data is partitioned. You could consider a Materialized... Source: over 1 year ago

• Materialized View: SQL Queries on Steroids

  Projects like https://readyset.io/, https://materialize.com/, https://github.com/mit-pdos/noria can keep your materialized views up to date as the underlying base tables change. - Source: Hacker News / over 1 year ago

• Real Time Data Infra Stack

  Even though these technical stacks are listed by category, some fields actually overlap. For example, although Materialize is classified as a stream processor, it makes sense to treat as a serving layer because it is essentially a streaming database, and the same is true for ksqlDB. - Source: dev.to / over 1 year ago

• Show HN: Logical Replication with Rust

  Postgres Logical Replication (the ability to listen to the write-ahead-log) is an extremely useful tool that opens up a lot of use cases which has been put to great effect by companies like Materialize[0] and Redhat with Debezium[1]. Recently there was a discussion here on 'push-based output patterns'[2] where I saw it was not immediately obvious how to implement this pattern with existing tooling. This project... - Source: Hacker News / over 1 year ago

• Differential Datalog: a programming language for incremental computation

  > I'm not sure if differential dataflow can provide this. Yes, it can, but you will have to write the views yourself, in Rust. Materialize (https://materialize.com) exists, though, and compiles SQL to differential-dataflow programs, in order to provide exactly what you're asking for. (I work for Materialize). - Source: Hacker News / over 1 year ago

• Headless BI with streaming data

  In the last few years, streaming SQL technologies such as ksqlDB, Materialize, and Apache Flink have significantly progressed. These technologies enable us to process streaming data and run analysis with SQL—without needing to learn a new language or build specific language-unique integrations. - Source: dev.to / over 1 year ago

• What are data apps?

  Naturally, a data application starts with the data—and the basis of the modern data stack is the cloud data warehouse. This can be a general purpose data warehouse like Snowflake or a real-time tool like Firebolt, ClickHouse, or Materialize. - Source: dev.to / over 1 year ago

• A Critique of SQL, 40 Years Later

  I think these are great suggestions. It seems like you're suggesting that someone could design a functional-style programming language that compiles to SQL. 2 & 3 are my biggest pain points. I can't just extract functions like I can with a regular programming language. Instead, SQL queries get increasingly complex with no great tools to manage that. For 3, products like https://materialize.com/ look interesting... - Source: Hacker News / over 1 year ago

• Ask HN: Anyone using Rust for server side application development?

  Not me, but Materialize has built a streaming query materialization solution completely in Rust AFAIK, and has raised around $100MM so far according to crunchbase. https://materialize.com/ Super cool product. - Source: Hacker News / over 1 year ago

• OctoSQL allows you to join data from different sources using SQL

  Thanks! Definitely considering adding a server-mode with Postgres wire protocol compatibility. It's tricky for the more dynamic/dataflow'y parts, as OctoSQL is able to give you a live updating output table (which Postgres wire protocol doesn't support), but I can go with a similar approach as Materialize[0] does for those use cases - creating a live-updating materialized view that you can query from. That said,... - Source: Hacker News / almost 2 years ago

• Convex vs. Firebase

  Hi! Sujay from convex here. I remember reading about your "reverse query engine" when we were getting started last year and really liking that framing of the broadcast problem here. As james mentions, we entirely re-run the javascript function whenever we detect any of its inputs change. Incrementality at this layer would be very difficult, since we're dealing with a general purpose programming language. also,... - Source: Hacker News / almost 2 years ago

• How we made data aggregation on PostgreSQL better and faster

  Https://materialize.com/ is billed to be that. That behavior is not trivial to implement. - Source: Hacker News / almost 2 years ago