Using object storage and concurrent requests.

In Delete Your Log Database, I described a different architecture for log storage and search. Logs are stored as files in your own object-storage bucket. Stateless compute nodes search those files directly. Storage and compute remain separate: you pay for storage continuously, but you use query compute only when a user runs a search.

The architecture uses Vector to collect and route logs, Amazon Simple Storage Service (S3) or an S3-compatible system to store them, Quickwit to index and search them, PostgreSQL to store the metadata catalog, and Grafana to display the results. This design allows Quickwit to store and process petabytes of logs. Query performance at this scale depends on how Quickwit accesses S3.

This post focuses on that storage layer. It explains how Quickwit can use an object store, where each request may take tens of milliseconds, and still return query results in less than one second at petabyte scale.

Why use an object store for a search engine?

Elasticsearch and many traditional search engines are designed for fast local disks. The index is stored on the node that serves it, often on NVMe drives with latency measured in microseconds. Quickwit uses a different design. It keeps little durable state on compute nodes and stores index data in object storage.

Three design choices support this approach.

• Stateless compute. The indexer, merger, and searcher keep as little local state as possible. The search index is stored in the bucket as files called splits. A small metastore, usually backed by PostgreSQL, records which splits belong to each index and which data each split contains. If a compute node fails, it can be replaced because it does not contain the source of truth.
• An inverted index built with Tantivy. Quickwit uses Tantivy, a search engine from the same family as Lucene. Tantivy creates an inverted index with term dictionaries and column-oriented fast fields. Lucene and Elasticsearch use a similar indexing model. This established approach is well suited to finding specific records in large volumes of text.
• Memory safety through Rust. Quickwit is written in Rust. Rust provides performance comparable to C while preventing common memory-safety problems such as buffer overflows, use-after-free errors, and data races. These guarantees reduce risk in software that reads data directly from a bucket containing production logs.

With this design, Quickwit delegates durable and scalable storage to an object-storage system. This system may be provided by a cloud vendor or deployed in a private data center.

The trade-off: higher latency for each storage request

Searching petabytes of logs over long retention periods is difficult. Many platforms use several specialized components: a buffering system such as Kafka for ingestion peaks, a database that scales storage and compute together, and a local-disk tier for recent data. They also need policies that move older data to cheaper storage.

Quickwit stores all index data in S3 from the beginning. This simplifies the architecture, but it introduces a clear performance cost.

A single object-storage request usually takes tens of milliseconds. A read from a local NVMe drive can take less than one millisecond. Therefore, a query that accesses a split that is not in memory must wait for at least one object-storage request.

In return, storage capacity can grow independently of the indexing and search nodes. The object store manages the stored data, while Quickwit nodes provide compute capacity. There is no separate Kafka cluster or local flash tier in this architecture. To add indexing or query capacity, an operator can add stateless compute nodes.

Quickwit reduces the effect of request latency through concurrency. A searcher sends many small byte-range requests at the same time instead of waiting for each request to finish before sending the next one. Under steady conditions, Little's Law relates throughput to concurrency and latency. In simplified form:

throughput ≈ concurrency / request latency

Consider a query that accesses 50 splits and requires four small reads from each split. The query sends 200 requests in total. If the requests run sequentially and each takes 30 milliseconds, the storage reads take about six seconds. If the searcher keeps 50 requests in flight, the work takes about four rounds. Without other bottlenecks, the storage time is then closer to 120 milliseconds.