For years, the smartest thing you could do with old logs was to delete them.
A human was only going to look at the last few hours anyway. Keeping a month of logs searchable meant paying for hot storage and for nodes that almost nobody used. So retention windows shrank to 7, 14, maybe 30 days. The rest went to a bucket that nobody could search, and everyone moved on. It was the rational choice.
It is not anymore. Three things changed at the same time. Together, they turn long and searchable retention from a niche compliance problem into a general engineering problem. And the two default open source answers, ELK and Grafana Loki, are not designed to solve it at a reasonable cost.
AI multiplies the logs and the reasons to read them. Every agent, copilot, and generated service is one more thing producing telemetry, and volume grows fast. But the more interesting change is on the consumer side. Old logs were not worthless because they had no signal. They were worthless because no human had time to dig through six months of history looking for a weak pattern. An AI analyst removes exactly that limit. An agent doing threat hunting or root cause analysis does not get tired. It will look into the past, because looking is cheap for it. To be clear: AI does not make your logs valuable as training data. Raw logs are noisy and full of PII, and almost nobody trains a model on them. AI makes them valuable because it is the first analyst with enough time to mine the history we used to delete.
Security needs that history, and the numbers are hard to ignore. When you detect a breach, the evidence you need is usually old. Mandiant’s M-Trends 2026 report puts the global median attacker dwell time at 14 days. For the dangerous categories, espionage and insider operations, the median is 122 days, and some intrusions stay hidden for more than a year. IBM’s Cost of a Data Breach 2025 report measures the full breach lifecycle at around 241 days. And there is a worse number: in about a third of the intrusions, Mandiant could not determine the entry point, in part because the logs were already gone. The most useful security question today is: “this indicator was published this morning, have we seen it anywhere in the last twelve months?” You cannot answer it with two weeks of retention.
Sovereignty is closing the easy way out. The path of least resistance was to send everything to a US SaaS. That path now collides with GDPR, Schrems II, the Cloud Act, and the European digital sovereignty initiatives. For many organizations, sending their most sensitive operational data, full of secrets and infrastructure details, to a third party vendor running on someone else’s cloud is no longer acceptable.
Put the three together: more logs, a real reason to keep them searchable for months, and the obligation to do it on infrastructure you control.
The two usual options fail, and they fail differently. That difference matters.
On a hot cluster, the cost of keeping a log searchable grows with every day you keep it. On object storage, it stays almost flat.
Elasticsearch links retention cost to compute. In an ELK or OpenSearch cluster, the index and its replicas live on local SSD, on nodes that are always on. The data is the cluster. Keeping a log searchable for a year means paying for a year of hot or warm nodes. The cost of retention grows with every day you keep the data. You can move old indices to cold snapshots, but then they are not really searchable: you have to restore complete indices back to the nodes before you can query them, and that is a slow batch operation.
Loki is cheap because it does not really index. Grafana Loki stores chunks in object storage and only indexes labels, not the content of the logs. It is a good design for cheap storage and for tailing logs by label. But to search the content, Loki has to scan every chunk inside the label and time window you select. For full text search over high cardinality data and long time ranges, which is exactly the security and forensic case, that becomes an expensive grep.
So the choice today is bad: an always-on search cluster with a bill that grows with retention, or cheap storage that you cannot really search. Neither one matches the three forces above.
The way out is the same change that transformed data warehousing ten years ago, when Snowflake and later the lakehouses separated storage from compute. That change is now arriving to observability, and the open source pieces to build it yourself already exist.
The core idea: stop running a log database. Your logs become immutable files in your own object storage, in an open format that you own. Search runs on stateless compute that reads those files directly and scales to zero between queries. Storage and compute are fully separated. Years of logs stay one query away, for more or less the price of a bucket, and you pay for query compute only when someone asks a question.