Where Does In-Stream Detection Fit in the SecOps Picture?

In the old days we did everything in stream, although it wasn’t called a stream back then. Single event matching became a thing based on platforms that could not do true correlation and were just really surfacing single events as notable. When you come from this world, then certainly you think everything that is detection in stream would be single-event matching, but that is not the reality.

In the SecOps context, correlation traditionally means establishing a relationship among multiple events, observations, entities, or data points—for example, connecting a failed login, a successful login from a new geography, and a privilege escalation. A rule that simply examines one event in isolation and matches fields against conditions is more precisely an event, match, filter, rule evaluation, or single-event detection than correlation.

Should we aim to run detections in the pipeline? It's an old question that came back loudly this year, and it now has real product money behind it on both sides. Clint Sharp wrote a good piece on this recently How I think about detection in the pipeline | Cribl. As the CEO of a security company and someone who has been working in the SIEM space for 25 years, I obviously have my own view on this topic. Here is how I see it.

At its core, threat detection has always been the same – take detection logic and point it at data. This has been the case since the very first SIEM and we could argue that most security tools work like that. Recently, the debate seems to be centered around the architectural question of – where is the data we are running detections against?

Are we applying detection logic to data as it is traversing our data pipeline? Have we sent it to a centralized storage like a SIEM or a Lakehouse? Are we leaving data where it is and accessing it over APIs in a federated approach?

All of them are valid and the question was never which one wins or is better, rather which one belongs where.

The case for faster detections

For a long time SOCs have been using time-based metrics to gauge their performance, usually centered around how quickly an alert is handled and split between different stages of the investigation. It has always felt odd that we track the speed of our analysts' work almost religiously, while not paying nearly the same attention to how fast we are to detect threats in the first place.

If we think about where that clock actually starts, we’ll realize that nearly every time-based SOC metric begins the moment an alert appears in a console. Everything that happened before that, collection, ingestion, indexing, the scheduled search waiting its turn, is uncounted. In result, we have been timing the race from the second lap.

The unfortunate truth behind many security logging architectures is that by the time an alert is displayed in a SIEM console, it may already be minutes if not hours from the initial event. It’s important to realize this especially now, in the era where adversaries are moving faster than before.  

According to CrowdStrike, in 2025 the average breakout time, which is the time between a threat actor's initial access and lateral movement, dropped to 29 minutes, with the fastest recorded case being only 27 seconds. This is roughly 65% faster than the prior year, and chances are adversaries will only speed up from there.  

AI makes attacks faster - from resource development, reconnaissance, to initial access, we can observe that attackers move faster and nowhere is it more visible than in the time it takes to exploit a new vulnerability. According to Mandiant's M-Trends 2026 the mean time to exploit a new vulnerability is –7 days. This means that exploitation happens on average a full week before a patch is released. Those are unprecedented numbers.

Traditional SIEMs are struggling to keep up. I strongly believe that in the AI-era, Mean Time To Detect should become a central concern for any SOC leader responsible for devising a security monitoring strategy.

Detections in the pipeline

While many Security Operations problems deal with process rather than technology, long MTTD is not one of them. The truth is that time to detect is not something SOCs have a lot of control over, and there is only so much we can do in a traditional SIEM. Yes, we can run detections on a tighter schedule, optimize ingestion latency, and route alerts automatically to gain back a couple of minutes.  

Unfortunately, shaving minutes off an alert queue will not get us to where we want to be.

The biggest detection speed gains come from moving detections left, to the data pipeline. That way we can demonstrably shorten the MTTD from hours or minutes to seconds. By doing so we enable SOCs to act faster and we make the other time-based SOC metrics meaningful. After all, what good is a sub-15-minute time to respond, if it took the SIEM an hour to surface the alert?

And, maybe even more importantly, we get to run detections against data we were never planning to ingest in the first place.

There is often no good business case to store verbose events centrally. DNS, proxy, flow logs, raw endpoint telemetry – those are only some of the log sources our clients find cost-prohibitive to store in a SIEM. In a traditional setup, we wouldn't be able to run detections against this data at all. With data pipelines, however, we can apply detection logic across broad data sources, effectively expanding our detection coverage from what's in the SIEM to what's in the organization.  

That way organizations can run their SOCs based on their own threat-models, not their vendors’ price lists.

Where does detection in-stream fit into the SecOps picture

Detecting threats in the pipeline has a number of advantages over the traditional SIEM. The main criticism of in-stream detections is that while they work great for single-event and short correlation window detections, they struggle with detections requiring long lookback times.

This is true, but also not always important. The right question to be asking is how many of your detections actually need that lookback.  

Here is how I think about it, strictly from a detection placement perspective. In a threat-driven approach to detection engineering, rooted in our understanding of adversary tactics, we quickly learn that our detection library falls into a number of shapes.

Some detections need a single specific event to surface a threat. Others need a short correlation window. The more complex ones run against hours' worth of data. And some are best executed where the data already lives, without centralizing it at all.

Single-event and short-correlation window detections are typically the majority of all detections. It is optimal to run them in the pipeline to ensure both cost-effectiveness and the shortest possible time to detect. Detections requiring long correlation windows are best run in a system that can store or access that data, for example a SIEM.  

For some organizations, single data source and short correlation window detections will cover 90% of their threat detection needs. Others will rely heavily on correlation and UEBA. And there will always be use-cases that call for a federated search, especially for data that’s typically not stored in a SIEM, like certain subsets of strictly operational telemetry.

None of these methods is in competition with the others. They are layers of the same detection fabric and I strongly believe that an AI-gen SIEM should weave them together.  

It means that the expectation of a modern threat detection system should be to deliver a holistic detection capability that allows to mix in-stream detections, scheduled queries and API-based federated detections into one, coherent capability.

Summary

Detections belong wherever they best serve the two things a SOC exists to do: detect and respond. For most detection libraries, that place is earlier than where we have historically put it. Moving detections left buys time and coverage, which is exactly why we built our SIEM on top of a security data pipeline.

None of this makes in-stream detection the single solution to an organization's threat detection strategy. Rather, it gives flexibility and options that traditionally were not on the table. Being able to pick and choose technology components to complete your threat detection stack is at the core of what we call composability.  

We believe the future is composable, and that a true AI-gen SIEM covers detections that run in the pipeline, against a centralized data store, and based on federated queries.

If this resonates, come and talk to us.

‍