C2 Corner: How MSSPs Skew SOC Metrics in Their Favor: Top 5 Techniques Observed Over Years in SecOps

In the world of MSSP SOCs, metrics become the primary way to demonstrate effectiveness, especially to non-technical stakeholders. The more pressure there is to hit Service Level Agreement (SLA) targets, the more likely those metrics are to be gamed.

Over my career I have had to go through a ton of month-end SLA reports to validate whether the metrics presented in them were correct. To nobody's surprise, nearly every other report showed skewed numbers. Having done that work (and having worked for MSSP SOCs myself) I have learned a lot of distinct ways to "cook" SOC metrics. Below I will go through the 5 most common ones.

Before I start listing the accountability evasion techniques, one disclaimer feels important: SLA targets are often unrealistic.

On one hand we say we want quality investigations. On the other, we reward analysts for speed or volume. We can't have it both ways, and if we give our MSSP 15 minutes to handle a critical alert, no wonder they can't do it. In other words, let's not blame analysts for failing to meet impossible expectations, ok?

With that said, let's look at the most common ways MSSPs manipulate metrics to appear compliant.

1. SEVERITY DOWNGRADE: AN ALL-TIME CLASSIC

Suppose the SLA requires high-severity alerts to be closed within one hour, but an analyst is short on time or needs a break. A quick downgrade to medium buys another hour, no questions asked. This tactic is especially common when SLAs are unrealistically tight or the team is overwhelmed.

Best practice for SOC leaders:

• Require justification for severity changes – make sure those are an outcome of triage, not SLA pressure

2. SLA RESET DURING ESCALATION: UNDERRATED, BUT EFFECTIVE

A common accountability evasion tactic is resetting the SLA clock when an alert is escalated between tiers.

Say the SLA defines a 1-hour time to acknowledge and a 1-hour time to close for high-severity alerts. The alert is picked up by Tier 1, then escalated to Tier 2, and finally to Tier 3. If each team restarts the timer, the total response time can stretch far beyond SLA limits while still appearing compliant.

‍

Best practice for SOC leaders:

• Ensure SLA timer begins at the initial alert generation and runs continuously across escalation

3. INCLUDING AUTOMATION IN TIME-BASED METRICS: ANOTHER CLASSIC

MTT-X metrics are only meaningful when they reflect human analyst activity. However, MSSPs surprisingly often include automatically assigned or closed alerts in the calculation. Automation works in seconds, so adding it to an average drags the total down significantly.

Best practice for SOC leaders:

• Verify if automatically handled alerts are excluded from SLA-based metrics

4. SELECTIVE SAMPLING: OR EXCLUDING ALERTS THAT BREACH SLA

MSSPs sometimes cherry-pick alerts that were handled within SLA and exclude alerts that were not. This is very common for alerts that occurred over the weekend. This is difficult to see at first glance and usually requires a second pair of eyes to spot.

Best practice for SOC leaders:

• Require SLA reporting to include all alerts within the period, including weekends, holidays, and peak activity windows
• Ask for raw data and calculate SLA yourself from time to time

5. FABRICATING TECH ISSUES TO EXCUSE SLA BREACHES: HONESTLY THE MOST DISAPPOINTING ONE

This one is, frankly, the most disappointing and unfortunately also very common. It’s the tactic that erodes trust in a vendor faster than anything else. When asked why an alert was missed or picked up late, sometimes the response will be a vague and convenient excuse like:

• We were facing technical issues
• The alert never showed up in the queue
• The VDI went down

Those are often easily disproven with a quick check of audit logs, access records or an ingestion dashboard.

Best practice for SOC leaders:

• Make sure that operational issues are logged, acknowledged and shared transparently. If the system is working, there’s no excuse not to act.

KEEPING YOUR SERVICE PROVIDER HONEST

Trust but verify is the foundation of a healthy relationship with your MSSP. In practice, that means having at least one deeply technical resource on your side who can validate what the provider reports. Ideally that’s someone who knows the platform, can pull raw data, and investigate at least as well as your service provider.

If you don't have that person yet, I hope this article is at least useful for spotting the ways metrics can be skewed in your provider's favour. And if you only do one thing with it, ask for the raw alert data behind next month's report and recalculate a number or two yourself.

And to reiterate the disclaimer at the top – sometimes the targets you’ve set are not achievable. The goal should not be to catch your MSSP red-handed, rather to spot signs that the partnership is defined incorrectly and work together to arrive at a realistic, mutually beneficial setup.

C2 PERSPECTIVE

by Chris Camacho, Co-Founder & COO at Abstract

Rafał’s piece gets at something I’ve seen for years: metrics can become the work instead of measuring the work. When SLA attainment is what everyone is judged on, people will optimize for SLA attainment. That happens anywhere a proxy starts carrying more weight than the outcome it was supposed to measure.

The practical takeaway is to keep access to the data underneath the report. If a metric says response is improving, you should be able to trace that number back to the alert, the timestamps, severity changes, escalation path, and the activity that followed. Otherwise, you’re taking the metric on faith.

That becomes even more important as security operations get faster and more automated. At Abstract, we think a lot about what happens between data arriving and a security decision being made. That chain needs to stay visible. You need to know what happened to a signal, what changed along the way, and why.

Rafał’s disclaimer matters too. An unrealistic SLA can create the wrong incentives even in a good partnership. When the target doesn’t reflect the reality of the work, the target and operating model need to be revisited.

The best MSSP relationships I’ve seen have enough transparency that the provider and the customer can look at the same underlying data and have an honest conversation about the outcome. That is what keeps metrics useful. They support trust instead of standing in for it.

‍