Independent Verification: What It Costs and What It Refuses to Do
Eight days ago I published a piece about thirty years of systems that reported success and shipped nothing. Enabled is not delivering. A green dashboard is a claim, not a measurement.
That article described a problem and stopped there, which is the honest limit of an article. This one is about what I built afterwards, what it costs, and what it deliberately refuses to do.
The service
It is called Konstatum. It answers one question in writing: does the control you believe you have actually do what you believe it does?
Ten working days. One site, one perimeter. One dated document in which every line has four parts — the claim you stated in writing before anything was measured, the method, the raw evidence, and one of three verdicts: confirmed, contradicted, or not tested.
That third verdict is unusual to see written down, and it is not a softer form of failure. It means a claim was not verified; it says nothing about whether the claim is sound. A report containing no such lines is asserting a coverage it does not possess.
The terms, in full
EUR 1,500, fixed. One site, one perimeter, ten working days.
EUR 350–500 per month afterwards, for re-verification of the same lines. Priced separately, and offered only once you have read the first report. A report that is never repeated becomes a document; a report that is repeated is a control.
No remediation. Ever. Not as an option, not for an additional fee, not at any point. An assessor who also sells the repair has an interest in finding one. Where something needs putting right it goes to whoever already looks after your systems — and I remain able to verify their work afterwards, which I could not do had I performed it myself.
Not a penetration test. Not monitoring. Not a certificate. Those exclusions are written into the engagement, because a document that states plainly what it did not examine is far harder to attack than one implying it examined everything.
Paid, deliberately. A complimentary security assessment is an instrument of sale and everyone in the room understands it as such. One you commissioned is an opinion.
Why the claims come first
Nothing is measured until you have written down what you believe to be true, and signed it. This is the declaration, and its position in the sequence is not a formality.
An assessor working alone cannot distinguish a deficiency from a decision. A firewall accepting all inbound traffic is a serious finding — unless filtering happens upstream by design, in which case reporting it is noise dressed as rigour. In the first full run of the method against my own infrastructure, two findings out of thirty-one were exactly that error. Six per cent of the report was an assessor mistaking a decision for a defect. That rate is not acceptable in a document someone signs, so the order changed.
Where this usually starts
Not with curiosity. It starts when an insurer sends a renewal questionnaire that must be answered in writing and signed, or when a larger client’s procurement department begins asking its suppliers the questions its own auditors ask of it.
Those answers get given from memory, from a vendor’s dashboard, or from what the IT provider said the last time anyone enquired. All three describe what was intended. None describe what a test would find — and insurers have contested claims on precisely that distinction.
Nullius in verba
“On the word of no one.” The motto of the Royal Society, adopted in its first charter of 1662 and taken from Horace. A claim is established by test rather than by the standing of whoever makes it.
Which is why the method was run against my own production infrastructure before it was offered to anyone. Thirty-three verdicts: fourteen confirmed, twelve contradicted, seven not tested. The full report — including the twelve failures and the two findings that turned out to be my error rather than the system’s — goes to anyone who asks.
The method, in full · what the report contains · alex@tothenoc.com
The Checkbox Is Not Delivery
Every failure I have cleaned up in twenty-five years had already been reported as working.
Not “degraded”. Not “warning”. Working. Green. Enabled. true.
The technology underneath changed six times. The failure did not. It is the same bug, wearing a new interface every decade, and it survives because each generation of engineers meets it for the first time.
The nineties: the backup that ran every night
Tape rotation. The job log said SUCCESS, every night, for years. Nobody ever restored from it. The first restore attempt was also the first test – and it happened on the worst day of that company’s year.
The job was not lying. It really did complete. It completed writing to a drive whose heads had drifted out of alignment months earlier. The job’s definition of success was “I finished my instructions”. Nobody had asked the only question that mattered: can we get the data back?
The two-thousands: the array that was healthy
RAID controllers gave us a green light and a sense of safety. The green light meant “the array is serving reads and writes”. It did not mean “you have redundancy left”.
I have opened more than one server to find a disk that had died eight months earlier. The array was healthy – it was doing exactly what a degraded array does, quietly, correctly, one failure away from nothing. The alert had been configured. The alert had nowhere to go. The monitoring had no monitoring.
The same decade: logging enabled, collector missing
Then syslog and SNMP arrived and we all learned a new way to be wrong. You tick the categories on the device. The device faithfully generates events. And the events go to a collector that was decommissioned in a migration two years ago, or to a partition that filled up in a quarter nobody remembers.
The device is not broken. The device is doing its job. It is shouting into a room where nobody has stood for years.
The twenty-tens: HA enabled
Virtualisation industrialised the pattern. “HA enabled” became a checkbox in a cluster settings pane, and the checkbox was true.
What it did not tell you: admission control was off, or the reservations did not fit, so the cluster had nowhere to restart the workloads it had promised to restart. The configuration was correct. The capacity was not there. You find out at 3am, in the only test that counts.
The twenty-twenties: the cloud made the window longer
Here is the version that should frighten you, and the reason I am writing this.
Earlier this year I audited an environment and found a cloud audit trail reporting IsLogging: true. Enabled. Healthy. Every console screen agreed.
It had delivered nothing since 2020. The destination bucket had been deleted years before. The trail kept reporting that it was logging, because from its point of view it was – it was doing its half of the job. Nobody had checked the other half.
That is five and a half years of a system that would have passed any audit that asked “is audit logging enabled?” – and produced not one line of evidence for any incident in that window.
Compare the windows. The dead RAID disk hid for eight months. The dead trail hid for five and a half years. The abstraction got thicker and the false-positive window got longer. That is not a coincidence, it is the trend.
Why every cycle makes it worse
Each generation moved the reporting layer further from the effect, and outsourced the verification to somebody else’s dashboard.
With tape, the reporter and the effect were in the same room – you could hold the cartridge. In the cloud, the thing reporting health is a managed service you cannot open, describing a delivery you cannot see, on infrastructure you do not own. The distance between “it says it works” and “it works” is now measured in years.
And the newest layer is already repeating it. Automation and AI agents report done, COMMITTED=True, task complete. I use these tools daily and they are genuinely good. But that string is a label the process printed about itself. It is the tape job’s SUCCESS, in a new font. If you have not looked at the artefact, you do not know.
The habit that catches all of them
Measure the effect, not the report. Everything else is a corollary:
- Never accept a flag as evidence. “Enabled” is a statement of intent. Ask what the system produced, and go look at it.
- Verify at the destination. Not at the source. The source always thinks it is fine – that is what makes it useless as a witness. Look in the bucket. Look on the collector. Look for the object, with a timestamp.
- Anything that has never been exercised does not work. It is untested code with a green light. Backups you have not restored, failovers you have not triggered, runbooks nobody has walked: treat them as broken until proven otherwise, because that is the honest prior.
- Ask what the reporter can actually see. Most false green is not a lie – it is a component honestly reporting on the half of the system it can observe, while the half it cannot observe is where the failure lives.
The test
Pick your most important safety net. The backup, the audit trail, the failover, the alert.
Do not open the settings page. Go find the thing it produced. The restored file. The log object with today’s date on it. The alert that actually landed in somebody’s hand.
If you cannot find it in ten minutes, you do not have the control. You have a checkbox.
In thirty years, that has never once been the wrong place to look first.
Written eight days before there was anywhere to send you. There is now: Konstatum is an independent verification of whether the controls you believe you have actually hold — tested, dated, and written down. It does not repair what it examines, which is the only way the answer stays worth reading.
The NOC Math Nobody Runs Before They Hire
Someone, right now, is building a business case for an in-house NOC. One headline number: an engineer’s salary. One conclusion: “cheaper than outsourcing.” They’re about to be wrong in a way that only shows up at 3 a.m. on a Sunday.
A NOC’s whole job is to be watching when nobody else is. That’s 8,760 hours a year. One engineer covers maybe 1,800 of them — after holidays, sick days, and sleep. True round-the-clock coverage takes four and a half people before anyone takes a vacation. Not one.
So the real choice was never “salary vs. fee.” It’s four idle desks vs. coverage you only pay for when it matters.
And here’s the part that stings: most of those desks are tier-1 — people who see a red light and forward it. They don’t fix anything. You’re paying a room to escalate problems to the one person who can actually solve them.
We built ToTheNOC to skip that room.
There’s a version of this where in-house is exactly right. There’s a far more common version where it quietly bleeds a budget for two years before anyone notices. Which one are you building?
When everything fails, we don’t. → the full cost breakdown · why we run no tier-1 team · what 24/7 coverage costs
Why We Don’t Have a Tier-1 Team (And Why That’s Better For You)
Every managed NOC company will tell you they have “experienced engineers” and “rapid response times.” What they won’t tell you is that your 3 AM alert goes to a Level 1 technician reading from a runbook.
Here’s how it actually works at most NOC providers: your alert fires, it lands in a queue, a junior technician opens a checklist, runs through basic steps, and if it’s not resolved in 15 minutes, escalates to Level 2. Who escalates to Level 3. Who might actually fix it — an hour later.
At ToTheNOC, there is no Level 1. There is no Level 2. There is one senior engineer with 25+ years of infrastructure experience who picks up the alert directly.
What “No Tier-1” Actually Means For Your Infrastructure
When your database crashes at 2 AM, you don’t need someone to ask “have you tried restarting it.” You need someone who has seen that crash 50 times before, knows it’s a corrupted InnoDB tablespace, and has the fix running before your users notice anything.
The tier model exists for scale. When a NOC provider manages 500 clients, they can’t put a senior engineer on every L1 ticket. So they build a funnel: cheap labor filters the noise, senior engineers handle only the complex cases.
That works fine for large enterprises with predictable, well-documented infrastructure. It doesn’t work for the web hosting provider, the growing SaaS startup, or the MSP whose clients have mixed, undocumented environments.
The Real Cost of Tier-1 Filtering
Every time a Level 1 technician touches a ticket before escalating, you lose time. In infrastructure, time is downtime. Downtime is revenue.
Consider a typical cascade failure: a memory leak in a PHP-FPM worker pool causes response times to spike, MySQL connections queue up, Nginx starts returning 502s. A Level 1 technician sees “Nginx service degraded” and restarts Nginx. Alert closes. Fifteen minutes later, the same thing happens — because the root cause was never touched.
A senior engineer sees the full picture immediately: the correlation between PHP-FPM memory, MySQL connection count, and Nginx errors. One intervention. Problem solved.
What Boutique NOC Actually Means
“Boutique” is not a marketing word. It means we take a limited number of clients intentionally so every environment gets direct senior attention.
We know your stack. We know your traffic patterns. We know which services are critical and which alerts can wait until morning. That context doesn’t exist in a ticketing system. It exists in the engineer who has been watching your infrastructure for months.
The trade-off is honest: we are not the cheapest option. If you want a call center with SLA dashboards, there are plenty of providers. If you want the engineer who has seen your specific failure before to be the one who responds — that’s ToTheNOC.
Who This Is Right For
Single-operator NOC monitoring is the right fit when your infrastructure is non-standard or hybrid, you’ve been burned by escalation delays, and you value resolution time over response time metrics.
What outsourced NOC monitoring actually covers, and what it costs: read the full guide. Plans and rates: NOC services pricing.
If your infrastructure deserves that, let’s talk.
What a 4-Hour NOC Response SLA Actually Means at 3am
SLAs are contracts. What matters is execution.
Every NOC provider advertises response SLAs. “4-hour response.” “1-hour critical response.” “24/7 coverage.” These numbers are easy to print on a website. What they mean in practice varies enormously — and you usually only find out when something is already broken.
I’ve been on both sides of this. As a senior NOC engineer handling emergency responses, and as someone who’s had to clean up after a “guaranteed 4-hour response” that turned into a 14-hour outage. Here’s what actually matters.
What “response” means — and what it doesn’t
Read the fine print on any SLA. “Response” is almost never defined as “your problem is solved.” It usually means one of three things:
- Acknowledgment: We received your alert. A ticket has been opened. An automated email was sent. This is the weakest possible definition.
- Initial triage: An engineer has looked at the alert and classified the severity. Still no guarantee of resolution timeline.
- Active engagement: An engineer is actively working the issue. This is what you actually want.
When evaluating a NOC provider, ask explicitly: “When you say 4-hour response, does that mean an engineer is actively working my issue within 4 hours, or that I’ve received an acknowledgment?” The answer tells you everything.
What actually happens at 3am when a server goes down
A realistic sequence with a well-run NOC:
- T+0:00 — Monitoring system detects anomaly (service timeout, disk full, interface down)
- T+0:02 — Alert fires. If it’s a transient spike, it clears and nothing happens. If it persists:
- T+0:05 — Engineer is paged. Not a bot. Not a tier-1 filter. An engineer.
- T+0:10 — Engineer is logged in, running diagnostics. Checks logs, service status, recent changes.
- T+0:20 — Root cause identified in most cases (disk, process crash, network, application). Remediation begins.
- T+0:45 — Service restored or escalation path activated if issue requires vendor involvement.
That’s what good looks like. A 4-hour SLA means the engineer is engaged within 4 hours of the alert — not that resolution takes 4 hours.
Red flags in NOC SLA agreements
SLA credits instead of resolution commitments. “If we miss our SLA, you get a credit on next month’s invoice.” A credit is nice. It doesn’t fix your 6-hour outage. Ask about resolution commitments, not just credit policies.
Tiered escalation with undefined timelines. Tier-1 responds in 1 hour, escalates to tier-2 in 2 hours, tier-2 escalates to tier-3… By the time someone who can actually solve your problem is on the call, you’re 6 hours in. For SMBs, a flat escalation path to a senior engineer is worth more than a multi-tier SLA.
Exclusions buried in the contract. “SLA applies during business hours.” “SLA excludes third-party service outages.” “SLA excludes hardware failures.” Read the exclusions before you sign.
No documentation of your environment. If the NOC doesn’t have documentation of your specific infrastructure — topology, credentials, runbooks — their engineer is starting from scratch during your incident. That costs time you don’t have.
What the ToTheNOC response SLA actually means
When a client in the NOC Command plan has an incident, the SLA is under 4 hours — but in practice it’s usually under 15 minutes. Why? Because there’s no tier-1 filter. The alert goes directly to me. I know the client’s environment because I documented it during onboarding. I’m not reading a wiki to figure out what credentials to use.
The advantage of a boutique NOC isn’t just the SLA number — it’s the context. One engineer who knows your environment responds faster and more effectively than a staffed NOC where the overnight shift has never seen your infrastructure before.
Questions to ask any NOC provider before signing
- What does “response” mean in your SLA — acknowledgment, triage, or active engagement?
- Who specifically responds to my incident at 3am — tier-1, tier-2, or a senior engineer?
- What documentation will you maintain about my environment?
- What are the SLA exclusions?
- Can I see a sample incident report from a past engagement?
- What’s your escalation path if the on-call engineer can’t resolve the issue?
Alexandru Cazan is a senior NOC engineer with 25+ years of remote infrastructure experience. Learn more about NOC Response services or book a free technical call. For the full picture on outsourced NOC monitoring, start here.
Outsourced NOC Monitoring: What It Covers, What It Costs, and How to Choose a Provider
Most pages about outsourced NOC monitoring are written by people selling outsourced NOC monitoring. This one is too — I run ToTheNOC. The difference is that I will tell you plainly where outsourcing this function is a bad idea, because the engagements that start on a wrong assumption end badly for both sides.
Below: what an outsourced network operations center actually covers, the four ways you can buy it, what each one really costs, and how to tell a real NOC from a ticket queue with a logo.
What outsourced NOC monitoring is
NOC stands for Network Operations Center. In practice it means someone has eyes on your infrastructure continuously — servers, network devices, services, certificates, backups, response times, error rates — and acts when something moves in the wrong direction. Outsourcing it means paying a third party for that function instead of staffing it yourself.
The word covers an enormous range. At one end, an automated service forwards alerts to your phone for $50 a month. At the other, a staffed enterprise NOC runs shift rotations for $10,000 a month and up. Both are sold as “outsourced NOC monitoring”. They are not the same product, and the gap between them is where most bad purchases happen.
What an outsourced network operations center actually covers
Ask any provider to sort their offer into these four layers before you compare prices. Most quotes that look cheap stop at the first one.
- Detection. Checks running against your estate on an interval, with thresholds that mean something. This is the layer every provider has, and the only layer some of them have.
- Triage. A human deciding whether an alert is real, correlating it with the other twelve that fired at the same time, and knowing which one is the cause. Without this, you have not outsourced monitoring — you have outsourced notification.
- Response. Someone with credentials and authority actually fixing it, at 3 AM, without waking you first to ask permission.
- Prevention. Patching, certificate renewals, capacity headroom, configuration drift, restore tests. The work that stops the 3 AM call from happening at all.
The honest test for any outsourced network operations center: ask what happens between the alert firing and the problem being fixed, and count how many times a human is expected to make a judgement call. If the answer is zero, you bought software. If the answer is “we escalate to your team”, you bought a pager.
The four ways to buy outsourced NOC services
1. Monitoring software you run yourself
Zabbix, Checkmk, PRTG, Datadog, and the rest. Cheap to license, excellent tooling, and genuinely the right answer if you have someone in-house who will own it. What people underestimate is that a monitoring platform is itself infrastructure: it needs tuning, maintenance, and someone who cares when it goes quiet. An unmaintained monitoring stack is worse than none, because it manufactures confidence.
2. Automated alert forwarding
$50–150 a month, no humans. It will tell you the disk is full. It will not tell you which of the four things that broke simultaneously is the cause, and it will not fix any of them. Fine as a smoke detector. Not a NOC.
3. A tiered enterprise NOC
Real 24/7 shift coverage, real headcount, real process maturity. If you run hundreds of nodes across regions and need documented shift handovers and formal escalation matrices, this is the correct purchase and you should make it.
The trade-off is structural, not a flaw: the person who picks up your alert at 3 AM is on tier one, follows a runbook, and has never seen your environment before. Anything the runbook does not cover gets escalated, and the escalation path has a clock on it. You are buying coverage and process, not judgement.
4. A boutique NOC — one senior engineer
This is what ToTheNOC is. One engineer with twenty-five years in production infrastructure — firewalls, virtualization, cloud, backup and disaster recovery — who knows your estate because he set up the monitoring for it and has looked at it every week since. No tier one. No script. No handover.
The trade-off is equally structural, and I will state it before you ask: one person is not a shift rotation. I am explicit about response windows per plan rather than implying an always-on room of staff. If your requirement is a contractual seat filled at every hour of every day, buy option three. That is not a sales tactic — it is the difference between an engagement that works and one that fails in month four.
Outsourced NOC software vs an outsourced NOC service
People searching for “outsourced NOC software” or “outsourced NOC tools” are usually asking one of two different questions, and the answer differs completely.
If you are asking which platform should we deploy, the honest answer is that the platform matters far less than who owns it. Zabbix is free and excellent, and I use it in production. It will also sit there generating noise nobody reads if no one is accountable for tuning it. Choose the tool your team will actually maintain. The stack underneath matters as much as the dashboard on top of it: how a hosting stack behaves under load decides how much of that alert noise is real.
If you are asking what tooling does an outsourced provider bring, ask a sharper version: who owns the monitoring stack when the contract ends? A provider that runs your monitoring inside their own tenant, with their own agents and their own dashboards, has made your visibility an asset you rent. When you leave, the estate goes dark and you rebuild from zero. Ask this before signing, not at renewal.
What outsourced network monitoring costs
Most providers will not publish a number, which is itself informative. Mine are on the NOC pricing page in full, but the shape is:
- $299/mo — monitoring, alert triage, and a monthly health report. Up to 5 nodes, next-business-day response. Detection and triage, no response layer.
- $699/mo — the above plus a monthly maintenance window, patch management, and configuration reviews. Up to 15 nodes, 4h response in business hours. This is the one most estates actually need.
- from $1,499/mo — full coverage including 24/7 emergency response with a sub-4h SLA, unlimited nodes, quarterly infrastructure audit. Scoped per environment, because estates differ and a flat number here would be fiction.
- $150/hr, 2h minimum — single incident triage and root cause analysis, no retainer.
For the comparison against hiring, the arithmetic is laid out in in-house vs outsourced — the real cost math. The short version: a single competent infrastructure engineer costs multiples of any of the above, cannot cover nights alone, and takes holidays.
One number to be suspicious of: any quote for 24/7 coverage that is cheaper than one salary. Round-the-clock staffing has a hard floor. If the price is below it, either the coverage is automated, or it is shared across so many customers that yours is one alert in a queue.
When outsourced NOC monitoring makes sense
- Your infrastructure matters more than your headcount can cover. Production systems that customers depend on, run by a team of two or three who are also building the product. Outsourcing monitoring buys back their attention.
- You have nobody who owns infrastructure. Developers who inherited the servers, an office manager with the firewall password. This is the most common case and the one where outsourcing pays for itself fastest.
- You cannot justify a full-time hire. The work is real but it is fifteen hours a month, not a hundred and sixty.
- You need coverage outside your hours. A team in one timezone with customers in another.
- An audit or a customer requires it. Monitoring and incident response are controls in most compliance frameworks, and “we check it sometimes” does not survive scrutiny.
When it does not make sense
- You already have a capable infrastructure team. Then you need better tooling or one more engineer, not a third party learning your estate from outside.
- Your infrastructure is genuinely trivial. One WordPress site on managed hosting does not need a NOC. It needs decent hosting and real backups.
- You want someone to blame. An outsourced NOC transfers work, not accountability. If the goal is a contract to point at after an outage, you are buying the wrong thing.
- You will not give real access. Monitoring without credentials, or “screen share while I watch”, produces theatre. I have declined engagements on exactly this.
- The budget forces the cheapest tier for a serious estate. Under-scoped monitoring is worse than none, because it tells you everything is fine right up until it isn’t.
How to evaluate an outsourced NOC provider
Six questions. The answers separate providers faster than any brochure.
- Who picks up at 3 AM, and what can they do without calling you? Name and seniority, not a department. If the honest answer is “escalate”, price it as an alerting service.
- How is the response clock measured? From alert firing, from ticket creation, or from a human reading it? The difference is hours. I wrote up how this is measured in NOC emergency response SLA, explained.
- What exactly is monitored, and who decided? A provider that never asked which services actually matter to your business is monitoring defaults.
- Who owns the monitoring configuration? Covered above. Ask it in writing.
- When did you last test a restore? Backups that have never been restored are not backups. This question embarrasses more providers than any other.
- What will you refuse to do? Any provider that claims to cover everything at every hour for a small monthly fee is describing something that does not exist.
The boutique versus enterprise trade-off
Enterprise NOCs sell coverage and process. Boutique operations sell judgement and continuity. Both are real; they solve different problems.
The failure mode of the enterprise model is that nobody in the room has ever seen your environment, so novel problems bounce up an escalation chain while the outage continues. The failure mode of the boutique model is that there is one person, and one person has a finite number of hours. Anyone selling you a model without naming its failure mode is selling.
My position is narrow on purpose: small and mid-sized estates where the value is in someone senior knowing the environment, not in a badge count. If that is not you, the enterprise providers are genuinely better and I will say so on the call.
Where to go from here
If you want the numbers, they are published on the NOC service pricing page — no tiers hidden behind a form. If you want the scope of what is covered, see managed and outsourced NOC services.
If you want to know whether any of this applies to your estate, book a 30-minute technical call. It is a conversation with the engineer who would do the work, not a qualification script — and if the answer is that you do not need an outsourced NOC, that is what I will tell you.