Control design and testing (design vs operating effectiveness)¶
A control can pass on paper and fail in practice¶
Scaldimet has a standard: the maintenance contractor's remote access to Beta must be logged and reviewed monthly. On paper that is a sound control. But two separate questions decide whether it actually reduces risk. First: if it were performed exactly as written, would it work? Second: is it actually performed, every month, by someone, leaving a record? These are different questions, and a control can pass one while failing the other. Confusing them is how organisations end up "compliant" and exposed at the same time. Telling them apart is the skill of this Lesson.
Design effectiveness versus operating effectiveness¶
- Design effectiveness asks whether the control, as designed, would prevent or detect the risk it targets, assuming it is performed correctly. A monthly review of contractor access is well designed if it would actually catch misuse. A control to "raise awareness" with no defined action is poorly designed: even performed perfectly, it changes nothing measurable.
- Operating effectiveness asks whether the control is actually performed, consistently, over a period of time. A perfectly designed monthly review that no one has run since March is operating ineffectively, and protects nothing.
The two failure modes are different and need different fixes. A design failure is fixed by redesigning the control. An operating failure is fixed by making it happen (an owner, a schedule, a reminder, the easy-way levers from Lesson 2.6). An untested control is, in effect, an assumed control, and Scaldimet has assumed far too many: the inherited "vendor-supported software" standard was well-intentioned, but not designed for OT reality, and unoperated for years.
How you test: four methods, from basic to stronger¶
Testing a control means gathering evidence about design or operation. Four methods are standard, in increasing order of strength and effort:
| Method | What it is | What it proves | Example at Scaldimet |
|---|---|---|---|
| Inquiry | Ask the people involved | Weakest: what people believe happens | Ask the Beta team whether access is reviewed |
| Observation | Watch the control being performed | It can happen, at least when watched | Watch a monthly review take place |
| Inspection | Examine records and configurations | It did happen, and how | Read the dated access-review logs for six months |
| Re-performance | Independently redo the control | It works and produces the right result | Re-run the access review yourself and compare |
Inquiry alone is the trap: people sincerely report what is supposed to happen. Operating effectiveness in particular needs inspection or re-performance across a period and usually a sample (you cannot re-check every day, so you check a representative set), because a control is about consistency over time, not a single good day for the auditor.
The risk rationale is direct: every control addresses a named risk (Chapter 3.3), whether inherited from the baseline or added for Scaldimet. Testing is how you confirm it is actually doing so. Without testing, the claimed risk reduction remains unverified. That is the check-box trap Chapter 3.7 examines.
Application: design a test¶
On paper, take Scaldimet's contractor-access-review control and design a test for it. Specify: which effectiveness you are testing (design, operating, or both); the method (for operating effectiveness, choose inspection or re-performance, not inquiry); the period and sample (for example, the review records for the last six months); and what result would count as a pass versus a finding.
Model answer (open after you've drafted yours)
A more complete answer works through the control's two properties in the right order, then the four specifications the question asks for.
Design first. Before testing whether the review happens, check whether the review, as designed, could catch contractor misuse at all. "Logged and reviewed monthly" is well designed only if "review" has a defined action: comparing the contractor's sessions against expected maintenance windows or change tickets, and flagging anything out of hours, unexpected, or reaching systems outside the contractor's scope. If "review" means glancing at a log, the control is poorly designed and will pass an operating test while catching nothing. So the first specification is: test both, and test design before operating, because testing the operation of a badly designed control is wasted effort.
Method. For operating effectiveness, use inspection as the baseline and re-performance as the stronger check. Inspection: read the dated review records and confirm each names a reviewer and records an outcome, not just "reviewed". Re-performance: for one month, independently redo the review from the raw access logs and check that the documented review would have caught an anomaly you can see, which is what separates a real review from a signature. Inquiry (asking the Beta team) and observation (watching one review) are too weak alone: inquiry reports what is supposed to happen, and one watched review says nothing about the other eleven months.
Period and sample. Test across a period long enough to show consistency, six to twelve months. Because this control runs only monthly, that period is a small population you can inspect in full; sampling is not needed here and only becomes necessary for high-frequency controls (a daily check over a year), where you inspect a representative set rather than every instance.
Pass versus finding. Pass: a review is evidenced for every month in the period, each with a named reviewer and a recorded result. Findings, graded: a missing month is an operating finding (the control is not being performed); a review that records "ok" with no comparison against the access logs is a design or quality finding (it runs, but cannot catch what it exists to catch); a systematic absence (no reviews at all) is the major nonconformity of Lesson 3.3.B, while an occasional gap is a minor one, closed with a corrective-action plan. The lesson underneath: a control can pass design and fail operation, or the reverse, so a complete test states which property it proves and produces evidence, not reassurance.
Key takeaway
A control has two separate properties: design effectiveness (would it work if performed?) and operating effectiveness (is it actually performed, consistently, over time?). A control can pass one and fail the other, which is how organisations are compliant and exposed at once. Test with rising rigour, inquiry, observation, inspection, re-performance, and never rely on inquiry alone for operating effectiveness, which needs inspection or re-performance across a period and a sample. An untested control is only an assumed one, and the test is how a control's promised risk reduction is shown to be real.