Every OT site has a seasoned engineer. The “go-to” person for solutions, no matter how niche the question. This is the Subject Matter Expert, or SME, that you ask when the HMI starts acting strange, a historian stops collecting, or a vendor asks for remote access to a box nobody else recognizes.
This is the engineer who knows which PLC still runs the logic an integrator wrote in 2011 or why that engineering workstation talks to that controller at 2 a.m. They know which alarms are noise and which ones mean get in the truck.
This expert is not the problem, but it can become one when all of the solutions are in one spot.
The problem is when nobody else can do what they do or knows what they know. Nobody would accept a single point of failure in a safety-instrumented system. Plenty of OT security programs quietly accept one in their people, and it almost never shows up on a risk register.
Why OT piles knowledge onto a few people
This isn’t a management failure. It’s how OT environments are built.
Control systems outlive the people who installed them.
A DCS or PLC network can run for 20 years or more. The engineer who commissioned it might be the only one left who remembers why it’s configured the way it is. Every emergency fix, firmware workaround, and integrator visit that never made it into the drawings pushes the documentation further from reality.
OT security is rarely anyone’s whole job.
SANS’s 2025 ICS/OT budget research found only 9% of professionals spend all of their time on ICS/OT security. Everyone else splits it with operations, engineering, or IT duties. When security is a part-time responsibility, knowledge pools with whoever has been around longest.
The context is hard to write down.
Knowing that a Modbus write to a specific register is normal means knowing the process behind it. That kind of understanding is learned and comes from walking the plant and watching the system through upsets and outages. It doesn’t always fit neatly in a wiki page, and likely wasn’t documented on one.
What the data already tells us
The SANS State of ICS/OT Security 2025 report doesn’t ask about staffing head-on. However, its findings still point to a similar pattern across the data:
- Resources are the top blocker. Lack of internal resources (time, budget, personnel) was the number one reason organizations hadn’t fully implemented secure remote access controls, cited by 60% of respondents.
- Critical details aren’t written down. Roughly a third (31%) have no formal centralized inventory of active ICS/OT remote access points, or no inventory at all. If it isn’t recorded, it lives in someone’s head.
- Recovery goes untested. Only about a third test or simulate OT-specific recovery, and 16% have never formally tested their resilience plans. An untested plan means recovery depends on whoever remembers how.
- Prepared programs spread the knowledge around. Organizations that described themselves as fully prepared were 66% more likely to include field technicians in their preparedness exercises.
The programs that feel ready are the ones that pull more people into the work. Less mature programs are prone to letting knowledge pool with a few experts, which results in a single point of failure when left undocumented.
What walks out the door with them
When a key person retires, takes a new job, or is simply on vacation the week something goes sideways, a knowledge gap tends to show up in four places.
| The Knowledge | What it looks like | What happens without it |
| Asset context | Which devices exist, which are critical, which ones you never touch during production | The inventory becomes a spreadsheet nobody trusts. Or worse, it doesn’t exist. |
| What normal looks like | Expected traffic, polling schedules, that 2 a.m. connection | Every alert turns into a judgment call without context; some risk getting ignored |
| Remote access paths | Which vendors connect, how they connect, and through what | Unintended access remains in place if no process exists, or if no one verifies revocation of permissions. |
| Recovery know-how | How to rebuild an HMI, where the good backup lives, the restart order | Restoration stalls while someone figures out the procedure from scratch |
Recovery is already the slow part. In the 2025 SANS data, 19% of incidents took more than a month to remediate. Losing the person who knows the restart order doesn’t make that number any better.
How to make the knowledge outlast the person
The answer isn’t cloning your best engineer. It’s getting what they know out of their head and into something the whole team can use and learn from:
- Build the inventory from the wire. Passive network monitoring shows what’s actually talking on the network. A spreadsheet from three audits ago only shows what someone thought was there.
- Baseline normal. Once expected communications are captured, “is this weird?” stops depending on who happens to be on shift.
- Map remote access and keep it current. Write down every path, every vendor, and who approves each connection. Then review it on a schedule, not just after an incident.
- Exercise recovery with the people who would do it. Pair your senior engineer with someone newer during tabletops and restore tests. Bring field technicians in too.
- Make knowledge transfer part of the job. Shadowing and walkdowns work better as a standing habit than as exit paperwork.
That’s a big part of how we think about Valkyrie. It passively monitors OT networks, builds the asset inventory from observed traffic, and captures what normal communication looks like. The context your most experienced engineer carries becomes something any analyst can see and act on. For sites where cloud or internet connectivity isn’t an option, Cygnet brings Valkyrie to the site on a self-contained flyaway kit.
Some gaps are about process, not tooling. Our Aesir professional services team runs OT tabletops and assessments that help turn tribal knowledge into plans people can actually follow.
One question to ask this week
If your most experienced OT engineer gave notice tomorrow, what would you lose first?
If an answer comes to mind right away, that’s where to start. Write it down, share it, and make sure at least one other person can handle it.
Your people are irreplaceable. Your program’s knowledge shouldn’t be.
Sources
- SANS State of ICS/OT Security 2025 (pp. 8, 14, 16, 19, 20)
It’s the operational context that lives in experienced engineers’ heads instead of documentation: which devices are critical, what normal network traffic looks like, how vendors connect, and how to recover systems after an outage.
Control systems often run for 20 years or more, outlasting the people who installed them. OT security is also rarely a full-time role, so knowledge pools with whoever has been around longest.
Build the asset inventory from observed network traffic, baseline normal communications, document remote access paths, exercise recovery with mixed-experience teams, and make knowledge transfer a standing habit rather than exit paperwork.