Nobody had attempted to restore from the backup in the eighteen months since the backup system was migrated to the new infrastructure. When the ransomware event required an emergency restoration of the primary database, the restoration procedure failed. The backup files existed. The database backup agent that was configured to restore them had been deprecated in the infrastructure migration and not replaced. The backup was there. The restoration capability was not.
The Difference Between a Backup and a Recovery Capability
A backup is a copy of data taken at a point in time. A recovery capability is the combination of the backup, the restoration procedure, the tools required to execute the restoration, the people who know how to use those tools, and the infrastructure that can receive the restored data. These are different things. Organizations with complete backups may not have complete recovery capabilities because one or more elements of the recovery capability have degraded, changed, or been lost while the backup continued to run.
The monitoring that confirms backup completion is monitoring the creation of the backup copy. It is not monitoring the recovery capability. A backup monitoring dashboard that shows green confirms that backup files are being written. It does not confirm that those files can be restored, that the restoration procedure is current, that the tools required for restoration are available, or that the infrastructure to receive the restored data exists and is sized appropriately for the volume being backed up.
Backup completion monitoring confirms that copies exist. Recovery capability validation confirms that those copies can be used when they are needed. Most backup programs have built the first. Fewer have validated the second consistently enough to catch the capability degradation that the infrastructure migration introduced.
How Recovery Capabilities Degrade
Tool Deprecation Without Restoration Testing
Backup systems depend on specific tools — backup agents, restoration utilities, database connectors — that are tied to specific versions of the systems being backed up. When the backed-up system is updated, migrated, or replaced, the backup tool version that created the backup may not be compatible with the restoration target. The backup files exist and were created correctly. The tool needed to read them is no longer the tool installed in the current environment. The incompatibility is not discovered until restoration is attempted.
Procedure Documentation That Has Not Been Updated
Restoration procedures documented when the backup system was implemented describe the procedures that were correct at that time. System migrations, tool updates, and infrastructure changes may have invalidated specific steps in the documented procedure. An operator who follows the documented procedure on an infrastructure that has changed may encounter steps that reference systems, commands, or interfaces that no longer exist.
Infrastructure That Has Changed Since the Last Restoration Test
Restoration testing confirms that the backup can be restored to the infrastructure that existed when the test was conducted. If the infrastructure has changed significantly since the test — a different storage platform, a different database version, a different network topology between backup storage and restoration target — the test result may not reflect the capability that exists in the changed infrastructure.
Personnel Changes That Took Restoration Knowledge With Them
Backup and restoration procedures that depend on individual expertise rather than on documented, testable procedures lose their operational reliability when the individuals who hold the expertise change. The person who built the backup system and knew its operational subtleties may have left the organization. The procedure they left behind may not capture the judgment and institutional knowledge that made the procedure work in practice.
What Validated Recovery Capability Requires
Recovery capability validation requires doing what backup completion monitoring cannot: actually attempting to restore from the backup to a test environment and confirming that the restoration produces a functional system state. This is the activity that most backup programs do not perform frequently enough.
The restoration test must be conducted: at a frequency that catches capability degradation before it produces a gap, using the current tools and procedures rather than the tools and procedures that existed when the backup system was implemented, in a test environment that is sufficiently similar to the production recovery target to surface compatibility issues, and by personnel who are not the individuals who built the backup system but who would be responsible for executing recovery in an actual incident.
The last condition is often overlooked. A restoration test conducted by the expert who built the backup system may succeed because the expert knows the undocumented subtleties that make the procedure work. A restoration test conducted by a different team member following only the documented procedure reveals whether the documentation is actually sufficient for recovery without expert intervention — which is the capability that is needed when recovery is required under incident conditions.
Test the restoration, not the backup. The backup is the data. The restoration is the capability. Validate both on the schedule that risk requires.
Restore from the backup quarterly. Use current tools. Use the documented procedure. Use a team member who was not involved in building the backup system. The gaps that appear are the gaps that will appear in the actual recovery event.
Enterprise practitioner perspective. Not legal advice. Part of the Deep Trust Governance Series by Verisq. Get the free weekly Breach Digest.