Backup that actually restores: RPO, RTO and restore testing

ILS Networks An 8-part guide

Most backup systems “work” — until someone tries to restore. This guide covers how to define RPO and RTO, how backup targets follow from them, where backup systems fail silently, and what a restore test that actually proves your objectives looks like.

In almost every organization we walk into, the backup “runs”. There is a green report in the morning, there is storage capacity, there is someone responsible. What is almost never there is a precise answer to two questions: how much data is lost if the most critical system fails right now, and how long it takes before that system is usable again. Without those two answers, a backup system is a set of scheduled jobs — not a recovery plan.

The gap always surfaces at the same bad moment: after a ransomware event, after a storage failure, after an accidental deletion. That is when it emerges that the backup was written but is not application-consistent, or that the only surviving copy sits on the same storage that got encrypted, or that a full restore takes three days because nobody ever measured it. The write path worked. The read path was never tested.

This guide follows the order in which the decisions actually have to be made: business objectives first, then workload tiering, then targets and architecture, and only at the end — the proof. ILS Networks has been designing and operating backup and continuity systems in Israel since 2009, working with Veeam Backup & Replication, Cove Backup, Dell Avamar, and Proxmox Backup Server, with 24/7 support. In the Ariston Group project, for example, the backup and DR system was redesigned as part of the VMware-to-Proxmox migration — because a platform change is the moment old assumptions stop being valid.

01

The two numbers come before the product

Every downstream decision — frequency, targets, replication, budget — is a consequence of two numbers. If they are undefined, the architecture ends up being whatever was already in the rack:

  • RPO (Recovery Point Objective) is how much data you can afford to lose. NIST defines it as “the point in time to which data must be recovered after an outage.” In practice: the age of the newest usable copy at the moment recovery begins.
  • RTO (Recovery Time Objective) is how long you can afford to be down. It is measured from the incident to the moment the service is usable by users — not to the moment the restore job finishes. The distance between those two points is usually measured in hours.
  • Both numbers are business decisions with a price tag. A one-hour RPO on a database costs more in storage, bandwidth, and licensing than a 24-hour RPO. Ask the system owner what an hour of lost data is worth, and show them the cost next to their answer.
  • Separate the stated objective from the achievable one. The stated RTO is what the business wants; the achievable RTO is what your restore path, link capacity, and staffing actually deliver today. The gap between them is your project plan, not a position paper.
  • Add the two numbers people forget: the retention period (how far back you must be able to reach — relevant both to ransomware that sat dormant for weeks and to regulatory or legal requests), and the maximum tolerable downtime, which caps RTO from above.
  • Document the objectives per system, have the business owner sign them, and revisit annually. An undocumented objective quietly becomes whatever the last restore happened to achieve.
02

Tier your workloads — one RPO cannot fit everything

Negotiating objectives against hundreds of systems individually is a waste of time. Define three or four tiers and negotiate the tier instead:

  • Tier 1 — systems whose downtime stops revenue, production, or safety: near-continuous protection, and usually replication in addition to backup, because restoring from backup alone will not meet the RTO.
  • Tier 2 — important systems that tolerate hours: daily or several-times-daily backups, restored from the backup platform, no replication.
  • Tier 3 — systems that can be rebuilt: standard daily backup, longer retention, no availability mechanisms.
  • Map dependencies, not just servers. An ERP restored in 30 minutes is worthless if its database, its identity service, its file share, and its licence server come back four hours later. Tier the service, then tier every component it leans on.
  • Flag systems that require a specific recovery order: domain controllers, DNS, certificate authorities, and the backup platform itself come up before applications.
  • Do not skip SaaS. Mailboxes, files, and collaboration sites are not automatically protected just because they live in the cloud — responsibility for the data stays with you.
03

Choosing backup targets: 3-2-1 and what has been added since

The 3-2-1 rule was formulated in the early 2000s, before the ransomware era. It is still the right foundation — but no longer sufficient on its own:

  • The original rule: three copies of the data, on two different media types, with at least one of them off-site.
  • The modern restatement Veeam promotes is 3-2-1-1-0: add one copy that is immutable or air-gapped, and zero errors from automated verification of recoverability.
  • Match the target to the RTO rather than the other way round: a local disk repository for fast restores, a second copy on a separate system, and an off-site or cloud copy for the scenario where the whole site is gone.
  • Three terms that get conflated: off-site is not offline, and offline is not immutable. A replicated NAS in your second office is genuinely off-site — and reachable with the same credentials the attacker already has.
  • Object storage with Object Lock, a hardened Linux repository, and tape each fill the role of the “one” at completely different cost, restore speed, and operational effort. That is a decision, not a default.
  • Compute the real restore time across the link: if your largest tier-1 system comes back from the cloud, data volume divided by bandwidth is your RTO. In Israel this is usually the factor that decides whether cloud is a primary recovery target or a last line of defence.
04

Protect the backup itself: immutability and identity separation

In a ransomware scenario the backup is not a bystander — it is a primary target. An attacker who deletes backups before encrypting has turned the decision into a ransom payment:

  • The UK NCSC has documented cases where ransomware encrypted not only the original on-disk data but also connected USB and network storage holding the backups. A permanently connected backup is part of the attack surface.
  • An immutable copy is one that cannot be altered or deleted for a defined window — not even by a compromised administrative account. Veeam’s security best-practice guide states that “all retention copies should be protected through air gap or immutability.”
  • Set the immutability window against attacker dwell time, not against operational convenience. An intruder who sits quietly for weeks defeats a seven-day lock.
  • Separate identity: backup infrastructure should not authenticate against the same directory service it protects. A domain-joined backup server with administrative rights is one compromise that takes down both production and the recovery path.
  • Prefer designs where the backup server holds no standing administrative login to the repository storage, and enforce multi-factor authentication on backup management consoles.
  • Alert on the actions an attacker performs first: deleting jobs, shortening retention, changing repository settings, and disabling encryption. Those are security events, not operational ones.
  • Encrypt backups at rest and in transit, and keep the encryption keys somewhere that survives the incident. A key that exists only inside the encrypted environment is not a key escrow.
05

Where backup systems fail silently

The dangerous failures are not the ones that light up red. They are the ones that look entirely healthy right up to the first restore attempt:

  • Coverage drift: machines, volumes, and SaaS tenants created after the job was defined and never added to it. Reconcile the protected list against the actual inventory monthly — against the inventory, not against the backup console’s own list.
  • Application consistency: a disk-level copy of a database may restore, and may not. Verify that quiescing mechanisms — VSS, pre/post scripts, application-aware processing — actually run, and read their logs rather than the job status.
  • Surviving exclusions: someone excluded a large volume once to fit a backup window, and nobody removed the exclusion after the storage was expanded.
  • Retention versus reality: a job that has been failing for six weeks while an aggressive prune policy keeps deleting older points — until only one very old restore point is left.
  • Silent chain and media degradation: incremental chains, deduplication metadata, and ageing disks. Proxmox Backup Server runs SHA-256-based verification jobs, and its documentation recommends re-verifying monthly because physical drives degrade over time.
  • Alerts nobody reads: a report going to a departed employee’s address, or a green dashboard nobody opens. Define one human alert destination and test that it reaches a person.
  • The backup server itself is not backed up. Without a copy of its configuration and catalogue, restoring hundreds of machines from a raw repository becomes archaeology on exactly the worst day.
06

What a real restore test looks like

A “success” status on a backup job is a statement about the write path only. A restore test is a statement about the read path — and the read path is the one that will have to work:

  • Define a scenario, not a job: “the ERP database server is gone — restore it to a usable state.” Start a stopwatch and measure time to service, not job duration.
  • Restore into an isolated environment, not over the production machine. Automated verification — for example Veeam SureBackup, which boots machines from backup in an isolated environment and runs tests against them, or Proxmox Backup Server verification jobs — is the floor. The scenario test with people in the room is what proves RTO.
  • Cover the three restore types separately, because each takes a different code path and fails differently: a single file, a whole VM, and a full site or bare-metal recovery.
  • Test specifically the restore you least want to test: the oldest point in retention, the encrypted copy from the remote target, and a restore performed by someone who did not build the system, using only the documentation.
  • Record measurements against objectives: measured RTO, measured RPO, and what went wrong. If measured exceeds stated, either the architecture changes or the objective changes. Writing the number down and doing nothing is the actual failure.
  • Set a cadence: automated verification runs continuously, and a full scenario test at least annually for tier-1 systems and after every material change — a platform migration, a storage replacement, a new site link. The NCSC puts it plainly: backups “should also be regularly tested to check they work as expected.”
07

Virtualized workloads have their own rules

Hypervisor-level backup solves a great deal, but it is also a common source of wrong assumptions after any platform change:

  • Image-level backup covers the server; what makes databases and mail restorable to a consistent point is application-aware processing, not the existence of the image.
  • Changed-block tracking is what keeps incremental backups within a sensible window. After a platform migration or a major upgrade, confirm the equivalent mechanism is actually active instead of assuming it is.
  • On Proxmox VE, Proxmox Backup Server provides incremental backups with deduplication, client-side AES-256-GCM encryption and SHA-256 verification, plus remote sync that transfers only the delta — an off-site copy without shipping the full volume every night.
  • Veeam Backup & Replication supports Proxmox VE from version 12.2, so an organization arriving from VMware can stay on the backup platform its team already knows.
  • A snapshot is not a backup. A point-in-time image on the same storage system disappears with it, and a long snapshot chain degrades performance well before it saves anything.
  • Storage-layer replication protects against hardware failure — and faithfully replicates both data corruption and an attacker’s encryption. Always keep a separate point-in-time copy as well.
  • During a platform migration, keep both backup paths running until the new one has passed a restore test — not until the migration is declared finished. That is how backup and DR were handled in the Ariston Group project moving from VMware to Proxmox.
08

DR is a runbook and people, not a product

In a real incident, what determines RTO is usually not storage throughput but how long it takes to decide, to find a password, and to work out the order of operations:

  • Write a recovery runbook someone else can execute at 3 a.m.: order of operations, where credentials live, the IP plan, DNS changes, and reactivation of hardware-bound licences.
  • Fix the recovery order in advance: identity, DNS, network and connectivity, backup infrastructure, the data layer, applications, and only then the user access layer.
  • Prepare an out-of-band communication channel and a copy of the runbook that survives the outage — printed, or on a device that does not depend on the systems being restored. A DR runbook that exists only on the encrypted file server is a theoretical runbook.
  • Decide beforehand who declares an incident and who authorizes failover to a secondary site. The two hours spent in that meeting count inside your RTO.
  • For multi-branch organizations, a branch failure is not a data-centre failure: it is usually a connectivity event rather than data loss. Redundant links to branches are part of the continuity design — as, for example, in the ten Ariston Group branches running on redundant links.
  • Settle the hours question: a four-hour RTO means nothing if the incident starts on a Friday evening and nobody answers. ILS Networks provides 24/7 support for the environments it manages.

What this looks like in practice

ILS Networks

ILS Networks has been building enterprise IT infrastructure in Israel since 2009 — virtualization, networking, information security, backup, and business continuity — with 24/7 support.

Published: August 26, 2026 · Updated: August 26, 2026

When did you last actually restore?

A short professional call with the team: what is protected today, what your real RPO and RTO are versus the stated ones, and what it takes to prove recovery instead of assuming it. No commitment.

Your details are used to respond to your inquiry, as described in our privacy notice.