The Night a Veterinary Group's Imaging Archive Went Dark

One multi-site veterinary group lost access to its imaging archive overnight. Here's what actually happened — and what vet PACS support should catch first.

At 1:40am on a Saturday, a technician at a six-site veterinary group tried to pull a prior study on a dog admitted with suspected obstruction. The viewer returned nothing. Not an error — nothing. Within twenty minutes it was clear that no study taken anywhere in the group over the previous eleven days could be retrieved. This is a composite account of a failure pattern we see repeatedly, and of the vet pacs support checks that should have caught it long before 1:40am.

Nothing about the night was exotic. Every step in the chain failed in an ordinary, well-documented way. That is the point: the incident was not caused by anything unusual, it was caused by nobody watching. RAD365 provides veterinary PACS support as a systems and infrastructure service — archive, routing, integration, monitoring and uptime — and the sequence below is what unmonitored infrastructure looks like from the inside.

Day one: a routine service visit

Eleven days earlier, an engineer visited the group's largest clinic to replace a failing DR detector. The work was clean and the modality passed its test image. What the visit also did — as service visits routinely do — was reset part of the device's network configuration. The modality's send destination reverted to a default that had been correct two archives ago.

Studies from that device stopped reaching the archive that afternoon. They did not vanish; they queued in the modality's local cache, exactly as designed, waiting for a destination that no longer answered. No alarm sounded because nobody had configured one, and the technologist saw studies appear on the acquisition console as normal.

Day four: the storage volume quietly fills

Meanwhile the central archive node was approaching a write ceiling nobody was tracking. Imaging volume across the group had grown roughly a third year on year — three new sites, a new CT — and headroom had been assumed rather than measured. On day four the volume crossed its threshold and began rejecting writes intermittently, then consistently.

This is the failure that does the real damage, because it is partial. Some studies wrote, some did not. Staff at three clinics reported "the system being slow" through the week, and each report was closed as a network issue by the group's general IT provider, who had no visibility into the imaging chain and no reason to connect three separate tickets.

Day seven: the backup that had been failing for months

The nightly backup job had been returning failures since a credential rotation months earlier. It had been configured to email a mailbox belonging to a practice manager who had left the group. The job ran, failed, and reported to nobody. On day seven, when the archive began rejecting writes, there was no verified restore point newer than the last time anyone had manually checked — which nobody could date.

1:40am: discovery

The overnight technician found the gap the way missing things are always found: by looking for one specific study and not finding it. The group's IT provider was reached at 2:20am, correctly identified the problem as outside its scope at 2:55am, and the PACS vendor's support line recorded a case for business-hours action. Clinical work continued without priors for the rest of the weekend.

Recovery took nine days. Studies still cached on modalities were recoverable and were forwarded once the destination was corrected and headroom restored. Studies from devices whose caches had rotated were not. The final gap was small — under two percent of eleven days of imaging — but it was permanent, and it included cases under active referral.

What should have caught it, and at which point

The five checks the group now runs monthly

1. Route verification after every service visit. Any engineer touching a modality triggers a send-destination test before sign-off.
2. Archive headroom against growth rate. Measured, not assumed, with alerting set months rather than days ahead.
3. Restore, not backup. A documented restore test with a date on it, quarterly, to a monitored distribution list.
4. Cross-site ticket correlation. Three "slow system" reports in a week from different clinics is one incident, not three.
5. A named overnight responder. With the runbook and the authority to act, per site, in writing.

Why multi-site groups are disproportionately exposed

Groups accumulate clinics faster than they accumulate standards. Each acquired practice arrives with its own modalities, its own local IT arrangement and its own undocumented configuration, and the imaging estate becomes a set of islands sharing one central archive that nobody owns end to end. A veterinary DICOM gateway fixes much of this structurally — centralised routing, tag normalisation, local queuing when a link drops — but only if someone is monitoring it. Where the boundary with a local IT provider is unclear, veterinary IT support should be defined in writing against the imaging chain specifically, and the study lifecycle itself managed through a documented veterinary imaging workflow rather than habit.

The uncomfortable part

The group had a support arrangement. It had a PACS vendor contract, a local IT provider and a practice manager who considered imaging covered. Every individual assumption was reasonable and the combination left an eleven-day blind spot across six clinics. That is the normal case, not the exceptional one — imaging infrastructure fails silently, and silence is indistinguishable from health unless something is actively looking.

Related Resources

• Veterinary PACS support from RAD365 — monitoring, uptime and incident response for practices and groups
• Veterinary DICOM gateway — routing, normalisation and store-and-forward across sites
• Veterinary imaging IT support — where the imaging boundary sits against general IT
• Veterinary imaging workflow management — study lifecycle from order to archive