A staged malware-scanning pipeline that prevents downstream import until completion, scan, and move verification all pass.
Automated downloads land faster than anyone can manually inspect them — a single infected or corrupted file reaching a media library is worse than a slower pipeline.
01
Threat model
An automated downloader is treated as an untrusted ingestion service. Its output isn't part of the library merely because a transfer completed — the risk being defended against is malicious or corrupted content reaching a media service before it has been scanned.
02
Trust boundary
Completed downloads land in a staging path that downstream applications cannot read. That boundary is enforced at the filesystem level, not just by application ordering, so a gate failure leaves content invisible rather than silently accessible.
03
Architecture
An explicit state machine — DOWNLOADING → COMPLETE → SCANNING → CLEAN → VERIFIED → RELEASED — with any ambiguous or failed outcome routed to INFECTED/ERROR → QUARANTINED. No state is inferred from a filename, and a scanner timeout is never treated as an implicit pass.
Hover or focus a node to explore it.
Downloader → isolated staging → scanner → verification → release. On scan failure, the flow branches from the scanner to quarantine instead of release.
04
Failure modes discovered
An incomplete transfer treated as complete
A scanner outage silently skipped instead of blocking release
A malicious test file reaching the release path
Two workers racing to release the same item
A move operation reporting success without actually relocating the file
05
Controls implemented
A filesystem-level staging boundary downstream services cannot read
An explicit state machine with no implicit transitions
Destination existence and size verification before a release counts as complete
A bounded retry policy with fail-closed behavior on verification failure
Logging of identifiers and outcomes without credentials or full private paths
06
Lessons learned
The state machine was easier to get right than the verification step. The tempting shortcut is trusting a successful-looking move command — that's exactly the assumption worth testing first.
Engineering timeline
01
Initial assumption
A successful move command was assumed to mean the file was safely in its final location.
02
First implementation
The intake used an explicit state machine, but the release step trusted the move operation's reported success.
03
Failure discovered
Testing showed a move could report success without the destination file actually existing under load — an ambiguous outcome the design didn't account for.
04
Root cause
The release step checked the move command's return code, not the actual state of the destination.
05
Control added
A verification step was added: confirm the destination exists and matches the expected size before a release counts as complete, with a bounded retry policy.
06
Validation
A scanner outage was simulated against the new verification step and the release step never ran — the one failure path with a recorded observed result.
More detail
Incomplete transfers, a known-bad test file, two workers racing the same item, and a false-success move were reported as exercised too, but without a recorded observed result; each stays Documented until one is recorded.
07
Current state
Evidence across the five failure paths: 1 Validated · 4 Documented. The design routes every ambiguous outcome to quarantine rather than release. Scan latency under heavy load remains a known, bounded limitation.
Evidence panel
Claim: Automated downloads land faster than anyone can manually inspect them — a single infected or corrupted file reaching a media library is worse than a slower pipeline.
Design
An explicit state machine — DOWNLOADING → COMPLETE → SCANNING → CLEAN → VERIFIED → RELEASED — with any ambiguous or failed outcome routed to INFECTED/ERROR → QUARANTINED. No state is inferred from a filename, and a scanner timeout is never treated as an implicit pass.
Test
Each failure mode above was reported as exercised deliberately — interrupting transfers mid-flight, taking the scanner offline, submitting a known-bad test file, and running two workers against the same item. Only the scanner outage has a recorded observed result: the scanner was taken offline and the release step never ran. The other four were reported as exercised but have no recorded observed result, so they remain Documented until one is.
Expected vs. observed
Expected: A control is only proven when its failure mode is safe. In this design, every ambiguous outcome — a stalled scan, a race, an unclear move result — leaves the file isolated rather than released.
Observed (recorded):
A scanner outage blocks release instead of being skipped: A scanner outage was simulated deliberately and the release step never ran.
No recorded result:
An incomplete transfer is never treated as complete: Reported as exercised by interrupting transfers mid-flight. No observed result was recorded.
A known-bad test file never reaches the release path: Reported as exercised by submitting a known-bad test file. No observed result was recorded.
Two workers racing the same item cannot both release it: Reported as exercised by running two workers against the same item. No observed result was recorded.
A move that reports success without relocating the file is caught by verification: Reported as exercised along with the other failure modes. No observed result was recorded.
1 as intended · 4 without recorded resultLimitations
Scan latency for very large files can leave content quarantined longer than expected under load; there's a bounded retry policy, not a latency guarantee.
