A staged malware-scanning pipeline that prevents downstream import until completion, scan, and move verification all pass.
Automated downloads land faster than anyone can manually inspect them — a single infected or corrupted file reaching a media library is worse than a slower pipeline.
01
Threat model
An automated downloader is treated as an untrusted ingestion service. Its output isn't part of the library merely because a transfer completed — the risk being defended against is malicious or corrupted content reaching a media service before it has been scanned.
02
Trust boundary
Completed downloads land in a staging path that downstream applications cannot read. That boundary is enforced at the filesystem level, not just by application ordering, so a gate failure leaves content invisible rather than silently accessible.
03
Architecture
An explicit state machine — DOWNLOADING → COMPLETE → SCANNING → CLEAN → VERIFIED → RELEASED — with any ambiguous or failed outcome routed to INFECTED/ERROR → QUARANTINED. No state is inferred from a filename, and a scanner timeout is never treated as an implicit pass.
Downloader → isolated staging → scanner → verification → release. On scan failure, the flow branches from the scanner to quarantine instead of release.
04
Failure modes discovered
An incomplete transfer treated as complete
A scanner outage silently skipped instead of blocking release
A malicious test file reaching the release path
Two workers racing to release the same item
A move operation reporting success without actually relocating the file
05
Controls implemented
A filesystem-level staging boundary downstream services cannot read
An explicit state machine with no implicit transitions
Destination existence and size verification before a release counts as complete
A bounded retry policy with fail-closed behavior on verification failure
Logging of identifiers and outcomes without credentials or full private paths
06
Lessons learned
The state machine was easier to get right than the verification step. The tempting shortcut is trusting a successful-looking move command — that's exactly the assumption worth testing first.
Engineering timeline
01
Initial assumption
A successful move command was assumed to mean the file was safely in its final location.
02
First implementation
The intake used an explicit state machine, but the release step trusted the move operation's reported success.
03
Failure discovered
Testing showed a move could report success without the destination file actually existing under load — an ambiguous outcome the design didn't account for.
04
Root cause
The release step checked the move command's return code, not the actual state of the destination.
05
Control added
A verification step was added: confirm the destination exists and matches the expected size before a release counts as complete, with a bounded retry policy.
06
Validation
Incomplete transfers, a scanner outage, a known-bad test file, and two workers racing the same item were all tested against the new verification step.
More detail
Every one of these was tested deliberately, not assumed safe — see the full failure-path testing in the guide.
07
Current state
Validated: every ambiguous outcome now leaves the file quarantined rather than released. Scan latency under heavy load remains a known, bounded limitation.
Evidence panel
Claim: Automated downloads land faster than anyone can manually inspect them — a single infected or corrupted file reaching a media library is worse than a slower pipeline.
Design
An explicit state machine — DOWNLOADING → COMPLETE → SCANNING → CLEAN → VERIFIED → RELEASED — with any ambiguous or failed outcome routed to INFECTED/ERROR → QUARANTINED. No state is inferred from a filename, and a scanner timeout is never treated as an implicit pass.
Test
Each failure mode above was exercised deliberately rather than assumed: interrupting transfers mid-flight, taking the scanner offline, submitting a known-bad test file, and running two workers against the same item concurrently.
Expected vs. observed
Expected: Every ambiguous or failed condition leaves the system in its safe state, never the exposed one.
Observed: A control is only proven when its failure mode is safe. In this design, every ambiguous outcome — a stalled scan, a race, an unclear move result — leaves the file isolated rather than released.
PassedLimitations
Scan latency for very large files can leave content quarantined longer than expected under load; there's a bounded retry policy, not a latency guarantee.