ADR-012: Bulk PDF upload — hybrid pipeline with on-prem processing agent

Hosting and release-pipeline correction (2026-08-30, tightened 2026-08-31): ADR-015 supersedes Decision 2 and every staging/preview Kubernetes or PVC hosting statement in this record. It also supersedes the seven-day completed-object expiry and direct notifier-to-agent presigned-URL path in Decisions 1, 3, and 6, plus Decision 8's unversioned completed-object delete/absence branch. Those statements are historical context only and must not be implemented. The corrected target uses separate ARRNC API containers, a durable version-bound notifier release gate through Project Management, and fresh version-pinned download capabilities minted only in accepted agent claim responses. Multipart, scan/process, normalization, and application storage-authority decisions remain authoritative where ADR-015 does not replace them.

Status

In-Review. The architecture, including the multipart reliability refinements and provisional timing defaults, was approved by the product owner on 2026-08-11; that approval is not approval to merge a rollout wave or enable production. Supersedes the unmerged branch document ADR-008-bulk-pdf-s3-only.md from PR #2373 (which collided with main's ADR-008 numbering and was contradicted by that branch's own implementation).

Context

Bulk PDF upload (epic #2223) lets project admins deliver full-text PDFs themselves instead of emailing them to the SyRF team. Final PDF storage is the Edinburgh IIS web server (ecrf1.clinicaltrials.ed.ac.uk/camarades/, ~700 GB of PDFs already served from there), reachable only from inside the university network — nothing in AWS or GKE can write to it.

The feature's storage architecture flip-flopped across 2026: hybrid (S3 staging + on-prem final, discussion #2094, Oct 2025) → S3-only (branch ADR-008, Jan 2026) → hybrid again (decision C-2, Mar 2026, recorded only as a Teams confirmation). The contradiction was never resolved in a merged decision record. This ADR settles it.

Decision

Matched-only browser selection (2026-10-27)

The browser validates the selected folder using the existing signature, portability, collision and size limits, then archives only valid files matched to the authoritative study-path snapshot. One file matching multiple studies is included once. The session file count and transfer bytes refer to that archive. With no matched files, no archive or upload session is created. Snapshot identity and server fingerprint checks remain mandatory; a changed path set requires a fresh preview and archive, including on retry.

Unmatched files are shown as skipped before upload, with a browser-local downloadable list. They are never presented as copied or included in the server report. The existing server report continues to describe submitted entries and missing study paths, including legacy uploads that contained unmatched entries. No old jobs, files, delivery markers or reports are rewritten. The existing bulkPdfUpload flag gates this correction; no additional flag or activation is needed.

Pipeline and storage

1. Final storage stays on the Edinburgh IIS share. S3 (syrfapp-uploads*) is staging only. Per ADR-015, completed Bulk PDF object versions have no age-based expiry while a durable release, claim, retry, or retention row can reference them; Project Management persists a deletion-independent immutable release record before acknowledging the object-ready command and uses it for every initial/retry claim. A terminal-before-publication hold instead receives a non-claimable cleanup-only record through the stable cleanup-registration command; it never emits agent work. The scheduled reconciler reclaims an exact version only after terminal/no-retriable-claim proof and snapshotted deployment-owned retention (PT24H staging, PT1H preview), followed by authenticated cleanup acknowledgement that converts the active release record to a non-authorizing audit receipt. The reconciler persists ReceiptRetirementPending plus the immutable delete/ack result, then one transaction atomically creates HoldRetirementOutbox and converts the stable capture key in place to replay-fenced CaptureTombstone. Only that outbox obtains and durably records Project Management retirement confirmation before deleting itself; receipt purge separately requires OutboxDrainedAt from an exact-key absence proof. TTL is never absence proof. Incomplete multipart uploads retain a separate lifecycle cleanup rule as a safety net.
2. One processing locus, on-prem. A new .NET worker (PDF agent) performs download, ClamAV scanning (clamd companion container), validation, extraction, conflict handling, and the final filesystem write. Per ADR-015, production, staging, and dynamic PR previews use separate Docker containers on arrnc-api with exact environment-child mounts. No PDF agent runs in Kubernetes.
3. The Lambda stays thin and capability-free. The existing s3-notifier gains a BulkPdfUpload kind: it validates the exact object version and durably sends a stable object-ready command to Project Management only after the release gate permits it. Neither the notifier nor a durable command creates or stores a URL. Project Management verifies the exact version, then sends stable work identity to the agent; every accepted/idempotently accepted agent claim receives a freshly minted version-pinned GET. No scanning or extraction occurs in Lambda.
4. Domain authority stays in the PM service. The agent reports per-file outcomes; PM performs study matching, marks studies, computes counts, and generates the CSV report in a single-writer consumer.
5. The browser builds the ZIP; the server owns its multipart upload session. Users select a folder of PDFs; the client validates (PDFs only), previews matches against the search, and zips in a web worker (STORE). It initiates a PM-backed S3 multipart session bound to exact size and whole-object SHA-256, then uploads parts directly to S3 using short-lived presigned UploadPart URLs whose SigV4 signature binds the exact part Content-Length and SHA-256. The browser uploads an exact known-length Blob.slice; its user agent supplies the forbidden Content-Length header, so that header is not returned as a JavaScript-settable RequiredHeaders value. The API does not proxy bytes.
6. No cloud credentials or inbound connections on university hosts. The agent pulls S3 through the fresh version-pinned URL returned by its accepted Project Management claim and uses AMQP outbound only. It never receives AWS credentials or a notifier-published capability.
7. PM owns upload liveness and completion. A configurable 20-minute lease is renewed by a 60-second heartbeat and after the backend verifies a completed part; part URLs have a separate 5-minute request-start validity. The API also advertises a configurable 10-minute maximum client UploadPart request duration; it is not the URL-expiry window. Only the backend calls CompleteMultipartUpload, after an atomic Uploading → Completing transition and exact manifest verification. That transition atomically persists a durable completion-operation claim shared by the original API request and PM recovery. The claim is three minutes by default, the server-owned storage operation is bounded to two minutes, and every storage side effect and result mutation revalidates exact live ownership. PM excludes a live claim, takes over an expired one atomically, and applies the existing 60-second recovery grace to legacy Completing rows with no claim. A final claimed object HEAD is mandatory before PM may enter Aborting. These accepted defaults remain subject to the pre-launch slow-network/background-device timing gate.
8. Terminal state converges monotonically. Abandon, cancel, and lease expiry atomically enter Aborting; no heartbeat, stale callback, reconnect, or late successful part may restore Uploading or trigger completion. The API persists the latest issued part-URL expiry before returning that signature. After entering Aborting, PM promptly issues AbortMultipartUpload, but cannot count an absence observation until max(AbortStartedAt, LatestSignedPartUrlExpiresAt) + MaximumUploadPartRequestDuration. With the accepted five-minute request-start validity and ten-minute maximum active request, the worst-case initial quiescence boundary is therefore 15 minutes, not a fixed post-abort grace. PM repeatedly aborts, lists parts, and checks the staging object. An objectless session requires two joint empty/NoSuchUpload MPU and object-absence observations at least one 30-second sweep interval apart; any non-empty MPU resets that proof. A present completed object instead stops the absence branch: both the sweeper and BulkPdfUploadController.CancelUpload record cleanup- pending while active capacity remains charged. ADR-015 durable notifier capture supplies the immutable exact version; Project Management atomically persists/reuses CleanupOnly, terminalizes the job, and only then releases capacity. Neither path performs an unversioned delete, creates a delete marker, or records object absence while a noncurrent exact version survives. Objectless proof permits Abandoned/Cancelled; completed-object history and Project deletion remain fenced until the matching cleanup receipt. Only legacy jobs that truly have no MPU may terminalize on direct authoritative absence proof. Each cleanup pass first atomically acquires a durable, separate exact-token abort-sweep lease (three-minute default) and is bounded to a shorter storage timeout (two-minute default); active claims are not selected, expired claims are safely taken over, and every storage call and proof mutation revalidates the unexpired claim. Normal passes release promptly; process/host cancellation safely leaves the claim for expiry takeover rather than delaying shutdown. The abort-sweep lease is not reused as the completion-operation lease. Conversely, a winning Completing/Uploaded transition rejects late abandon. Retry v1 creates a fresh independent session; it is not resumability.
9. Processing requires a PM claim before side effects. After object completion, the agent's first operation is an atomic PM claim of the exact upload/size/hash binding. Only an accepted claim returns the opaque token required on progress and finalize messages; download, scratch, sentinel, scan, and copy all happen after the claim.
10. The browser renders truthful, reconciled state. Uploading means an UploadPart request or valid bounded retry is actually active. Recoverable loss is shown as retrying; terminal local failure stops animation, queues idempotent abandonment durably, and offers fresh Retry. PM state is reconciled on reconnect/reload through SignalR plus heartbeat/status and bounded polling fallback. A terminal server state cancels local work; a completion winner is rendered as completion.
11. Bulk storage authority is explicitly environment-owned. BulkPdfStorage:EnvironmentRoot has no default and accepts only the canonical values production, staging, or previews/pr-N (a positive, non-zero-padded PR number). A preview root must exactly match SyrfSettings:PrNumber; every root must also match the independently rendered RuntimeEnvironment and ActiveEnvironment exactly (production, staging, or preview). This catches a staging workload configured with the production root (and the converse) instead of treating the root configuration as self-authenticating. The API resolves this setting only after the default-off bulkPdfUpload gate is open, so disabled deployments remain healthy without it. Once the gate is on, absent, malformed, or deployment-mismatched authority returns typed 503 rather than falling back to a shared or inferred root. New sessions persist the canonical root immutably in the PM job and in the S3 object/MPU metadata. Status reads do not grant storage authority; every signing, heartbeat, completion, abandonment, retry, or other storage-mutating operation requires the current root to exactly equal the job root. A legacy/rootless job, or a job from another root, is rejected fail-closed (409) for mutation and is never adopted by a later deployment.

The accepted environment root is also carried unchanged on every agent progress and finalize command. The PM consumer requires an exact three-way match between message, persisted job, and resolved local authority before any state, CSV, or Study effect. A deterministic authority rejection faults to the MassTransit error transport without entering either retry policy; silently acknowledging a one-way command on the wrong deployment would lose it. Retryable aggregate races and transient Mongo/network failures use five bounded 200 ms immediate retries followed by durable publish-scheduler redelivery after 5 seconds, 30 seconds, 2 minutes, 10 minutes and 30 minutes. Exhaustion reaches error transport; authority faults and ordinary defects never enter either retry policy. This is defense in depth, not broker routing: activation also requires environment-specific RabbitMQ queue routing or namespaces so competing consumers from another environment cannot take a command intended for the local PM instance. The PDF-agent publisher must be deployed with this contract before any upload is activated (or zero in-flight legacy messages must be proven), because the fail-closed PM consumer deliberately error-transports a follow-up whose environment authority is absent.

The always-running sweeper may converge only positively stale rootless records without storage authority. Queued and unmarked Expired records are immediately eligible. Uploading is eligible only after its persisted lease expires. Aborting is eligible only after any durable abort-sweep claim expires and the maximum part-request duration has elapsed from the later of abort start (falling back to creation) or the latest signed-part-URL expiry. Completing is eligible after an exact completion claim pair expires; a missing/partial legacy pair instead waits the configured completion-recovery grace from completion start (falling back to creation). Uploaded is eligible only when both processing-claim fields are absent and the same recovery grace has elapsed from upload completion (falling back to completion start or creation). Each accepted retirement moves the job to non-deletable Expired with a positive RetiredWithoutStorageAuthorityAt marker. Only that explicit rootless marker releases active capacity; rooted or unmarked Expired records retain capacity until ordinary storage-cleanup proof reaches Abandoned or Cancelled. Retirement never claims, aborts, deletes, reads, or otherwise touches storage, and a sweeper lacking exact local deployment identity performs no retirement or storage work.

CSV report reads are storage access and therefore require the loaded job's root to match the local authority before the file stream is opened. Study PDF path/link reads remain an activation dependency: the availability/provenance work must prove that a new Bulk-owned PDF physically exists beneath the local environment root before exposing it. Existing/manual legacy paths are not adopted or reclassified by this release.

This is an application authority contract, not evidence that Kubernetes mounts, CIFS export permissions, static serving, or S3 prefixes are isolated. Those controls require focused server-config/GitOps/hosting work and independent validation. Until that work exists, the feature remains default-off; this decision neither creates directories nor authorizes a deployment or storage cleanup. 12. Bulk PDF path keys use a pinned cross-runtime Unicode contract. The browser preview, PM finalizer/link builder, and PDF agent all apply NFC → the narrow existing typographic punctuation fold → Unicode 16.0.0 default full case folding → NFC → existing separator and marker rules. The checked source is PdfPathNormalization/CaseFolding/CaseFolding-16.0.0.txt, SHA-256 6f1f9c588eb4a5c718d9e8f93b782685e5c7fec872cf05e8e6878053599e09bb; the checked generator emits one JSON mapping consumed by both C# and TypeScript. It selects the Unicode F mapping when present, otherwise C, and excludes S and locale-specific Turkic T mappings. Runtime ToLowerInvariant()/toLowerCase() are not an acceptable substitute. The full fold remains only a matching/collision key: path traversal, Windows portability, component byte budget and canonical destination-path validation remain independent, fail-closed safeguards applied after normalization. Updating Unicode data requires a separately reviewed mapping version, checksum, generator run, shared vectors, C#/web/agent validation, and explicit compatibility review. 12. An upload is bound to the exact Study PDF-path set that was previewed. The API returns the path list with protocol version study-pdf-path-set-v1 and a server-computed SHA-256 fingerprint. Initiate and fresh Retry must present that exact binding; a changed set is rejected with typed study_pdf_paths_changed before storage initiation. The binding hashes strict UTF-8, length-prefixed tuples of Study ID and the exact raw PdfRelativePath, ordered by Study ID. PM recomputes it at finalization before report or Study writes, then applies each delivery marker with an exact project/search/study/raw-path compare-and-set and verifies the marker postcondition. Each string is prefixed by its four-byte unsigned big-endian UTF-8 byte length; the protocol version is the first field and Study IDs use lowercase D format. Drift fails the job truthfully and requires a fresh preview and Retry; it never links delivered bytes to a Study path the user did not preview. Jobs persisted before the binding fields existed remain readable and retain legacy finalization during rolling deployment, but every newly initiated session requires the versioned binding.

Unicode path-key consumers and rollout boundary

The contract is deliberately one matching key, not an alternative filesystem path rule. Its consumers and boundaries are:

Layer	Consumers	Boundary protected by the same key
Browser	pdf-path-normalization.ts, folder-selection.ts, match-preview.ts	selected folder entry vs the search's Study.PdfRelativePath, preview collision/portability result
Shared kernel	PdfPathNormalizer, PdfPathCollisionDetector, PdfPathPortability	one canonical match key; collision and Windows/CIFS validation remain fail-closed
PM	BulkPdfUploadFinalizeConsumer, PdfLinkBuilder	agent outcome vs study match; immutable delivered-path link validation and CSV normalized_path
PDF agent	ZipEntryPlanner	raw ZIP entry, pre-write collision detection, path portability and canonical destination relative path

The artifact foundation must land before the web mirror can import it. The web change is a required dependent change: replace its current runtime toLowerCase() step with the shared JSON lookup and a second NFC operation, with no locale-sensitive alternative. The agent inherits the shared-kernel implementation once rebuilt against it. The normalizer is feature-path scoped and Bulk PDF remains default-off; nevertheless, no Bulk activation is permitted until the browser, PM and agent all run the same checked artifact and the shared vectors pass in each relevant suite.

Consequences

Positive

• Zero broken links: a study's PDF link activates only after the file is verifiably on disk.
• One place to debug processing; no cross-cloud two-hop pipeline (the #2373 shape).
• No clamd-in-cloud. ADR-015 deliberately adds the durable hold/reconciler and exact-version IAM contracts required to make release and cleanup safe while retaining the repository's supported Lambda network shape. A dedicated notifier VPC, DNS firewall, or stateful egress firewall is optional future hardening, not an activation dependency for this migration. The existing Terraform-owned SyRFS3NotifierLambdaBoundary still needs the narrow IAM-only hold-table and exact-version-delete cap defined by ADR-015; that prerequisite grants no network authority.
• Preview/staging environments run the identical ARRNC Docker contract, so the on-prem production shape is E2E-testable before production promotion.
• ClamAV signatures stay current via freshclam without image rebuilds.
• A killed browser, powered-off device, or severed connection cannot leave an upload Pending on the assumption that the original non-resumable transfer may spontaneously continue.

Negative / accepted

• Production depends on a single manually-applied host (arrnc-api) and a CIFS mount; mitigated by fail-closed sentinel checks, durable queueing while the agent is down, and server-config's recovery/verify roles.
• Unscanned bytes touch the agent's isolated container before scanning: the ZIP is downloaded and extracted into container-scratch space, and every extracted file is scanned there before anything is copied to the share or any served location. The security boundary is "nothing unscanned ever reaches final storage", not "nothing is extracted pre-scan" — scratch is private to the agent container (same trust boundary as scanning inside a Lambda).
• AMQP to rabbitmq.camarades.net:5672 is currently plaintext over the internet, and this applies to both out-of-cluster clients: the existing production Lambda and the new PDF agent on arrnc-api. A separate hardening PR adds AMQPS; both flip their URIs when it lands.
• The bulk layout (projects/{p}/searches/{s}/pdfs/…) coexists with the legacy flat layout; link generation branches on Study.BulkPdfDeliveredAt.
• Multipart upload introduces a PM sweeper and state/storage reconciliation path. Capacity is deliberately retained through the derived latest-signed-URL quiescence boundary and two-observation cleanup proof. The incomplete-MPU lifecycle rule is only a fallback; completed object versions follow ADR-015's durable terminal reclaimer, so transient S3 errors cannot manufacture a terminal state or expire a still-referenced payload.
• The accepted timing defaults trade prompt interruption detection against slow links and browser throttling. Staging must validate slow 1 GB upload, prove each exact 16 MiB part completes safely inside the advertised 600-second request timeout, and exercise transient loss, heartbeat jitter, and device sleep/background behaviour before production enablement; unsafe evidence requires an explicit narrow timing decision rather than silent retuning.

Alternatives considered

• S3-only storage, no agent (branch ADR-008; revisited 2026-08-11 as "GKE worker + S3 final"): cleanest platform-native end-state, but changes the PDF serving model and contradicts the standing storage decision; rejected consciously.
• Keep #2373's two-hop pipeline (scan-in-Lambda → safe bucket → agent): requires clamd reachable from AWS (new always-on infra), doubles S3 I/O, and splits processing across a cloud/on-prem boundary; dominated once an on-prem writer exists anyway.
• Direct upload to an ingest service on arrnc-api (no S3): loses durable staging, presigned upload UX, and per-PR preview isolation; couples availability to one host.
• One presigned PUT plus read-time/24-hour expiry: the server cannot know promptly that the browser's non-resumable request is irrecoverably severed, cannot make continuation impossible, and may strand active capacity. Rejected in favour of renewable server-owned multipart state.
• Browser-owned multipart completion: allows a stale client to race terminal abandonment and publish an object after PM has ended the session. Rejected; completion is backend-only.
• Resume/adopt prior parts on Retry: adds cross-session trust and ownership complexity. Rejected for v1; Retry is explicit and fresh after authoritative cleanup.

References

Design: docs/superpowers/plans/2026-08-11-bulk-pdf-upload-v2-design.md · Plan: docs/superpowers/plans/2026-08-11-bulk-pdf-upload-v2-plan.md · Epic #2223 · discussions #2093/#2094/#2095 · superseded attempt PR #2373 · ADR-010 (s3-notifier IAM via ACK) and its 2026-08-11 audit.

Serving and availability amendment (2026-10-27)

Upload, study attachment and retrieval are separate outcomes. A successful filesystem transfer and MarkBulkPdfDeliveredAsync.AllDeliveriesVerified prove neither HTTP retrieval nor viewer readiness. The latter verifies database markers, not the PDF endpoint.

The bulk-pdf-delivery-v1 processing-claim protocol fixes a physical-name version at first claim. Version zero preserves an already-claimed legacy job's destination. Version one preserves ordinary normalized paths but maps paths containing percent text, plus signs, or the reserved __syrf_encoded_v1__/ prefix to that directory plus the SHA-256 of the normalized matching key and .pdf. The matching key and CSV filename stay unchanged. URI escaping still happens exactly once per physical path segment in PdfLinkBuilder; neither filenames nor bibliographic paths are URL-decoded. This avoids IIS double-escape rejection without weakening its request filter. Hash names bound component length and prevent a literal reserved-path input from impersonating an alias.

An upgraded agent requires the explicit protocol response before any processing side effect. A claim's version never changes on replay, including retained copy journals. Older agents defer version-one jobs. PM rejects a version-one finalization without the negotiated protocol marker and expires ordinary retry authority; support must confirm the delivery protocol before recovery. Upload admission remains behind the existing bulkPdfUpload flag; there is no additional runtime flag or data migration. Changing that flag does not migrate filenames and does not stop accepted jobs from converging.

After durable delivery, the agent verifies the canonical, environment-scoped URL supplied by PM's trusted storage configuration. It requests only bytes 0–4, requires HTTP 200/206 and %PDF-, follows no redirects, forwards no credentials, disables URL logging, and reads at most five bytes even if the server ignores Range. A rolling pool runs at most four checks at once, so a slow response does not hold up unrelated files. Each check has a ten-second timeout. The whole verification budget allows ten seconds per four delivered files, with a two-minute floor and a thirty-minute ceiling. Scratch lease heartbeats continue throughout, and shutdown cancellation still stops every worker. A timeout, missing authority, transport error or exhausted budget means unverified, never available. This is a retrieval check at completion, not a continuing uptime promise.

PM attaches matching studies, then summarizes retrieval for unique matched delivered files. Multiple studies sharing a file count once. Any unavailable/unverified result produces a failed terminal job with upload counts and attachment evidence retained. Zero eligible matched files instead produces an input-specific failure asking the user to check matching and PDF validity, with ordinary retry retained. Retrieval is exposed as viewer-access evidence only after the study markers are confirmed. A conflicting immutable marker produces a repair-required failure, keeps its original link, and expires ordinary retry authority in the same terminal write; a re-upload cannot repoint it. The UI distinguishes partial PDF access from a transfer failure and does not offer the ordinary retry action for access failures. Historical jobs without evidence say access was not verified. CSV uses uploaded for the internal Copied outcome; protocol enum names and counters remain stable.

Rollout and rollback — require separate authorization

1. Review and merge the application PRs only when authorized. No deployment, flag change, ARRNC apply, service restart, remount or incident upload retry is authorized by this amendment.
2. At an approved rollout window, stop admission and drain existing processing as the established operational procedure permits. Upgrade all PM/API aggregate writers before the agent. Do not run old aggregate writers alongside version-one claims: an old writer could discard new fields.
3. Compare PM's and API's effective BulkPdfStorage:PublicBaseUrl and environment root for each target environment; require exact agreement before upgrading the agent. Both resolve through BulkPdfStorageSettings.ResolvePublicBaseUrl, but per-service deployment overrides can drift. For a synthetic delivered path, compare the agent's canonical check URL with the API-generated viewer URL and stop if they differ. Upgrade the agent and UI, then use synthetic PDFs containing literal %2F, %28, %29 and +, plus ordinary names. Confirm safe physical destinations, attachment paths, retrieval summary and viewer URL. Prove a synthetic 404 produces a failed/partial access result with its successful transfers retained.
4. Resume admission only after that authorized proof. Disabling admission later leaves in-flight processing active. Before downgrading PM or the agent, drain every version-one job and preserve its journal until finalization. Existing study delivery paths already contain their physical alias, so older URL builders can serve them without reverse mapping.

Existing incident attachments — recovery plan, not executed

The October 26 staging attempt at 11:40 UTC reported 30 files uploaded, 17 unmatched and 13 matching study attachments. Scoped read-only checks found all 13 canonical URLs rejected with IIS 404.11 through both direct and proxied retrieval. Their stored names contained literal percent-hex text; ordinary/single-escaped synthetic controls reached 404.0. This identifies a request-filter rejection before filesystem lookup. It does not independently prove current physical file presence.

The browser preserves selected filenames into ZIP paths. Agent and PM normalize those paths without URL decoding; PdfLinkBuilder then correctly escapes literal percent signs. EndNote's internal-pdf:// parser also retains percent text after removing the scheme prefix. The evidence therefore locates the problem at the filename/matching boundary; it does not establish who originally created those names.

Do not retry the historical upload or repoint its immutable markers automatically. After separately approved recovery:

1. Privately snapshot the exact 13 study IDs, original delivered paths/timestamps, project/search bindings and existing job evidence. Use staging (syrf_staging) and C# legacy GUID subtype 3. Re-read the exact scope and stop if counts or markers differ. Keep filenames out of public logs.
2. On the verified storage root, stat each original file without changing it. Build a reviewed manifest mapping the stored normalized path through BulkPdfDeliveryPath.Resolve(path, 1). Do not unescape %2F or infer directories from filename text.
3. Preserve originals. Copy each file to its alias using the existing confined, durable destination writer behavior; verify size and SHA-256. Stop on an existing alias with different bytes, symlinks, missing sources, authority/sentinel failures or any scope change. Do not remount as part of recovery.
4. Probe each alias's canonical URL through the actual serving path and application proxy. Require successful PDF signature retrieval for all 13 before any marker change. Record only opaque IDs, hashes, sizes, HTTP outcomes and times in the reviewable evidence.
5. In a separately reviewed repair, compare-and-set only each exact study's BulkPdfDeliveredPath, requiring its old path, timestamp and project/search binding still match the private snapshot. Preserve bibliographic paths and historical job counts. The normal uploader cannot perform this repair because delivered markers are intentionally immutable. Abort on any failed comparison; reconcile a partial repair from the manifest rather than retrying blindly.
6. Re-read every repaired marker and generated viewer URL, repeat scoped retrieval checks, and retain originals plus the manifest for rollback. A rollback must compare against the exact newly written path before restoring the old marker; the old URL will still have the original serving failure. Remove no historical files in this change. Historical Complete jobs remain explicitly unverified; the repair evidence is a separate operational record, not a retroactive upload success claim.

Ordered follow-up PRs after the incident fix

The matched-only Opus review approved that slice without blocking findings. Keep its optional refinements out of the delivery path:

1. Preview accessibility and resumed-count coverage: PR #4574 implements the warning status announcement, reattachment regression and derived matched-byte total. It remains a separately reviewed follow-up to the already merged matched-only fix.
2. Folder-admission policy: evaluate limits on the matched subset separately from limits on inspecting the whole selected folder. The current 10,000-file and 1 GB whole-folder limits are intentionally preserved, including validation and portability/security checks. A large folder containing only a small matched subset still follows those established limits in this release.
3. Outcome guidance and bounded verification throughput: implemented in the serving fix after the owner requested the review follow-ups: distinct zero-eligible-file guidance, no ordinary retry for an unconfirmed protocol, and a bounded rolling pool with a file-count-aware budget. Regression tests cover invalid/unmatched input, protocol failure, slow requests and shutdown cancellation.
4. Maintainability cleanup: simplify claim/finalization methods, extract nested conditional expressions in finalization and UI presenters, and adopt the exception-helper idiom flagged by SonarCloud. The quality gates pass; these findings are maintainability issues, not reported bugs or vulnerabilities. Keep this refactor separate from the incident fix and retain its regressions.