Skip to content

MongoDB Database Reference

This document provides detailed reference information about SyRF's MongoDB database architecture.

Project-load concurrency

For member project loads, ProjectController.GetProject records project history on pmInvestigator and access on pmInvestigatorUsage. Each investigator save attempt loads an uncached aggregate inside the retry delegate. Repository caches are singletons and hold mutable objects, so writes must not reuse those objects across requests or retries. The usage delegate already reads directly from MongoDB through RetrieveOrCreate on each attempt.

MongoExtensions.SaveAsync filters by ID and the original version, then replaces with upsert enabled. A competing update can make that filter miss and produce E11000 on the existing ID. OnSaving increments the in-memory version before the write, even when it fails; retrying that same object can eventually match a newer version and overwrite the competing changes. Reloading and reapplying the operation preserves those changes. The unit of work also invalidates the repository cache on both successful and failed saves.

The project-load regression tests reproduce controlled contention against disposable local MongoDB, including preservation of another project view, a profile update, and a usage favourite. This demonstrates the failure mechanism; it does not establish the precise competing requests behind a staging incident. The existing three-retry limit and project authorization remain in place. This correctness fix is unflagged because it restores the intended concurrency protection without a new user workflow.

Database Instances

Atlas Clusters

SyRF uses two MongoDB Atlas clusters:

Cluster Environments MCP Server (for querying)
Prod cluster Production mcp__mongodb-syrf-prod__*
Preview cluster Staging + PR previews mcp__mongodb-syrf-preview__*

A local standalone container (mcp__mongodb-syrf-local__*) holds a backup snapshot of the prod cluster taken 2026-02-21. Useful for safe exploration without touching live data.

Environment Connections (Isolated)

Environment Database Cluster
Production syrftest ⚠️ Prod
Staging syrf_staging Preview
PR Preview {n} syrf_pr_{n} Preview
Local development Configurable Local or local container

Warning: The production database is named syrftest despite its name suggesting a test environment. Treat it with care at all times.

How the charts bind a service to a cluster (no default)

The api, project-management and quartz Helm charts have no default cluster or database: mongoDb.clusterAddress and mongoDb.databaseName default to "", and each environment's cluster-gitops values must set both (production and staging in <env>.values.yaml; previews in preview.values.yaml plus the mongoDb.databaseName=syrf_pr_{n} ApplicationSet parameter), or name a ConfigMap through mongoDb.configMapName. If either is unset, empty, whitespace-only or the literal "", the render fails, for example:

syrf-common: mongoDb.clusterAddress must be set by the environment values (or set mongoDb.configMapName); refusing to fall back to a default

The ApplicationSets use ignoreMissingValueFiles: true, so before this guard (issue #4231) a missing environment values file silently bound a service to the production cluster the charts used to default to. The guard is generated from required: true on the two env vars in src/charts/syrf-common/env-mapping.yaml (helper syrf-common.requireEnvValue). E2E and the local stack do not render these charts; they set ConnectionStrings__MongoConnection__* on the processes directly. Identity's separate mongodb.* values are unaffected.

The same charts (plus identity and web) also require an explicit runtime environment: see Runtime Environment.

Legacy Databases (Prod Cluster)

Database Name Purpose
syrftest PRODUCTION database — all live data
syrfdev Old development snapshot, mostly unused

Collection Architecture

Bounded Context Prefixes

Collections are named using a bounded context prefix derived from the entity's namespace:

// From MongoContext.cs
public static string GetBoundedContextCode(string? namespaceName)
    => namespaceName switch
    {
        null => throw new ArgumentException("AggregateRoot has no namespace"),
        _ when namespaceName.StartsWith("SyRF.ProjectManagement") => "pm",
        _ when namespaceName.StartsWith("SyRF.FileListings") => "as",
        _ when namespaceName.StartsWith("SyRF.LiteratureSearch") => "ls",
        _ => ""
    };

Collection naming formula: {prefix}{EntityClassName}

See MongoContext.GetCollection() at MongoContext.cs:154-163 for the implementation.

Project Management Collections (pm prefix)

The project management domain contains the core business entities:

Collection Entity Class Description
pmProject Project Systematic review projects with stages, memberships, questions
pmStudy Study Individual studies with screening, extraction, annotations
pmInvestigator Investigator User accounts and profile information
pmSystematicSearch SystematicSearch Literature search definitions linked to projects
pmDataExportJob DataExportJob Background export job tracking
pmStudyCorrection StudyCorrection PDF metadata correction requests
pmInvestigatorUsage InvestigatorUsage User activity and usage statistics
pmRiskOfBiasAiJob RiskOfBiasAiJob AI-assisted risk of bias analysis jobs
pmActivityClaimRevocationOutbox ActivityClaimRevocationIntent Durable intent for the targeted claim-revoked SignalR event (#3720), written with the release under reviewEligibilityPolicy. Owner-only data; delivered and abandoned intents expire after 7 days (TTL on DeleteAtUtc)
pmBulkStudyUpdateOperation BulkStudyUpdateOperation ADR-020 operation record of an all-or-nothing Bulk Study Update: the attempt-scoped guard (lease, generation), phase, input manifest and counts. Permanent, value-free. Created on first use; dark until bulkStudyUpdateAtomicApply
pmBulkStudyUpdatePlan BulkStudyUpdatePlanEntry (chunks of 100) ADR-020 validated plan: each study's planned version, field changes and hashes. Deleted when the operation ends
pmBulkStudyUpdateBeforeImage BulkStudyUpdateBeforeImage ADR-020 before-image of each changed study (touched fields, content hashes, restore state). Expires 30 days after the operation is terminal (TTL on ExpiresAtUtc); a quarantined operation keeps its before-images
pmRobRunOperation RobRunOperation Application-authority M5b: the all-or-nothing save of one batch risk-of-bias run (_id = RoB job ID): lease, generation, phase, outcome, counts. ActiveSearch (unique partial index) is present while it applies or rolls back, so runs in one search are serialised. Permanent, value-free. Created on first use; dark until batchRiskOfBiasAtomicApply
pmRobRunBeforeImage prior RiskOfBiasInfoV2 per study M5b before-image of each study a run wrote: its prior result exactly as stored (value, explicit null or absent) and restore state (Applied, Restored, Superseded). Expires 30 days after the run ends (TTL on ExpiresAtUtc)
pmInboxNotification InboxNotification The saved personal notification inbox (#3932, ADR-026): one row per recipient, kind and source occurrence (RecipientId, Kind, ProjectId, SourceId, CreatedAtUtc, ReadAtUtc, from N1 the additive Source sub-document, and from N2 the additive per-recipient DismissedAtUtc, ResolvedAtUtc and Resolution). Captured only while notificationInbox decides on and its indexes exist; flag off keeps every row, and the inbox API still serves people who have saved rows; a well-formed request from anyone else gets 404 (N2, M7). See below
pmImportNotificationAdmission ImportNotificationAdmission The import-completed notice's recipient, bound before the upload signature is returned (#3938, ADR-026): _id = the search ID, ProjectId, RecipientId, set once ($setOnInsert) and never changed. Written only while notificationInbox decides on and the inbox indexes exist. No secondary index (read by _id only). See below
pmNotificationEmailPreferences NotificationEmailPreferences One account's notification email settings (#3942, ADR-026 decision 13): _id = investigator ID, Revision, Categories (keyed by email category, each Mode and server-set EnabledAtUtc), MutedProjectIds, AnyEnabled, digest timing. Written only while notificationEmail (with notificationInbox) decides on and its indexes exist. See below
pmNotificationEmailDelivery NotificationEmailDelivery The email delivery ledger (#3942): one entry per saved inbox item (_id = the item's ID), RecipientId, Kind, Mode, State (ready, leased, dispatching, sent, suppressed, ambiguous), lease fields. Same admission as the preferences. See below
pmNotificationEmailDigest NotificationEmailDigest Daily digest windows (#3943, ADR-026 decision 16): one per recipient and local day (_id derived from both), with the frozen NotificationIds, the day's send time, State and DueAtUtc. Holds its members' ledger entries. Same admission as the preferences plus 0032. See below
pmNotificationFanOut NotificationFanOut Recorded notification fan-out intents (N6, #4586, ADR-026 decision 14): one per occurrence too large to capture inline, _id derived from the kind and SourceId, with the row template, a recipient selector, lease fields and the expansion cursor. Written only by an inbox capture while notificationInbox decides on; expanded only while it decides on and its index exists. Never deleted. See below
pmPublication Publication [TARGET - Phase 12] Approved future system-scoped bibliographic identity, paired with immutable project-scoped Citation values. Migration, privacy and cross-project authorisation remain design gates — see three-level data model

Schema-v0 conditional parent answers

Schema-v0 annotation targets normally persist boolean and single-option conditions in legacy fields. Multi-option schema-v0 conditions use the additive ConditionalParentAnswers subdocument defined in ADR-011. Because project writes replace the full document, all API and project-management writers must be on that compatibility release before such data is created; older writers are below the safe rollback floor. The compatibility release is the permanent minimum writer version; the temporary runtime gate has been retired.

Reliable creation times (createdAtUtc)

pmProject and pmInvestigator documents created from spec 6F.F2 onwards carry createdAtUtc, a UTC BSON date (whole milliseconds) set once by the constructor that makes the new aggregate and never changed (Project.CreatedAtUtc, Investigator.CreatedAtUtc). It is additive and unflagged:

  • Existing documents are never backfilled or rewritten. A document without the element reads as null, and a save writes no element (the member is IgnoreIfNull), so a record created before the field keeps exactly its stored shape. No migration, index or start-up work exists for it.
  • Project.CreationDate is unchanged (the server's local clock, DateTime.Now); createdAtUtc is the reliable value.
  • Clock: the writing host's DateTime.UtcNow at construction. Aggregates are saved by a whole-document replace (ReplaceOne, upsert), which cannot take a server-side $$NOW, and the in-memory aggregate must carry the value or its next replace would drop it. Hosts keep NTP time, so the skew against the database clock is the cluster's clock synchronisation bound (well under a second).
  • Rollback: a binary that predates the field reads it into Entity.ExtraElements ([BsonExtraElements]; neither class map is strict) and writes it back on its next replace, so a rollback neither fails nor drops it (BsonClassMapTests, CreatedAtUtcPersistenceTests).
  • Readers: the project authority read (ProjectAuthorityFacts.CreatedAtUtc) and the Investigator authority snapshot (AuthoritySnapshot.CreatedAtUtc, a projected field) carry it to the feature target context, where its only reader is an exact rollout Pause: a subject created at or after the pause is kept out (Managing Feature Flags). It is never an authorization input.

Notification inbox (pmInboxNotification)

The saved personal inbox (ADR-026). The collection is new and empty until notificationInbox is switched on somewhere. Its indexes exist only as pre-deploy migrations in PmMigrations (ADR-023); InboxNotificationRepository builds none at start-up, and every consumer (export capture, the inbox API, InboxInvalidationWorker) asks IInboxAdmission first, which keeps the inbox closed while any of them is missing: no capture, 503 on reads, no change stream. From N2 the inbox API serves, with the indexes present, every caller while the flag is on and only callers with saved rows while it is off (a well-formed request from anyone else gets 404); with the indexes missing it answers 503 while on and 404 while off.

Index (migration) Purpose
RecipientId_1_CreatedAtUtc_-1__id_-1 (0023) A recipient's list, newest first; serves the (CreatedAtUtc, _id) keyset cursor in order
RecipientId_1_ReadAtUtc_1 (0024) A recipient's unread count (ReadAtUtc null)
unique RecipientId_1_Kind_1_SourceId_1 (0025) One row per recipient, kind and source occurrence: a replayed capture converges on the existing row and keeps its read state; also finds the row a workflow resolves
RecipientId_1_ReadAtUtc_1_DismissedAtUtc_1_ResolvedAtUtc_1 (0026, N2) A recipient's unread-count candidates: {RecipientId, ReadAtUtc: null, DismissedAtUtc: null, ResolvedAtUtc: null}, equality on every key (absent fields index as null). Makes 0024 redundant (kept: the catalogue is append-only)
  • Writers. DataExportJobRepository upserts the export-completed row (_id = the export job ID) in the same transaction as the terminal export write, insert-only ($setOnInsert), only while the inbox is admitted. SourceInboxCapture (#3938) upserts the importCompleted row inside the import-completion transaction (Project Management) and the projectInvitation row inside the invitation save (API, known accounts only), keyed on the unique source identity (0025) with a new _id, insert-only. Each host decides once per operation, before the transaction, through IInboxAdmission. From N1 (C15 v2) every writer, the export and stage captures included, writes through the one capture path, InboxNotificationWriter, and each stores exactly the elements it stored before plus Source (below).
  • Source (N1, additive, nullable). A sub-document naming what the notice is about: Type (dataExportJob, systematicSearch, projectInvitation, stage), EntityId (the job, search, invitation or stage ID), ProjectId (null for a global kind, which also stores no top-level ProjectId), StageIds and TaskId (no element when unset). It is set once on insert and never updated. Rows saved by L1-L3 have no Source and are never backfilled (no data migration); every read works from the typed fields, so they list exactly as before. A build that predates Source keeps it in the aggregate's extra elements (rollback-safe), and Source itself keeps elements a newer build adds. No index covers it.
  • Import admission (pmImportNotificationAdmission). SearchController binds the uploader as the recipient (keyed by search ID) before it returns the upload signature, only while the inbox is admitted; a different principal or project for the same search ID gets 409. Completion reads the binding inside its transaction and never guesses a recipient when there is none (legacy, living-search or system imports, or uploads made while off). A binding whose project differs from the completing import's, or that has no recipient, is treated as none: no row, and a warning that names IDs only. A binding made while on whose import completes while off stays inert. Flag off writes and reads nothing here. Rollback leaves the rows; no earlier build reads them. InboxNotificationRepository.MarkReadAsync sets ReadAtUtc on the caller's own row.
  • Per-recipient state (N2, additive, nullable). DismissedAtUtc (set once by dismiss or clear all, removed with $unset by restore), ResolvedAtUtc and Resolution (Reason: decided, completed, withdrawn, expired, superseded; SourceTransition, for example accepted or declined; ResolvedByRef, an opaque actor reference that no API returns yet; extra elements kept). No element exists until set, and rows saved earlier are never backfilled. Dismiss, restore and clear all are recipient-filtered UpdateOne/UpdateMany calls on the caller's rows. Only a workflow resolves: InboxNotificationWriter.ResolveAsync updates the existing unresolved row matched on (RecipientId, Kind, SourceId) (0025) inside the source transaction, never upserting; today the invitation answer (projectInvitation, completed, accepted/declined) while the inbox is open. A build before N2 reads these fields into its extra elements and updates rows only field by field, so it keeps them.
  • Readers. InboxNotificationRepository (recipient-scoped list with a status filter, unread-count candidates, get), and InboxInvalidationWorker's change stream (one per API pod, only while admitted; it sends an empty InboxChanged hint to the recipient's group).
  • Flag off. Nothing captures, resolves or watches; rows are kept with their state. From N2 the inbox API still serves a recipient who has saved rows (one existence read on RecipientId, limit 1) and answers a well-formed request from anyone without any with 404 (M7).
  • Rollback. Down 0026 drops the N2 index (an N1 build does not use it); Down 0025–0023 drops the indexes their Up created; the rows stay (no earlier build reads them). Retention and account-deletion erasure are not built yet (follow-ups in the notifications plan).

Notification email (pmNotificationEmailPreferences, pmNotificationEmailDelivery)

Opt-in email for saved notifications (#3942, ADR-026 decision 13). Both collections are new and empty until notificationEmail is switched on somewhere. Their indexes exist only as pre-deploy migrations; neither repository builds one at start-up, and the preference API and NotificationEmailWorker ask INotificationEmailAdmission first (notificationEmail and notificationInbox from one decision capture, then the inbox index set 0023–0026 and migrations 0027–0030; the daily digest worker and the admission's check also need 0032 on pmNotificationEmailDigest, #3943, ADR-026 decision 16).

Index (migration) Collection Purpose
Mode_1_State_1_AvailableAtUtc_1 (0027) delivery The immediate claim: due ready (or expired leased) entries, oldest first. Adopts the identical index an earlier build of #3942 created at start-up
RecipientId_1_Kind_1_State_1 (0028) delivery Moving a recipient's ready entries of a category's kinds between immediate and daily
State_1_LeaseExpiresAtUtc_1 (0029) delivery Recovering dispatches whose lease expired as ambiguous
AnyEnabled_1__id_1 (0030) preferences The worker's keyset page over accounts with any category on
State_1_DueAtUtc_1 (0032) pmNotificationEmailDigest The digest worker's window claim (ready, or leased past its lease) and recovery of expired dispatches
  • Writers. NotificationEmailPreferencesRepository.TrySaveAsync (versioned; keeps unknown category keys; sets EnabledAtUtc itself; recomputes AnyEnabled). The ledger is admitted insert-only by the worker's discovery over saved pmInboxNotification rows, leased and acknowledged by the worker; dispatch admission increments the preferences' DispatchFence in the same transaction so a concurrent opt-out, timing change or mute conflicts with it. NotificationEmailDigestRepository prepares one window per recipient and local day in a transaction that holds its members' ledger entries (daily, leased, owner digest:{windowId}, no LeaseExpiresAtUtc), so the immediate claim and recovery never touch them; the window carries the lease, and its dispatch admission fences the preferences too. Preparation also finishes the ready entries of the daily kinds that no digest will take as suppressed, with Reason leftOutOfDigest (notice read, dismissed or resolved) or tooOld, so no entry stays ready for ever.
  • Readers. The preference API (the caller's own document) and the worker.
  • Flag off. Nothing reads or writes either collection; every row is kept and used again when the flag is back on.
  • Paused (NotificationEmailDelivery:Paused, the default): discovery still admits ledger entries; nothing is claimed or sent, and no digest window is prepared.
  • Rollback. Down 0030–0027 drops the indexes their Up created and keeps an adopted one; the rows stay (no earlier build reads them). Down 0032 drops the window index; windows stay, and an earlier build never claims a held entry. Retention and account-deletion erasure (windows included) are follow-ups (E32).

Recorded notification fan-out (pmNotificationFanOut)

Recorded fan-out intents (N6, ADR-026 decision 14). When one occurrence has more recipients than its kind may capture inline (never more than 200), the source transaction writes one intent here instead of the pmInboxNotification rows, and NotificationFanOutExpander writes the rows later in leased, bounded batches. The collection is new and empty until notificationInbox is on and such an occurrence happens.

Index (migration) Purpose
State_1_DueAtUtc_1 (0031) The expander's claim scan: pending intents by when they are next due
  • Document. _id = SHA-256 over inbox-fanout:v1|{Kind}|{SourceId} (recording the same occurrence again is an insert-only no-op); the row template (Kind, SourceId, ProjectId, Source, WritesStage, StageId, OccurredAtUtc); Selector (Type: frozenList with RecipientIds, or projectCapability; optional ProjectId, StageId, Capability); State (Pending, Completed, as strings); DueAtUtc (recording time, lease expiry while leased, or retry time); LeaseOwner, LeaseToken; Cursor (the selector's opaque position after the last committed batch); RowsWritten, Batches, Failures, LastErrorClass (an exception type, never a message); RecordedAtUtc, CompletedAtUtc. Unknown elements are kept on read (C16 floor).
  • Writers. SourceInboxCapture.RecordStageFanOutAsync (API and Project Management) records inside the source transaction. NotificationFanOutStore claims, advances and releases for the expander; every write after the claim is fenced on LeaseToken, and the cursor advance commits in the same transaction as the batch's rows.
  • Readers. Only the expander (API). An intent whose selector type the build does not know is never claimed.
  • Flag off. Nothing reads or writes the collection; intents keep their progress and resume when it is back on.
  • Rollback. Down 0031 drops the index; the intents stay (no earlier build reads them). Completed intents are kept; retention is a follow-up with the inbox's.

Source code locations:

  • Domain models: src/libs/project-management/SyRF.ProjectManagement.Core/Model/
  • Repositories: src/libs/project-management/SyRF.ProjectManagement.Mongo.Data/Repositories/

Runtime feature release collections (no prefix)

Runtime feature releases keep their data in the project-management database, written only by the API's SyRF.API.RuntimeAdministration module. Their indexes exist only as pre-deploy migrations in PmMigrations (ADR-023); nothing builds them at start-up, and each writer refuses to run without the indexes it needs. See Managing Feature Flags.

Collection Indexes (migration) Description
runtimeFeatureRelease {environmentId, status} (0012); unique {environmentId, featureId, releaseKey} (0021) One record per release ID, advanced by a version compare-and-set
runtimeFeatureReleaseEvent unique {mutationId, type} (0013); {releaseId, recordedAtUtc} (0014); {environmentId, featureId, recordedAtUtc} (0015) Append-only release events; the unique index makes each event exactly-once
runtimeFeatureReleaseOutbox unique {dedupKey} (0016); {state, nextAttemptAtUtc} (0017, the dispatcher's due scan); {releaseId, state} (0022, a release's rows by state) Announcement side effects queued by Apply. Done, Cancelled and Failed rows are kept permanently: no TTL index and no clean-up job
runtimeFeatureReleaseTestSink _id only (the deduplication key) Retired (D60 A, 26 Oct 2026): staging's stand-in for in-app delivery before D60, one row per notice. Nothing writes or reads it any more; existing rows stay until a later clean-up (staging only)
runtimeFeatureDiscoveryState _id only ("{userId:D}:{announcementId}"); no migration One person's seen and dismissed state for one release announcement (6E.3): userId, announcementId, releaseId, revision, seenAtUtc, dismissedAtUtc, updatedAtUtc (MongoDB time), no project field. Written by one upsert by _id from PUT api/account/announcements/{id}/state and read by _id $in from feature availability; only RuntimeFeatureDiscoveryStore names it (RuntimeReleaseCollectionsForwardRuleTests). Account deletion does not erase it yet: D27 option A, pending Chris, leaves erasure to the OD-6 subject-deletion work
runtimeFeatureConsent _id only ("{featureId}:{subjectType}:{subjectId:D}"); no migration One project's or person's consent choice for one beta feature (6F.7, D29 A): featureId, subjectType (project or person), subjectId (lower-case D string, never BinData), environmentId, bindingEpoch, choice (Joined or Left), consentMode (the effective consent when chosen), policyRevision, actorId (the identity the request ran as), realActorId (the signed-in person who made it; the administrator under support impersonation), recordedUnderImpersonation, realActorRecordedAtUtc (equal to updatedAtUtc unless an older build rewrote the row since; the three are additive and absent on older rows), chosenAtUtc and updatedAtUtc (MongoDB time); no names or emails. Written by one upsert by _id from PUT api/account/features/{featureId}/consent and PUT api/projects/{projectId}/features/{featureId}/consent; read by _id $in for the features a request decides with consent, and counted by one anchored _id range ("{featureId}:" to "{featureId};") for GET api/runtime-feature-flags/features/{featureId}/choices. Reads ignore a row of another binding epoch (the next write replaces it). Only RuntimeFeatureConsentReader and RuntimeFeatureConsentStore name it (RuntimeReleaseCollectionsForwardRuleTests). Account or project deletion does not erase it yet: erasure joins the D27 / D34 subject-deletion follow-up
runtimeHelpPublication _id only ("{environmentId}:{topicId}:{variantId}:{revision}"); no migration The help publication record (6G.P): one row per environment and public help topic variant revision that the environment's own public guide site was seen to serve (an anonymous GET of the guide image's /syrf-publication.json manifest and of the page: 200, no redirect, section anchor present). Set on insert only: environmentId, bindingEpoch (CSUUID), clusterFingerprint (the verified binding's), featureId, topicId, variantId, revision, publication (PublicPreview or PublicReleased only), path, anchor, guideOrigin, firstObservedAtUtc (MongoDB time), firstManifestDigest, firstObservedBy (the pod's instance UID); updated every pass: lastObservedAtUtc, lastManifestDigest, lastPageSha256 (SHA-256 of the served HTML), lastGuideOrigin (the origin that served this observation; additive, older rows lack it). A page counts as served only for 200, text/html and a body containing <html. No people, request data or IP addresses. Written by one unordered majority bulk upsert by _id per pass of the API's RuntimeHelpPublicationRecorder (Shared-mode writer, RuntimeFeatureReleases:Enabled); read by one anchored _id range per topic variant (RuntimeHelpPublicationStore.FindAsync, the contract 6H will use with RuntimeHelpPublicationEvidence.IsProductionEvidence). Kept indefinitely (it is the evidence; it grows only with page edits). Only RuntimeHelpPublicationStore names it (RuntimeReleaseCollectionsForwardRuleTests). See Managing Feature Flags

Which component needs which index. Each one checks its indexes before it writes, and refuses (writing nothing) when an index is missing or incompatible:

Component Required indexes
Prepare the audit indexes 0009–0010 on runtimeFeatureFlagAudit, plus 0012–0015 and 0021
Apply the audit indexes 0009–0010 on runtimeFeatureFlagAudit, plus 0012–0017 and 0021
Reconciler (confirmation, stop and restore history) 0009–0010, 0012–0015, 0021, plus 0022 for the stop-time cancellation
Outbox dispatcher 0009, 0010, 0013, 0016, 0017 and 0022. It also refuses if the outbox has any TTL index
Historical import (operator command) 0012–0015 and 0021. It writes no outbox row

The availability reader (6E.2) reads runtimeFeatureRelease only: one bounded, projection-only query that uses no new index. The admin "Ready to settle" read (6G AD3) is read-only and also adds no index: the policy document by _id, then bounded reads through 0012 (released GeneralRelease records), 0014 (their stop and restore events) and the audit's 0010 (the current lineage's receipts, newest first, at most 500); see Ready to settle. Two leases are stored on the documents themselves: reconcileLease on a record (30 s), and lease with state: "Leased" on an outbox row (60 s). Both are taken and expire against MongoDB's $$NOW, so a crashed holder releases its work without a clean-up job. The retired test sink never had a migration (its first insert created it). Per-person discovery state (6E.3, #4460) is not on main yet.

Identity Collections

The Identity database contains IdentityInvestigatorMappings, whose one-row-per-InvestigatorId tombstones permanently record mapping history and survive account deletion, and IdentityEmailReservations, whose holder set serializes claims on canonical email addresses. These collections deliberately live beside IdentityUsers, not in the project-management database, so Identity accounts and their mapping state share one backup and restore surface.

Other Bounded Contexts

Prefix Namespace Example Collections
as SyRF.FileListings File attachment storage
ls SyRF.LiteratureSearch Search configuration
(none) Other Uses entity name directly

GUID Representation

CSUUID (C# Legacy) Format

All document IDs in SyRF use CSUUID (C# Legacy GUID) format, which stores GUIDs as BinData subtype 3.

Configuration (from MongoUtils.cs):

public static void EnsureLegacyGuidSerializer()
{
    try
    {
        BsonSerializer.RegisterSerializer(new GuidSerializer(GuidRepresentation.CSharpLegacy));
    }
    catch (BsonSerializationException e)
    {
        if (BsonSerializer.LookupSerializer<Guid>() is not GuidSerializer
            {
                GuidRepresentation: GuidRepresentation.CSharpLegacy
            })
        {
            throw new InvalidOperationException(
                $"The Guid serializer is not the expected representation...", e);
        }
    }
}

Why This Matters

Format BinData Subtype Byte Order Use in SyRF
CSUUID (Legacy) 3 Little-endian first 3 groups Yes - all IDs
Standard UUID 4 Big-endian throughout No

The byte order difference means the same GUID string produces different binary representations:

GUID string: 550e8400-e29b-41d4-a716-446655440000

CSUUID binary:  00 84 0e 55  9b e2  d4 41  a7 16 44 66 55 44 00 00
                [reversed]   [rev]  [rev]  [preserved as-is]

UUID binary:    55 0e 84 00  e2 9b  41 d4  a7 16 44 66 55 44 00 00
                [as-is throughout]

Querying with CSUUID

When using MCP MongoDB tools, mongosh, or Compass:

// ❌ WRONG - Standard UUID won't match existing documents
db.pmStudy.find({ _id: UUID("550e8400-e29b-41d4-a716-446655440000") })

// ✅ CORRECT - Use CSUUID function
db.pmStudy.find({ _id: CSUUID("550e8400-e29b-41d4-a716-446655440000") })

// ✅ CORRECT - Direct BinData with subtype 3
db.pmStudy.find({ _id: BinData(3, "AIQOVZvi1EGnFkRmVUQAAA==") })

Working with GUIDs in Code

Always let the MongoDB C# driver handle serialization:

// ✅ CORRECT - Let the driver serialize
var studyId = Guid.Parse("550e8400-e29b-41d4-a716-446655440000");
var study = await collection.Find(s => s.Id == studyId).FirstOrDefaultAsync();

// ❌ WRONG - Manual binary conversion will likely use wrong byte order
var bytes = studyId.ToByteArray();
var bsonBinary = new BsonBinaryData(bytes, BsonBinarySubType.UuidLegacy);

Driver Connection Behaviour

directConnection=true for single-server local / TestContainers setups

What the code does

When ConnectionStrings:MongoConnection:ClusterAddresses resolves to a single server, both the connection-string builder in MongoConnectionSettings.ConnectionString and the MongoClientSettings builder in MongoContext.BuildSettingsFromClusterAddresses set directConnection=true. Multi-server (real replica set) and SRV (ClusterAddress) paths are untouched.

Why

The local-dev docker-compose.dev.yml runs mongo:7 with --replSet rs0 and initialises the replica set using members:[{_id:0, host:'localhost:27017'}]. That localhost:27017 is the mongo process's internal address inside the container. When the container is published on a different host port (as in side-by-side worktrees, where each worktree gets its own port — e.g. PORT_MONGODB=29585), connecting from the host to localhost:29585 succeeds, but then the MongoDB driver performs SDAM topology discovery: it asks the server for replica-set membership, the server reports localhost:27017, and the driver replaces the configured endpoint with that self-advertised address. The host has nothing on 27017, so every subsequent operation times out with "Connection refused" against Unspecified/localhost:27017.

directConnection=true pins the driver to the configured host:port, skipping SDAM replacement. Change streams still work because the server is genuinely part of a replica set — the flag only affects client-side topology discovery, not server capabilities.

Symptoms if this regresses

Service startup logs timeouts of this shape, even though appsettings.local.json clearly points at the right port:

TimeoutException: ... selecting a server ... Client view of cluster state is
{ ClusterId : "N", Type : "ReplicaSet", State : "Disconnected",
  Servers : [{ EndPoint: "Unspecified/localhost:27017", ... }] }

The give-away is Type : "ReplicaSet" combined with localhost:27017 when you configured a different port — that's the self-advertised address winning over your settings.

Alternative fixes considered

Changing mongo's internal port to match the host-mapped port (as e2e/docker-compose.yml does, with --port 27018 + 27018:27018) also works but requires docker volume reset when the port changes and complicates side-by-side worktrees where each picks a different port. The directConnection flag avoids all of that.

Scope

Tests: MongoContextTests.BuildMongoClientSettings_WithClusterAddresses_* and MongoConnectionSettingsTests.ConnectionString_* pin the behaviour: single server → DirectConnection=true / ?directConnection=true; multi-server → neither.