Screening source-write overhead: diagnosis and measured plan

Read-only analysis of the materialised screening write path, against the merged artifact evidence/source-writes/PS-WRITE-01_ScreeningSourceWrites_2026-09-13.json and the source at 7e0ed90c9. The harness itself is described in screening-write-benchmark.md.

Headline

The uncontended arithmetic is checkable from the artifact, not inferred:

cell	submissions	commands	commands/submission
same_study_1_reviewers_source_only	30	60	2.0
same_study_1_reviewers_materialized	30	900	30.0
different_studies_1_reviewers_source_only	30	60	2.0
different_studies_1_reviewers_materialized	30	900	30.0

A conflict-free screening decision costs 2 MongoDB commands with the flag off and exactly 30 with it on — a 15x fan-out, not 6x. The aggregate 28,104 / 4,649 = 6.05x is diluted by the baseline's own same-study conflicts and by materialised attempts that abort early.

Latency follows the command count almost linearly on the benchmark host. In same_study_1, submission p95 84.44 ms against attempt p95 72.81 ms is an 11.6 ms gap covering the 5 pre-transaction commands — about 2.3 ms per round trip. Twenty-eight extra commands at 2.3 ms is ~64 ms, which is essentially the whole observed delta. The overhead is round-trip count, not any single slow operation.

Indexes are not the cause. Every hot lookup on the write path is served by a declared unique index (ProjectStatisticsCurrentRepository, ...NotificationOutboxRepository, ...PublicationGuardRepository, ...ControlRepository, ...SourceOperationReceiptRepository, ...RevisionRepository InitialiseIndexesAsync), and ProjectStatisticsWriteTestContext creates them in the fixture.

The 30-command trace

Flags: materializedProjectStatisticsWrites on, family ProjectScreening serving, project allowlisted, ActiveReviewerTrackingEnabled = false.

Flag off — 2 commands

#	command	site
1	find pmStudy	benchmark reload -> MongoRepositoryBase.GetAsync (a fresh RepositoryCache per submission, so always a real read)
—	none	ProjectScreeningStatisticsWriter.BeginOperationAsync returns null (IsWriteRequested false)
—	none	RequireDurableModeAgreementAsync(session: null, writesEnabled: false) short-circuits
2	findAndModify pmStudy	the capacity+version guard replace, un-sessioned

No session, no transaction, no statistics read. A guard miss adds one projected find (DiagnoseCapacityGuardFailureAsync) plus one reload find per retry, which is why the baseline aggregate is 4,649 rather than 2,160.

Flag on — 30 commands

Pre-transaction (5, un-sessioned, serial)

#	command	site
1	find pmStudy	benchmark reload
2	find GlobalControl	ProjectScreeningStatisticsWriter.BeginOperationAsync (epoch capture)
3	find Control	same
4	find PublicationGuard(ProjectScreening)	same
—	none	the reviewer-family epoch loop is empty (reviewer families not serving)
5	find SourceOperationReceipt	ResolveByReceiptAsync pre-flight duplicate check

writer.Prepare(...) is pure — classify, dependants, size routing — and issues no command.

Transaction (24 + commit), opened with ProjectStatisticsTransaction.SnapshotOptions (ReadConcern.Snapshot).

#	command	site	document
6	find GlobalControl	StudyRepository durable-mode gate	singleton
7	findAndModify pmStudy	source replace under the shared BuildScreeningSaveFilter guard	per-study
8	find GlobalControl	coordinator	singleton — duplicate of #6, same snapshot
9	find Control	coordinator	per-project, hot
10	find Receipt	idempotency authority	per-operation
11	find Guard(ProjectScreening)	CheckFamilyAdmissionAsync	per-project, hot
12	find Guard(ProjectScreening)	CheckRowAdmissionAsync	duplicate of #11
13	find Current row (project scope)	LoadRowAsync	per-project, hot
14	insert Receipt	coordinator	per-operation
15	find Guard(ProjectScreening)	MaterializeAsync	duplicate of #11/#12
16	find Current row	LoadRowAsync	duplicate of #13
17	update Current row (CAS on Version)	MaterializeAsync	per-project, hot — first shared write
18	update Guard(ProjectScreening) (CAS)	RecordPointMaterialization	per-project, hot
19	find Revision	delta duplicate check	per-operation
20	insert Revision (signed delta)	coordinator	per-operation
21	find Guard(MembershipScreening)	MarkFamilyStaleAsync; guard is null so it returns	wasted
22	find Guard(ReviewerScreening)	same	wasted
23	find Outbox slot (ProjectScreening)	CoalesceSlotAsync	per-project, hot
24	update Outbox slot (ProjectScreening)	same	per-project, hot
25	find Outbox slot (MembershipScreening)	same	per-project, hot
26	update Outbox slot (MembershipScreening)	same	per-project, hot
27	find Outbox slot (ReviewerScreening)	same	per-project, hot
28	update Outbox slot (ReviewerScreening)	same	per-project, hot
29	update Control (CAS on Version)	coordinator	per-project, hottest
30	commitTransaction	ProjectStatisticsTransaction	—

Total 30, matching the artifact's commands=900 / 30 submissions and commit_attempts=30 exactly.

Where the fan-out comes from

• Eleven of the 24 in-transaction commands are reads of documents already read in the same snapshot, or of documents that do not exist. #8 repeats #6; #12 and #15 repeat #11; #16 repeats #13; #21/#22 look up guards for two families that have never been materialised and always return null.
• Every write is a read-then-CAS-replace pair, never an atomic update. ProjectStatisticsRepositoryBase.TryReplaceAsync does a whole-document ReplaceOne filtered on Version, so each mutated document costs two commands.
• Three notification slots are coalesced per save. UpsertNotificationSlotsAsync unions the affected family with every dependant, and ProjectScreeningClassifier.DependentFamilies unconditionally adds MembershipScreening and ReviewerScreening: three slots x (find+update).
• Nothing is batched or pipelined. Every repository call is one await on one round trip.
• Even a no-delta save pays a transaction. transaction-admission.md records this under "Accepted costs"; the artifact shows it biting (different_studies_10 ... no_delta_saves=116 of 176 saves).

Contention hot spots

Every concurrent screening save in a project writes these documents, regardless of which Study it targets:

document	identity	why it is written every time
ProjectStatisticsControl	one row per project	AdvanceClocks bumps SourceInvalidationRevision, CommittedProjectionRevision and ClientInvalidationRevision, then the row is CAS-replaced
ProjectStatisticsCurrent (ProjectScreening, project scope)	one row per project	all screening moves are emitted at ProjectStatisticsScopeKey.ForProject(); there is no per-study or per-stage sharding
ProjectStatisticsPublicationGuard (ProjectScreening)	one row per project	RecordPointMaterialization + CAS
ProjectStatisticsNotificationOutbox x3	three rows per project	one slot per family, coalesced on every save

That is six shared per-project documents on the critical path of every reviewer decision. Only the receipt and the delta record are contention-free.

The artifact is unambiguous about the consequence. Baseline different-study cells record version_conflicts=0 at 1, 2, 5 and 10 reviewers — the source documents never collide. The materialised different-study cells record 0 / 59 / 195 / 432 conflicts and 0 / 0 / 45 / 124 exhausted. All different-study contention is manufactured by the shared statistics documents.

Under ReadConcern.Snapshot a write conflict is raised at the document level whenever a document has been modified since the transaction's snapshot. Reads never conflict, so in the original ordering the first detection point is command #17, the current-row CAS — after about eleven in-transaction round trips of work that is then thrown away.

The retry budget

• The six-argument overload does not retry internally. It catches TransientTransactionError, aborts, and returns CapacityGuardSaveResult.VersionStale. This is deliberate: the callback has already run OnSaving and a classified transition, so replaying it against newer source data is prohibited (transaction-admission.md, "A mutable Study callback...").
• CommitWithUnknownResultRetryAsync retries only UnknownTransactionCommitResult. The artifact records unknown_commit_failures=0 in every cell, so it contributed nothing to the measured cost.
• The budget that matters is the caller's: ReviewController.TrySaveScreeningAsync with maxAttempts = 3, an immediate reload and no backoff, mirrored by the benchmark.

Conflicts rose 1,005 -> 1,883 and exhaustion 263 -> 553 because the transaction's write set grew from one document to seven, its duration grew about sevenfold (widening the window in which a competitor invalidates the snapshot), and the budget is unchanged. Raising the budget is rejected: it trades exhaustion for more load on the same hot document.

What the harness measures, and what it does not

Not measured, and material:

1. A different overload from the current HTTP path. With tracking off, ReviewController takes TrySaveScreeningWithStatisticsAsync (version-only filter), not the capacity-guarded overload. This applies to both arms, so it does not bias the ratio, but the absolute p95 is not the production branch.
2. The controller does strictly more work than the harness, not less: ResolveVerifiedActiveReviewerTrackingModeAsync() runs once per attempt, adding one un-sessioned GlobalControl find per attempt. Thirty commands is a floor for the HTTP path, not a ceiling.
3. Everything around the save is excluded — HTTP pipeline, authorization, project load, next-study allocation, reservation cleanup, domain-event dispatch, SignalR. Those costs land on both arms in production, so the endpoint-level percentage regression will be materially lower than 660%. The absolute regression (~+60-70 ms of serial round trips) is the honest number to carry forward.

Harness artefacts that could inflate the candidate arm specifically:

• Command recording itself. A CommandStartedEvent subscriber forces the driver to materialise every command document. The candidate issues 15x more commands, so the recorder's cost is ~15x higher on the candidate arm. It is now opt-in via SYRF_STATS_COUNT_ROUNDTRIPS, so a no-subscriber control run is possible; before that change it was always on and was an un-quantified, asymmetric confound.
• Snapshot read concern is only on the candidate arm. Correct and required, but its cost is not separable from command count in the present data.
• n = 30 per cell. A p95 over 30 samples is effectively the second-worst observation. Treat cell-to-cell ordering as noise-dominated.
• Host. 48 cores, ContainerResourceCapsApplied = false, Mongo container co-resident with the driver and the test process.

Measured plan

Every item is re-measured with the existing benchmark, the same eight cells, SYRF_STATS_DATASET=PS-WRITE-01. Read every gate as the artifact's *_p95_gate plus the *_outcomes note. All-submission p95 including exhausted requests remains the metric: a better p95 with a worse saved count is not an improvement.

Measurement hygiene required before any of this counts: run unchanged main and the candidate back to back on the same idle host with the same iteration and warmup settings, and report both.

Cheap and safe — no protocol change

A1. Memoize in-snapshot reads inside one CommitAsync. Removes commands #12, #15 and #16 by threading the already-loaded guard and current row through DecideAsync -> CheckRowAdmissionAsync -> materialisation. Invariant: epochs must still be stamped from authorities read in this transaction's snapshot. Memoizing a read taken inside the same session is the same snapshot — so this must be session-scoped state, never a process cache, or the fail-closed durable-mode contract breaks. Residual: the duplicate in-transaction GlobalControl read (#8 repeating #6) is not removed. The two reads straddle a component boundary — #6 is StudyRepository's durable-mode gate, which is the fail-closed mode authority, and #8 is the coordinator's own — so sharing them would change the IProjectStatisticsTransactionCoordinator.CommitAsync contract rather than memoize inside it. Both are still present in the measured trace below. One command, deliberately left.

A2. One guard lookup per transaction, for all families. Replaces the separate GetByFamilyAsync calls (#11, #12, #15, #21, #22) with a single find over ProjectId + MetricFamily $in [...] covering affected and dependent families, cached for the transaction. The existing (ProjectId, MetricFamily) unique index serves it. Invariant: fence and epoch checks must still see the snapshot's values, and a family with no guard must stay a no-op, not an insert.

A3. Batch the notification slots. One find with the named slot keys plus one BulkWrite for the three slots, replacing six commands with two. Invariant: the outbox contract the benchmark asserts — slots.Max(TargetClientInvalidationRevision) == control.ClientInvalidationRevision — and the CAS-on-Version fail-closed semantics of ReplaceOrFailAsync must survive as per-op results of the bulk write.

A4. Order the hot writes first, so a losing attempt fails fast. Issue the control-row CAS and the current-row CAS immediately after the source replace. A loser then aborts after about three round trips instead of eleven. Invariant: nothing about what commits changes — the transaction is still all-or-nothing, and AdvanceClocks already runs before materialisation. The routing decision must still run before any write, so the source-only fallback still writes no point-path row. Gate: different_studies_{2,5,10}_reviewers_materialized commands down sharply with version_conflicts and exhausted unchanged or better. If exhausted rises, reject.

A5. Parallelise the pre-transaction reads. The epoch capture's independent un-sessioned reads are issued together. Invariant: none — these reads are deliberately outside any transaction, and a disagreement with the in-transaction authorities already routes to the source-only fallback. Gate: commands unchanged; submission p95 minus attempt p95 narrows.

Cumulative effect on the uncontended path: 30 commands -> 21, asserted by ProjectScreeningWriteRoundTripBudgetTests. The nine commands removed are four guard reads (#12, #15, #21, #22), one current-row read (#16) and four of the six outbox commands. This does not reach the <10% gate and must not be presented as if it might. The remaining floor is one transaction plus a commit plus the serial round trips against six shared documents.

Needs design — not adopted on this evidence

B1. One atomic update per hot document instead of read-then-CAS-replace. $inc/$set pipelines on Control, Current and Guard would halve their command cost. It does not reduce conflicts — document-level write conflicts are independent of operator choice — and it changes the optimistic-concurrency contract that ReplaceOrFailAsync's PublicationRaceLost fail-closed path depends on.

B2. Striped or split counters for the project-scope screening row. This is the plan's own escape hatch, and the ½/5/10-reviewer gate asked whether the small project summary is actually hot. It is. But striping breaks equality-gated serving as currently written: the reader requires row.LastChangedRevision <= control.CommittedProjectionRevision and an exact AppliedWriteEpoch/digest/version match. A shard-and-sum read path, checkpoint compatibility and rebuild semantics all need specifying first. This is the highest-value design item on #3255.

B3. Take the control-row clock advance off the per-write path. Three monotonic counters on one per-project document is the true serialisation point; no batching fixes it. Any alternative changes the ordering guarantee that serving equality, checkpoints and the client-invalidation contract rest on.

B4. Do not coalesce slots for families that were never materialised. Commands #25-#28 write two shared outbox rows for MembershipScreening and ReviewerScreening whose guards do not exist and whose MarkFamilyStale was a no-op. Skipping them would remove two of the six hot documents. Whether a client subscribed to a dark family must still see a revision bump is a contract question.

Open questions this analysis could not answer

1. How much of the 84 ms is commit versus round trips versus snapshot read concern. The 2.3 ms/RTT figure comes from one subtraction in one cell; a per-command-duration breakdown needs a run with CommandSucceededEvent durations.
2. Whether CommandStartedEvent subscription materially inflates the candidate arm. Structurally asymmetric, but the magnitude needs the A/B that SYRF_STATS_COUNT_ROUNDTRIPS now makes possible.
3. Whether ReadConcern.Snapshot costs anything measurable versus the default on this topology. No A/B exists, and one must not be created by relaxing a correctness invariant.
4. The write concern in force on commit. A w:majority commit on a three-node container could be a large single-digit-millisecond constant on both arms.
5. The real HTTP endpoint's overhead percentage. Whether it is 50% or 400% is not derivable from this artifact.
6. Why no_delta_saves is 116/176 in different_studies_10_reviewers_materialized. The likely reading is that exhausted submissions leave a study at an older decision, so a later round re-applies the same value and Prepare returns null — inference from the counts, not proof.
7. Production behaviour under a remote (Atlas) Mongo. Round-trip cost dominates, so a higher RTT would scale the absolute regression roughly linearly with the extra commands. Untested.
8. Whether any of these documents are hot in production for reasons outside screening — assignment, annotation and import all touch the same control row.

Outcome (2026-09-15)

A1-A5 landed in #3475. The B items were not started.

Measured: 30 -> 21 commands per uncontended materialised save, asserted rather than estimated by ProjectScreeningWriteRoundTripBudgetTests, which counts the driver's commands for a real end-to-end save on a replica set and fails above 21. The nine removed commands are four guard reads, one current-row read and four outbox commands. The recorded trace after the change:

find(pmStudy), find(GlobalControl), find(Control), find(Guard), find(Receipt),
find(GlobalControl), findAndModify(pmStudy),
find(GlobalControl), find(Control), find(Receipt), find(Guard), find(Current),
update(Control), update(Current), insert(Receipt), update(Guard),
find(Revision), insert(Revision),
find(Outbox), update(Outbox)  <- one bulk write of three slots,
commitTransaction

Two in-transaction find(GlobalControl) remain, for the reason recorded under A1 above.

Paired run 2026-09-15 (loaded host)

main at 0fe1716f6 and #3475 at 6546bb25d, back to back on the same host in the same minute, SYRF_STATS_ITERATIONS=30 SYRF_STATS_WARMUP=5 SYRF_STATS_COUNT_ROUNDTRIPS=1. Artifacts: main arm, #3475 arm. Fuller commentary in screening-write-benchmark.md.

The host was not idle — load average 20-45 with roughly 130 CI containers running through both arms — so this run does not satisfy the measurement hygiene rule stated above. The p95 columns are indicative only and no ratio derived from them may be quoted. The command, conflict, exhaustion and saved counts are deterministic work counts and are the reliable part.

Cell	Commands (main → #3475)	Conflicts	Exhausted	Saved	Materialised p95 ms	Source-only p95 ms
same_study 1	900 → 630	0 → 0	0 → 0	30 → 30	118.72 → 56.53	58.55 → 12.62
same_study 2	1390 → 1090	82 → 80	25 → 25	35 → 35	95.67 → 53.62	13.29 → 17.18
same_study 5	2812 → 2553	343 → 343	111 → 111	39 → 39	84.33 → 56.52	15.73 → 26.41
same_study 10	5196 → 4974	779 → 770	251 → 246	49 → 54	58.99 → 71.41	41.99 → 30.72
different_studies 1	900 → 630	0 → 0	0 → 0	30 → 30	47.91 → 42.95	6.44 → 8.68
different_studies 2	2612 → 1860	58 → 60	0 → 0	60 → 60	103.98 → 75.32	38.86 → 5.90
different_studies 5	5090 → 3779	193 → 193	44 → 43	106 → 107	103.06 → 93.50	7.31 → 5.85
different_studies 10	9296 → 7142	434 → 428	125 → 118	175 → 182	94.07 → 98.10	9.94 → 10.15

Established: the command reductions, confirmed against the unit measurement. The uncontended cells fall 900 -> 630 over 30 submissions, i.e. 30 -> 21 per save. The contended cells fall further in absolute terms because a losing attempt also does less wasted work.

Not established: A4's contention effect. Conflicts and exhaustion are essentially unchanged (58->60 / 0->0; 193->193 / 44->43; 434->428 / 125->118) — ordering the contended writes first cuts the work a loser does, not the number of losers, and the counts show exactly that. No latency claim is made: the source-only p95 of the same cell moved by up to 6.6x between arms from host load alone (different_studies 2: 38.86 ms against 5.90 ms), so under this load the overhead ratios measure the scheduler rather than the change. All eight <10% gates still fail in both arms.

An idle-host re-run is still owed before any rollout claim, and before any p95 or contention statement about this change is made.

Appendix: cell-level evidence

cell	saved	conflicts	exhausted	commands	commit attempts	p95 overhead
same_study_1 baseline / materialized	30/30	0/0	0/0	60/900	0/30	660.79%
same_study_2	60/36	30/78	0/24	210/1356	0/36	134.28%
same_study_5	90/46	270/334	60/104	990/2781	0/46	99.70%
same_study_10	97/44	705/785	203/256	2309/5199	0/44	96.22%
different_studies_1	30/30	0/0	0/0	60/900	0/30	647.97%
different_studies_2	60/60	0/59	0/0	120/2626	0/60	1010.16%
different_studies_5	150/105	0/195	0/45	300/5070	0/105	745.78%
different_studies_10	300/176	0/432	0/124	600/9272	0/176	495.69%

unknown_commit_failures = 0 in every cell. Maximum summary bytes 1,199-1,787 — size is not a factor.