Skip to content

Screening source-write development benchmark

ProjectScreeningWriteBenchmark measures the existing capacity-save source path with materialized writes off against the six-argument source-plus-statistics transaction with writes enabled. Both use real MongoDB replica-set source storage. The enabled path includes the durable-mode snapshot read, operation/receipt admission and real screening writer/coordinator. Each request owns a separate repository/cache and reloads and reclassifies after an optimistic conflict, with the API's three-attempt limit (ReviewController.TrySaveScreeningAsync, maxAttempts = 3, immediate reload with no delay). The harness uses GetAsync for its known surviving fixtures where the controller uses GetOrDefaultAsync and also handles deletion, capacity and duplicate-operation outcomes. It does not replay the mutable aggregate inside a transaction retry.

Current ReviewController also resolves durable tracking mode on each attempt and chooses the plain save path when tracking is off. This benchmark deliberately retains the capacity-save repository overloads with tracking off; its three-attempt limit matches the controller, but its measured path and command counts do not represent that current HTTP branch. Reviewer membership/authorization, controller mode resolution and event dispatch are outside this fixture. Integrating reviewer maintenance code does not enable its flags in this ProjectScreening-only benchmark.

The minimum corpus, PS-WRITE-01, contains one screening project and ten studies. Each matrix cell uses 1, 2, 5 or 10 reviewers submitting to the same study or separate studies. Later rounds alternate inclusion/exclusion decisions by the same reviewers. The artifact's corpus counts describe maximum matrix dimensions, not a claim that every cell has ten active reviewers. This is deliberately not a large-project PS-DS-02 benchmark. Scale values other than 1 are refused.

Reproduce

From the PR worktree root:

SYRF_STATS_DATASET=PS-WRITE-01 SYRF_STATS_ITERATIONS=100 SYRF_STATS_WARMUP=10 \
SYRF_STATS_RESULTS_DIR=/tmp/syrf-screening-write-results \
dotnet test src/libs/project-management/SyRF.ProjectManagement.Mongo.Data.Tests/SyRF.ProjectManagement.Mongo.Data.Tests.csproj \
  --filter FullyQualifiedName~ProjectScreeningWriteBenchmark

Command counting is opt-in. Add SYRF_STATS_COUNT_ROUNDTRIPS=1 to the command above for the command breakdown; leave it unset for a no-subscriber control timing. Without the dataset variable, ordinary CI skips this benchmark. The existing replica-set fixture owns and removes its mongo:8.0 container. Existing shared-runner resource limits apply when SYRF_TEST_JOB_KEY is set; the artifact records that policy and the runtime/server environment. No deployed database, credentials or live flags are needed.

What the harness now records

Following the write-overhead diagnosis (screening-write-overhead-diagnosis.md, #3255), three things changed in what this harness reports. The artifact schema is extended, never renamed, so an existing reader of Durations, Notes or the *_outcomes keys is unaffected.

  • Command counting is opt-in via SYRF_STATS_COUNT_ROUNDTRIPS. A CommandStartedEvent subscriber forces the driver to materialise every command document, and the materialised arm issues many times more commands than the source-only arm, so the recorder's own cost is asymmetric between the two arms being compared. Running once without it gives a no-subscriber control timing. When it is off, commands and commit_attempts read not_counted rather than zero, and no RoundTrips record is emitted.
  • Commands are broken down by driver command name, in the artifact's existing RoundTrips field, so a reduction is attributable to a specific command class rather than only to a batch total. The per-name counts are windowed exactly as the total is, so they sum to commands.
  • Per-attempt command counts are recorded in the *_commands_per_attempt note, for the single-reviewer cells only. With two or more reviewers the submissions run concurrently against one shared counter, so a per-attempt delta would include another reviewer's commands; the artifact says so rather than publishing a figure it cannot support.

The duplicate pre-check's assertion also moved outside the timed window. ResolveByReceiptAsync itself stays inside it, because production runs the same pre-check before opening a transaction.

Evidence and interpretation

The existing BenchmarkResults format records p50/p95 submission duration and repository-attempt duration. Submission time includes source reload, envelope/receipt reads, classification and retries. It excludes seeding, assertion queries, HTTP middleware, assignment, event dispatch and background work. The source-only baseline performs no statistics admission reads. Candidate/baseline ordering alternates by reviewer count. Each variant has its own warmup.

Every batch checks exact persisted decisions against successful submissions, and enabled batches also check exact authoritative/projection parity, operation receipt/delta presence, committed projection revision advancement and the coalesced outbox's client revision. Exhausted retries are reported as failures to save. A repeated decision can save source without a classified delta; these saves are counted separately and correctly have no statistics receipt/delta. There is no silent exception-to-success conversion. Driver command monitoring counts all commands during submissions, commit attempts and UnknownTransactionCommitResult failures; it excludes seeding and post-batch assertion queries. There is no injected unknown-commit failure in this timing run.

The p95 overhead gate is strictly less than 10%. Its result is emitted for each matrix cell, never asserted as a CI correctness test on a shared development host. All-submission latency includes exhausted requests, so a lower p95 with a worse success rate cannot establish a usable performance improvement. The artifact includes saved, conflict, exhausted and no-delta counts so that tradeoff remains visible. Maximum serialized summary bytes are also recorded.

This slice keeps active-reviewer tracking off and enables only ProjectScreening maintenance. It does not establish reviewer-family maintenance overhead, reservation/assignment throughput, annotation write cost, non-capacity API latency, large-project capacity, live rollout consistency or production acceptance. Those required programme measurements remain ordered follow-up evidence. Existing deterministic replica-set tests separately prove the non-capacity transaction conflict response and rollback.

Development result: 5 September 2026

The 100-iteration artifact records 10 warmup batches per variant. All exact correctness assertions passed. Every p95 overhead gate failed. These measurements leave the performance acceptance gate open.

Target Reviewers Baseline p95 ms Materialized p95 ms Overhead Baseline saved Materialized saved
Same study 1 8.57 64.64 654% 100/100 100/100
Same study 2 11.24 47.56 323% 200/200 117/200
Same study 5 20.35 44.77 120% 300/500 141/500
Same study 10 28.90 44.55 54% 329/1000 146/1000
Different studies 1 6.19 48.65 686% 100/100 100/100
Different studies 2 6.14 94.55 1441% 200/200 200/200
Different studies 5 7.71 80.96 950% 500/500 350/500
Different studies 10 21.89 134.89 516% 1000/1000 598/1000

The different-study result exposes contention on shared project statistics despite independent source records: 1,406 optimistic conflicts and 402 exhausted submissions in its ten-reviewer candidate cell. No unknown-commit failure was observed. Summary documents remained at most 1,982 bytes in this small corpus. The artifact's no_delta_saves field counts successful source saves without a statistics transition, so it includes every source-only baseline save; it is not a count of unchanged source data.

Required continuation #3255 owns the bounded write-path performance and contention improvement before claiming activation readiness. Increasing retries alone would trade exhausted requests for more load and latency; it does not establish the required overhead threshold. Re-run this matrix after the chosen change and add the required reviewer-family and active-tracking variants as their writers become available. The present slice does not silently expand maintenance or alter the existing three-attempt source contract to make its numbers pass.

Dependency integration measurement: 13 September 2026

After merging main and reviewer-maintenance prerequisite #3297 at 0803a4ec5, the opt-in smoke run (3 measured batches, 1 warmup) passed. A separate 30-batch artifact then ran all eight cells with five warmup batches per variant against disposable MongoDB 8.0.28 on .NET 10.0.0. Each cell still has one project and ten studies, and only ProjectScreening is enabled. The measured total is 1,080 submissions per variant; warmup is excluded. Every source/projection, receipt/delta and outbox correctness assertion passed. No unknown-commit failure was observed.

All eight overhead gates failed again. Thirty batches are a modest local measurement, not a large-project, current-controller, staging or production acceptance result. The historical 100-iteration artifact above remains unchanged. This run records that dependency revision; it does not establish a comparable host-controlled trend against September 5.

Target Reviewers Baseline p95 ms Materialized p95 ms Overhead Baseline saved Materialized saved
Same study 1 11.10 84.44 660.79% 30/30 30/30
Same study 2 24.68 57.83 134.28% 60/60 36/60
Same study 5 26.49 52.90 99.70% 90/150 46/150
Same study 10 30.73 60.30 96.22% 97/300 44/300
Different studies 1 8.40 62.83 647.97% 30/30 30/30
Different studies 2 8.46 93.91 1010.16% 60/60 60/60
Different studies 5 11.36 96.04 745.78% 150/150 105/150
Different studies 10 14.60 86.96 495.69% 300/300 176/300

Baseline exhausted 263 of 1,080 submissions after 1,005 total optimistic conflicts; materialized exhausted 553 after 1,883 conflicts. Exhaustion is a failed save, never counted as success. During measured submissions the driver recorded 4,649 baseline commands versus 28,104 materialized commands, including 527 materialized commit attempts. Those command counts exclude assertion queries and seeding. The higher command count and contention are observed diagnostic evidence; this slice introduces no unmeasured partitioning or write-protocol changes to chase the threshold.

Reproduce this sample with the command above using SYRF_STATS_ITERATIONS=30 and SYRF_STATS_WARMUP=5. Focused integration validation additionally passed 67 tests covering durable mode saves, optimistic contention and reviewer materialization; with no dataset variable the benchmark was skipped as intended. Performance follow-up remains #3255, and rollout stays gated by the failed write-overhead criterion and the separate live acceptance requirements.

The later integration of #3297 at 266a8dac3 includes Sonar fixes, legacy null-list recovery and a real Project-document admission write for reviewer maintenance. The recorded artifact predates those fixes. Reviewer maintenance remains disabled in this benchmark; measuring its additional write and contention requires a separately enabled workload before rollout acceptance. No fresh performance measurement is claimed by this integration.

Paired run 2026-09-15 (loaded host)

main at 0fe1716f6 and #3475 at 6546bb25d were run back to back, same host, same minute, SYRF_STATS_ITERATIONS=30 SYRF_STATS_WARMUP=5 SYRF_STATS_COUNT_ROUNDTRIPS=1, MongoDB 8.0.28 on .NET 10.0.0. Artifacts: main arm, #3475 arm.

The host was not idle. Load average was 20-45 with roughly 130 CI containers running throughout both arms. This measurement therefore fails the hygiene rule this programme set for itself — "run unchanged main and the candidate back to back on the same idle host" — in one specific respect: the p95 columns below are indicative only and no ratio derived from them may be quoted. The command, conflict, exhaustion and saved counts are deterministic work counts rather than timings and are the reliable part of this run.

Cell Commands (main → #3475) Conflicts Exhausted Saved Materialised p95 ms Source-only p95 ms
same_study 1 900 → 630 0 → 0 0 → 0 30 → 30 118.72 → 56.53 58.55 → 12.62
same_study 2 1390 → 1090 82 → 80 25 → 25 35 → 35 95.67 → 53.62 13.29 → 17.18
same_study 5 2812 → 2553 343 → 343 111 → 111 39 → 39 84.33 → 56.52 15.73 → 26.41
same_study 10 5196 → 4974 779 → 770 251 → 246 49 → 54 58.99 → 71.41 41.99 → 30.72
different_studies 1 900 → 630 0 → 0 0 → 0 30 → 30 47.91 → 42.95 6.44 → 8.68
different_studies 2 2612 → 1860 58 → 60 0 → 0 60 → 60 103.98 → 75.32 38.86 → 5.90
different_studies 5 5090 → 3779 193 → 193 44 → 43 106 → 107 103.06 → 93.50 7.31 → 5.85
different_studies 10 9296 → 7142 434 → 428 125 → 118 175 → 182 94.07 → 98.10 9.94 → 10.15

What this establishes

  • The command reductions are real and match the unit measurement. The uncontended cells fall 900 -> 630 over 30 submissions, i.e. 30 -> 21 commands per save, exactly the figure ProjectScreeningWriteRoundTripBudgetTests asserts. The candidate artifact's per-command-name breakdown for that cell is find 390, update 120, insert 60, findAndModify 30, commitTransaction 30 — 13 finds, 4 updates, 2 inserts, 1 findAndModify and 1 commit per save — and its per-attempt note records 16 commands inside the repository save, leaving 5 before the transaction.
  • The contended cells drop by more in absolute terms, because a conflicting attempt also does less wasted work: 2612 -> 1860 at two different-study reviewers, 5090 -> 3779 at five, 9296 -> 7142 at ten.

What this does not establish

  • A4's contention effect is not demonstrated. Conflicts and exhaustion are essentially unchanged (58->60 / 0->0; 193->193 / 44->43; 434->428 / 125->118). Ordering the contended writes first was expected to cut the work a loser does, not the number of losers, and that is what the counts show — a loser still loses. The small movements in saved (175 -> 182 at ten different-study reviewers, 49 -> 54 at ten same-study) are within the noise of a loaded host and are not claimed as an improvement.
  • No latency claim. Materialised p95 is lower in six of the eight cells and higher in two, but the source-only p95 of the same cell moved by as much as 6.6x between arms (different_studies 2: 38.86 ms on main against 5.90 ms on the candidate) purely from host load. That is why the artifacts' own *_p95_gate overhead percentages are not reproduced here: under this load they measure the scheduler, not the change.
  • Every one of the eight <10% overhead gates still fails, in both arms. Nothing here moves the performance acceptance gate.

An idle-host re-run is still owed before any rollout claim, and before any p95 or contention statement about this change is made. Until then the defensible statement is the one the command counts support: nine fewer round trips per conflict-free save, and proportionally less wasted work per losing attempt.