rate-my-server · Release review for APIs, workers, and pipelines
The happy path proves nothing.
Your backend works when everything goes right. Now make it survive retries, races, dependency failures, restarts, and real users — with five evidence lanes that cannot substitute for each other.
Issue list
BLOCKER · F-001A tenant can read another tenant's report — private data crosses the ownership boundary.
HIGH · F-003A duplicated webhook delivery applies the same credit twice — there is no durable uniqueness on the effect.
To verify
UNKNOWN · U-001Backups exist but restoration has never been exercised — recoverability is not yet proved.
Evidence lanes
Scope
What it actually audits.
Only things that would actually hurt someone using it. It will not tell you your code is ugly.
Authentication, object ownership, role policy, tenant isolation, secrets, and privacy boundaries.
Request, job, stream, and batch contracts — including whether reported success is actually true.
Transactions, idempotency, durable uniqueness, ordering, leases, acknowledgement, replay, reconciliation.
Timeouts, retry budgets, dependency failure, admission control, queue bounds, saturation, backpressure.
Graceful shutdown, restart, deploy, migration, rollback, telemetry, backup, restore, incident recovery.
Cross-component journeys: request → commit/outbox → queue → worker → external effect.
With and without
What changes once it is installed.
Any decent AI will read your code and have opinions. This is the list of things it will not do unless this is installed.
Without it
It reads the auth middleware, sees a session check, and calls the boundary fine.
With it
It reproduces an authenticated Alpha caller reading a Beta report by ID because the lookup carries no tenant predicate — with an own-tenant read kept as the clean control.
Without it
It suggests “add idempotency” as a best practice.
With it
It reproduces the effect committing and then timing out, and shows retry creating a second effect for the same business identity.
Without it
Green tests and a healthy golden path read as production-ready.
With it
It reproduces blocked in-flight work outliving the join grace and leaving a job in processing, then connects startup recovery to that state without claiming data loss it did not observe.
The part every other review skips
Then who reviews the fix?
Most reviews end at the list. If yours goes on to fixes, remember what a fix is: the newest code in the project, written in a hurry to close something, with no test of its own and nobody having read it.
The pass that writes the fix never grades its own work.
Whoever made the fix does not get to be the one who says it worked. Someone else checks.
A diff is not a fix.
Every finding ships with the exact test that would close it. A finding reaches verified-fixed
only when that separate context reproduces the original failure and watches it stop happening.
The patch itself gets audited.
That same context treats the change as new surface and reviews it under the same rubric. Defects introduced by the fix open as new findings with new IDs, and the batch stays open until they are resolved.
The loop ends on evidence, not on an empty list.
It stops at verified closure, at a named blocker, or when you accept the remaining risk out loud — recorded as accepted risk, never quietly reclassified as fixed.
This site was audited this way. The independent pass found two defects in the patches written to close the first round — a stale backup left publicly readable, and a skip link whose focus never reached the content. Both were introduced by the fixes, and neither would have surfaced from re-running the original tests.
How it runs
Two settings, then evidence.
It asks. It will not quietly pick the harshest setting for you, or the kindest.
Reviewer setting
Service mode matched to what the service actually owns
Review degree
Quick check / Strict review / Launch gate / Real stakes / Life-or-death
No score averages away a veto.
Some problems stop a release no matter how good the overall score looks. Saying you can live with one does not turn it into a pass.
Start with the quick check.
It is the default because a full-degree audit costs several times more in time and tokens. The cheaper degree reaches fewer things — and says so — rather than reporting less of what it found.
The five
One evidence contract, five layers.
Each installs on its own. Take the one that matches what you are shipping.
Install
One command, or a plugin marketplace.
One method per client. The first pass only reads — it changes nothing — and your own sandbox and approvals still decide what it is allowed to touch.
npx skills add AmsonntagChow/rate-my-server --skill rate-my-server
/plugin marketplace add AmsonntagChow/rate-my-server
codex plugin marketplace add AmsonntagChow/rate-my-server && codex plugin add rate-my-server@amsonntagchow-rate-my-server