We Open-Sourced Our Production Risk Review. Run Cold, It Found a Critical Our Paid Review Missed

We open-sourced the method behind our $495 production risk review as Open Risk Review, a skill your own AI coding tool runs on your repo. Before releasing it, we ran it cold on our own code. On FoodFiles it found a Critical that our paid review of the same app had missed a month earlier: any new user could grant themselves unlimited free credits. Fixing that led us to a second Critical in the auth service underneath. Both Criticals were fixed on October 15. Two ownership checks we shipped on October 15, edge-auth's key-to-tenant binding and the Trust Page fix from run two, were too broad and were tightened on October 16. This post covers what the open-source risk review found, what it got wrong, and what changed in the skill before release.

Updated October 17 and 18: we corrected the credit limit (each request was capped at 10,000 credits, but could be repeated without limit). We had also said each signup is an admin of its own organisation. In fact every customer of a product shares one. That is why two fixes needed a per-tenant follow-up on October 16. It is also why a dogfood allowlist we found while checking this post let every Stackbilder signup use the entitlement-gated consultations until we removed it on October 17. From July 21 to September 11, the same allowlist could have exposed an operator credential to them. Our request traces show no one loaded that page in that window, though they can't fully prove it. We rotated the credential on October 18. We also made a few smaller wording corrections.

Open Risk Review: our production risk review as an open-source AI skill

In September we put a production risk review on sale and then ran it on ourselves three times. That review is a method: map the repository, run four focused review passes, re-verify every Critical and High by hand, and write a report with evidence at file and line. The method was always the AI part. What a buyer pays for is a named person who verifies the serious findings and walks you through them.

So we gave the method away. Open Risk Review is MIT-licensed. It installs as a Claude Code plugin or a Cursor rule, and it works in any agent that can read files. It runs against the same 59 controls in 12 areas as our production readiness checklist: auth, secrets, data access, API input, billing, AI-specific risk, logging, error handling, testing, governance, deployment and supply chain.

/plugin marketplace add Stackbilt-dev/open-risk-review
/plugin install open-risk-review@open-risk-review

Then ask your agent to run an open risk review on the repo.

It doesn't change your code. The only things it writes are its report and, if you agree, one .gitignore line that keeps the report out of your repo. It never calls your live systems or uses a credential it finds. It runs inside your own tool, so your code is never sent to us. Every control gets a status, and every finding cites a file and line. Each finding also comes with a fix prompt you can paste back into your coding tool.

It is still an AI review. It can miss things, and it can be wrong. The rest of this post shows how often each of those happened.

What the open-source risk review found in our own repos

We tested it the way a stranger would use it. A fresh agent got only the skill's three files and a repository, with no notes from us and no hints about what we already knew.

Run one, FoodFiles. The run found a Critical that the September review had missed. The credits endpoint let any tenant owner, and any tenant API key, post a "credit adjustment". Every self-signup becomes the owner of their own tenant. The endpoint capped each request at 10,000 credits but set no overall limit, and the request could be repeated with a new idempotency key:

POST /v2/billing/credits/mutate
{ "mutationType": "credit_adjustment", "creditsDelta": 10000 }

Any new free user could have sent that request again and again and granted themselves unlimited credits. The same run found that sign-in followed a ?redirect= parameter without checking it, so a crafted link could send a user's freshly signed-in session to a javascript: URL.

The fix found a deeper one. The obvious fix for credits was to allow positive adjustments only for platform operators, or for API keys that carry an admin:operator scope. Every fix pull request got an adversarial review before merge. That review asked who can create a key with that scope. The answer was anyone. Our shared auth service, edge-auth, stored whatever scopes the caller asked for, and every new signup is an admin of the organisation that all of that product's customers share. A new signup could mint an operator key with one request, and the credits fix would have trusted it. The same route also let a caller bind a key to any customer's tenant.

The fixes, each with regression tests:

• edge-auth reserves *, admin and admin:* scopes for internal services (October 15), and binds a key only to a tenant the caller belongs to. The October 15 version checked the tenant against the caller's organisation, which every customer of a product shares, so the per-tenant check followed on October 16.
• FoodFiles accepts positive credit adjustments only from platform operators: an operator's session, or an admin:operator key, which, since October 15, only our internal services and operators can issue.
• FoodFiles follows a sign-in redirect only when it stays on the same site (October 15).

The credits fix had looked complete on its own. It was only safe once edge-auth stopped issuing operator scopes to signups, which shipped the same day.

Run two, the Stackbilder front end. This run was a known-answer test. We pointed it at the same commit the September review saw, before any of that review's fixes. The September review had found one Critical and three Highs there. The cold run found all four, but rated two of them Medium where the September review had said High: the login relay that trusted a token from the query string, and the operator credential sent to browsers. It also found two new Highs. The paid CTO and CISO consultations checked entitlement on the page but not on the API behind it, so any signed-in user could call the API directly. And any signup could attach files to another company's public Trust Page. We added the entitlement check to the API before publishing this post, but it didn't close the gap. While checking this post on October 17, we found that a dogfood allowlist we had set up for our own account matched our shared organisation, so both the page and the API still let every Stackbilder signup use the consultations without a paid entitlement. We removed it that day. From July 21 to September 11, the same allowlist also decided which browsers received the operator credential from the September review, so any Stackbilder signup who opened the CTO consultation page in that window could have received it, not only us. We rotated that credential on October 18, and it hasn't worked since. We then checked the Stackbilder app's request traces, which have recorded every page and API request to it since April 9. Every successful load of a consultation page was by our own account, the last in April. Nine requests to the consultation API succeeded, the last on May 28. Four, all in April, came within 45 minutes of one of our own page loads, and one is stored under our account. The other four were direct API calls charged to the organisation every Stackbilder signup shares, so we can't say whose they were. No consultation page load or API request through the app succeeded after May 28, so no browser loaded the CTO page, or received the credential through it, between July 21 and September 11. Trace writes can occasionally fail, so this is strong evidence rather than proof. Our first Trust Page fix also shipped before publishing, but it checked ownership by organisation, which every Stackbilder signup shares; the per-tenant fix went out on October 16. No files had been attached.

Run three, a public starter we didn't write. Both earlier runs were on our own Cloudflare stack, written by the same people who wrote the skill. So we also ran it on a public open-source Next.js, Supabase and Stripe starter. It reported two Highs. An independent check confirmed one, a billing flaw. It downgraded the other to Medium: an unhandled webhook event caused retries and alert noise, not lost billing data. The check found no false alarms, and it found nothing the run had missed. We aren't naming the project or giving details of the confirmed flaw. Our own README asks people not to post real vulnerabilities in other people's apps in public issues, and the same principle applies to us.

What the cold runs taught us about the skill

The third run taught us the most about the skill itself. On a stack it wasn't written for, the cold agent listed 19 places where the instructions were unclear or didn't fit a Next.js and Supabase stack. A few examples:

• "Ask the owner" had no default when nobody answers. The skill now assumes the app is live, takes payments and holds other people's data, and says so in the report's Context line, so the owner can correct it.
• The skill had no rule for missing items. It now has one: a missing item is FAIL if it belongs in the repo (tests, CI, migrations, a threat model). It's NOT VERIFIABLE if it could live in a provider dashboard (backups, alert rules, key scopes), and the report says where.
• Built-in framework behaviour needed a citation rule. A Next.js default now counts as evidence only when the report names the version and exactly what the default covers. Server actions check the request's origin; route handlers don't.
• Severity drifted on facts the reviewer couldn't see. The skill now separates two cases. A fact about the owner's own setup gets the higher rating plus a Condition line naming what would lower it. How a third-party service behaves gets checked in that service's documentation instead of guessed.
• Stack hints needed checking against the libraries. Our first draft of these fixes said @supabase/ssr's defaults covered the Secure cookie flag. The reviewer of that draft downloaded the package and showed they don't. The hint now says what the library actually sets.

Each round of fixes went through the same adversarial review as the code fixes. That took three rounds. The first round found that one of our own new hints would have produced false passes. The second found two contradictions that the first round of fixes had introduced. A review method that we couldn't apply to itself wouldn't be worth publishing.

What the paid review adds

If the free skill found a Critical our paid review missed, why pay $495?

That run counts against our paid review, and the issue we filed says so. The September review on FoodFiles found a different Critical and missed this one. A cold second pass found what the first missed, and the review of the fix found what both missed. More passes find more. The step that matters most is a person re-reading every Critical and High before anyone acts on it.

That's what the $495 Production Risk Review is. It covers the same 59 controls on one repository, with a named reviewer who verifies every Critical and High finding. It includes a remediation order and a 30-minute findings call. Turnaround is three business days after access is ready.

We also changed something since September. The September reports ended with verdicts like "hold for one fix, then go". The paid review no longer gives a readiness verdict, and neither does the skill. A review can say what it found and what it couldn't see. Deciding to launch is the owner's call. The gate that has no laws is the one nobody has read, and a verdict reads like a law to someone who hasn't read the findings.

Run it on your own repo

Open Risk Review is on GitHub at Stackbilt-dev/open-risk-review. It includes an example report for a fictional app, so you can see the output before you run it. The checklist it runs against is published in full, with the same control IDs.

If it finds something you're not sure about, ask a person you trust to look before you act. If that person should be us, the review is at stackbilder.com/production-risk-review.

We'll keep running it on our own code first. The findings in this post are the reason.

Open Risk Review: https://github.com/Stackbilt-dev/open-risk-review · Checklist: https://stackbilder.com/production-readiness-checklist Production Risk Review: https://stackbilder.com/production-risk-review · Stackbilder: https://stackbilder.com · FoodFiles: https://foodfiles.app