Post We Sold a $495 AI App Production Risk Review, Then Ran It on Ourselves Three Times
We ran our $495 AI app production risk review on three of our own production repos before selling one. It found a Critical in each of the first two, a root cause in the third, and two fixes shipped the same day. What a paid review has to get right.
Update, October 15, 2026: we've since open-sourced this review method as Open Risk Review, a free skill your own AI coding tool runs on your repo. Run cold on the same codebases, it found a Critical this review missed. Read the follow-up.
We sold a $495 AI App Production Risk Review, verified the checkout with a real card, and then ran the review on our own three production repos before selling a single one. It found a Critical in each of the first two, a root cause in the third, and two of the three fixes shipped the same day. This post is what it found, what a production risk review has to get right to be worth paying for, and the one mistake we made in the process.
Why we ran the AI app production risk review on ourselves first
The offer is simple. Read-only access to one repo, a review against our published production readiness checklist, four deliverable sections, and a thirty-minute findings call. Three business days. We built it for teams shipping AI-assisted apps fast, the ones where Cursor or Claude Code wrote the auth middleware and nobody has read it since.
We have written before about the difference between merged and live, and about trusting an agent to touch production. A paid review is a promise about exactly those gaps. Before promising it to someone else, we had to know what the deliverable looked like when it hit a real codebase, whether the severity scale meant anything, and whether the pipeline that produces it could be repeated by someone who wasn't there for the first run.
So the first buyer was us. The checkout smoke test paid ninety-nine cents through a single-use promo code, the webhook provisioned the purchase, and both emails landed. Then we pointed the review at FoodFiles, at the Stackbilder platform front end, and at the shared auth and billing service underneath both.
What the production risk review found in three Stackbilt repos
Run one, FoodFiles. One Critical. An admin endpoint that writes platform-wide configuration accepted any user whose role in their own workspace was "owner." Every self-signed-up customer is the owner of their own workspace. So every customer could zero the public rate limits for everyone. The write was audited, so we would have found out afterwards. Nothing stopped it. The fix is a one-middleware swap. Across the 59 checklist controls: 28 pass, 11 partial, 11 fail, 9 not verifiable because they live in the auth provider.
Run two, the platform front end. One Critical and three Highs. The promo-code routes required a login but not admin, so any signed-in user could list every promotion code, revoke any of them, or mint a 100%-off code with no redemption cap. The login relay trusted whatever session token it received in a query string. An operator credential for our internal agent was being handed to browsers behind a dogfood allowlist. And deploys ran from a laptop with no test gate, twelve of thirteen end-to-end suites never ran in CI, and their default target was production. We also learned, from the review, that production had been deployed from an unmerged branch the night before.
Run three, the auth and billing service. Zero Critical by our scale and seven Highs. The RPC surface that every product calls over service bindings has no notion of which caller is asking. Forty-five of sixty-one methods act on whatever tenant or user ID they are handed. That is the root cause of both consumer Criticals: a consumer bug that forwards client input becomes a platform-wide compromise. The same run found stored payment credentials encrypted under a key derived mostly from public material, a public demo-quota endpoint that trusted an IP address from the request body, and user display names going into HTML emails unescaped, including the one that lands in our own inbox on every signup.
The verdicts were "hold for one fix, then go" twice and "go for today's consumers, hold new ones" once. None of the three was "stop." All three had more strong controls than weak ones. Tenant scoping on every query, signature-first webhook handling, credit reservations that cannot replay across tenants, constant-time credential checks, hermetic test suites in the hundreds. A review that only lists failures is not a review a buyer can act on.
What shipped the same day, and what stopped at a boundary
Two of the four pre-promotion fixes on the front end merged and deployed within hours: the admin gate on the promo routes, with nine regression tests, and the removal of the operator credential from the page. The smoke test on the live site confirmed anonymous callers now get 401 on all three promo operations.
Three of the auth-service Highs went into a pull request the same evening: HTML escaping at every email interpolation, the demo-quota endpoint keyed on the connection IP the middleware already resolves, and the deletion of a deprecated "repair" method that could grant a paid tier from a fabricated subscription ID and had no callers. That PR stopped at a boundary rather than deploying, because deploying the auth service under every product is a human decision.
The FoodFiles Critical is being fixed in its own session. The RPC trust model is a design decision, identity on the binding versus splitting the entrypoint, and it is the first item on the findings-call agenda.
Every finding is tracked in the repos it belongs to. Fifteen issues on the auth service, twelve on the front end, ten on FoodFiles, each with the evidence at file and line, the fix, and acceptance checks copied from the report's verification column. The tracker for each repo puts the pre-promotion set at the top. This is what audit trails for autonomous agents look like when the agent is the reviewer.
What a production risk review has to get right to be worth paying for
Three runs taught us more about the deliverable than about the code. These are the rules the review now runs on.
Define the severity scale before the review starts, and make the lead enforce it. Critical means exploitable now by an unprivileged party with platform-wide impact. High means exploitable now, or a regression that ships to customers without a human noticing. On the auth service, the review passes reported five Criticals. Every one required a caller that already holds a service binding. All five were rated High, with the root cause named. The findings did not get less serious. The number got true. A buyer who sees "Critical" on a report and then learns it needed insider access will not trust the next report.
Re-verify every Critical and High by hand, across repos if needed. The FoodFiles Critical only became a Critical after reading the auth service's provisioning code and confirming that every signup becomes a workspace owner. The operator-token finding was reported as Medium until tracing the socket handler in the agent's repo showed the token gates every route, not just voice. Agents surface candidates. Someone still has to walk the chain.
Mark what you could not see, and say where it lives. Ten of FoodFiles' controls sit in the auth service. Twelve of the front end's do. A coverage table that silently marks those "pass" is a lie, and one that marks them "fail" is unfair. "Not verifiable, owned by edge-auth" is the honest cell, and it doubles as the scoping for the next engagement.
Reproduce with the product's own client settings before calling a third-party API broken. This is the mistake. During the checkout test we hit Stripe with a raw call to mint a promo code and got a parameter error. We filed it as a bug in the billing service and wrote, in two reports, that the promo hole was "blocked by accident" by that bug. It was not. The service pins a Stripe API version under which the old parameter is correct. The product path worked the whole time, which means the hole was fully live until the fix deployed. We corrected both reports, closed the issue as not-a-bug, and put the lesson in the skill. Evidence over confidence applies to the reviewer too.
Report what held up. Buyers need to know what not to touch as much as what to fix. The strongest control in each codebase went in the executive summary, above the fold.
Ship the pre-promotion set in about a day. If the first four fixes take a sprint, the review found architecture debt, not a launch blocker, and the verdict should say so.
The pipeline itself is what we described in naming agent failure modes: one survey pass to map the repo, four adversarial passes split by control area, each forced to cite file and line and to separate confirmed findings from suspicions, then lead verification and synthesis. About twenty-five minutes of wall time for the passes, about an hour of session time for the whole thing. The three-day turnaround is for access and the call, not the work.
The broader lesson: review the thing you sell before you sell it
We did the same with the GEO audit: the first run of that tool found two real problems on our own site. It is not a coincidence that both offers found their most useful bugs at home. A review product is a claim about what a careful outsider would find. The cheapest way to test the claim is to be the outsider on your own code and see whether the report would have been worth paying for.
It was. The front-end Critical had been live for as long as the promo routes existed, and the live promotion-code list shows only our own internal codes, so nobody found it first. That is luck, and luck is not a control. The gate that has no laws is the one nobody has read, and AI-assisted codebases accumulate those faster than any other kind.
If you are shipping an AI app and the auth middleware was written by an agent, the review is at stackbilder.com/production-risk-review. Read-only access, three business days, four sections, one call. The first three reports are ours.
Production Risk Review: https://stackbilder.com/production-risk-review · Checklist: https://stackbilder.com/production-readiness-checklist Stackbilder: https://stackbilder.com · FoodFiles: https://foodfiles.app · AEGIS: https://aegis.stackbilt.dev