A self-serve billing portal that cut support tickets in half.

Lumen’s merchants loved the core payments rail, but every “simple” account change routed through support. The legacy portal mixed admin and end-user flows, leaked permission boundaries, and terrified engineering every release window. Stripe objects drifted from the internal ledger after manual fixes — a quiet risk nobody wanted on a slide.

The trigger was a failed SOC observation about separation of duties. Timeline: nine weeks to a phased launch before renewal season. Budget capped contractors; we embedded with their existing team.

We refused a rewrite-the-world proposal from an internal skunkworks group. The constraint was prove safety weekly — not hero architecture.

// TODO: Verify with client legal before publishing — audit context, timelines, and ticket metrics.

Framing. A payments portal is a controls product: every mutation is an auditable event. That framing changed UI copy, engineering tests, and how PMs triaged scope.

Architecture. Next.js with explicit server actions for mutations, PostgreSQL as the reconciliation store of record, Stripe as the payments engine with typed idempotency keys. We introduced a “two-person rule” simulation in staging for high-risk roles before exposing it to customers.

Phasing. Weeks 1–2: joint threat model + data map. Weeks 3–6: vertical slices — payment methods, invoices, seats. Weeks 7–9: hardening, chaos tests on webhook delays, and a canary cohort.

Judgment. We delayed “pretty charts” to ship immutable audit trails first — politically hard, technically correct.

// TODO: Verify with client legal before publishing — security controls and process descriptions.

Billing-related support tickets dropped 51% in the first ninety days post-cutover versus the prior baseline, while self-serve actions (payment updates, downloads, plan changes merchants were allowed to perform themselves) rose 37%. Checkout errors reported by clients fell 38% once retries and idempotency were aligned.

Engineering velocity recovered because releases stopped being “call the billing team and pray.” Lumen extended the team for hardening sprints through renewal; we handed off runbooks their SOC relied on in follow-up sampling.

// TODO: Verify with client legal before publishing — ticket taxonomies and error definitions.

We should have instrumented support macro usage earlier — it masked product gaps and fooled us about true ticket drivers. We were surprised how much merchants cared about PDF footers and invoice numbering — small UX, huge trust.

Methodology tweak: we now pair a technical writer with engineering during week one for error strings; saves a retrofit sprint. We would also schedule executive demos on failed paths, not happy paths, sooner.

// TODO: Verify with client legal before publishing — qualitative reflections tied to a real client programme.