runConformanceSuite
runConformanceSuite({ targetBaseUrl })
Runs the full battery against a gateway and returns pass/fail per check.
Is your gateway actually OpenAI-compatible?
A gateway that drops the last streamed chunk still looks like it worked in a quick manual test. One that miscalculates cost still returns a number. One that swallows an upstream error and reports finish_reason "stop" anyway looks exactly like success. None of these announce themselves.
gatewayproof answers one question: does this gateway actually preserve streaming, cost, and errors -- or just look like it does?
Test against ground truth, not a live model
gatewayproof starts a deterministic mock upstream that speaks the real /v1/chat/completions API but scripts every response ahead of time. You point the gateway under test at that mock, then gatewayproof sends requests through your gateway and compares what comes back.
src/check.ts
import { runConformanceSuite } from "gatewayproof";
const report = await runConformanceSuite({
targetBaseUrl: "http://localhost:8080",
});
if (!report.passed) {
console.error(report.summary);
process.exit(1);
}One suite, four failure classes
streaming / usage checks
checks: streaming, usage, cost
Catches dropped chunks and miscalculated token usage or cost.
concurrency / error checks
checks: concurrency, error propagation
Catches swallowed upstream failures reported as successful completions.
Why this exists
An LLM gateway has to re-stream chunks, recompute token usage, calculate cost, and correctly propagate upstream failures on every single request -- stateful, timing-sensitive logic in the hot path. There was no independent, reusable way to check for this class of bug before deploying a gateway or a change to one.
Also ships as a GitHub Action, published on the GitHub Marketplace.
gatewayproof → requests → [gateway under test] → mock upstream
responses ←——————— responses ←————
streaming pass · usage/cost pass
error handling fail · concurrency pass
What gatewayproof doesn't do
- Can't test against a live provider.There's no ground truth to compare against when a real LLM is involved.
- Not a load-testing tool.It checks conformance, not throughput or latency under load.
- Doesn't fix your gateway.It tells you exactly what's wrong; the fix is yours.