Qwake guide

Test an OpenAI-compatible relay with Qwake

Build a trusted reference profile, sample a target endpoint, and compare answer distributions in a reviewable local relay audit.

This guide uses Qwake's experimental fingerprint commands to audit an OpenAI-compatible relay endpoint. Only test services you are authorized to access. Confirm that sampling is permitted by the service terms and estimate the cost first: the default mini preset makes 120 very short requests in one language.

Start with a sound baseline

Prepare a trusted reference endpoint first. Ideally this is a clearly versioned official API from the model vendor; it can also be an endpoint you have independently verified and whose configuration is stable. An untrusted reference makes the comparison meaningless. Keep the model name, preset, language, temperature, and max-tokens settings identical at both endpoints.

1. Install Qwake and prepare API keys

npm install -g @sysiphus/qwake

# Set only in the current shell. Do not put keys in screenshots or commits.
export OFFICIAL_API_KEY='replace-with-your-key'
export RELAY_API_KEY='replace-with-your-key'

Qwake reads keys through --api-key-env rather than accepting them on the command line. It does not write keys to ~/.qwake/fingerprints, but the shell environment still needs to be protected.

2. Collect and enroll a trusted reference

qwake fingerprint collect \
  --base-url https://api.example.com/v1 \
  --api-key-env OFFICIAL_API_KEY \
  --model gpt-4o \
  --preset mini \
  --languages en \
  --samples 15

The command prints the path to a run JSON file. Turn that run into a clearly named reference profile:

qwake fingerprint enroll \
  --name gpt-4o-official-2026-07 \
  --from ~/.qwake/fingerprints/runs/<run>.json

For a model that supports Chinese, collect a separate run with --languages en,zh. The two sides of every comparison must match exactly. When cost allows, use --preset full and more samples to establish a more stable baseline.

3. Audit the relay endpoint

qwake fingerprint audit \
  --claim gpt-4o-official-2026-07 \
  --base-url https://relay.example.com/v1 \
  --api-key-env RELAY_API_KEY \
  --model gpt-4o \
  --preset mini \
  --languages en \
  --samples 15

This collects a fresh run, derives an audit profile, and prints a report. The report uses likely_match for a close distribution, suspicious_drift for a difference worth investigating, and likely_mismatch for substantial divergence. When too few comparable cells are available it returns inconclusive.

4. Treat an anomaly as a lead

Repeat the measurement before concluding anything. Check that model label, version, region, temperature, max tokens, language, and time window are consistent. Then investigate relay-side caching, load balancing, fallback behavior, or hidden reasoning. Only persistent drift after those controls is good grounds for a technical question to the provider, accompanied by the run files, configuration, and timestamps.

A durable practice is to establish a fresh reference after material model or provider changes, periodically compare new samples to the saved baseline, and treat the result as an observation log. It should help you ask better questions, not replace a provider's transparency commitments.

Where evidence lives

~/.qwake/fingerprints/
  runs/       # normalized results for each collection
  profiles/   # named reference profiles

qwake fingerprint compare \
  --left gpt-4o-official-2026-07 \
  --right ~/.qwake/fingerprints/profiles/another-run.json

Run files do not contain an API key, but they can contain endpoint URLs, model names, and answer statistics. Review them against your organization's security policy before sharing them outside the team.