Skip to main content

GenAI Smart Router

GenAI Smart Router is a self-managed gateway for OpenAI Chat, OpenAI Responses, Anthropic Messages, VLM, and tool-capable agent traffic. Applications call one stable endpoint while deployment operators control provider credentials, model-group access, request-shape eligibility, quotas, routing policy, and usage accounting.

Why Use It​

  • Keep provider credentials server-side and give callers revocable scoped router tokens.
  • Change validated provider/model mixes behind deployment-defined model groups without rewriting every client.
  • Filter tool, image, dialect, context, and output-cap requests to targets validated for those shapes.
  • Enforce RPM, TPM, concurrency, and request/token budgets (and per-request estimated cost ceilings where configured) before upstream calls. These are not cumulative dollar spend limits or multi-replica atomic spend accounting.
  • Preserve sanitized request IDs, attempts, latency, token, fallback, and request-time cost evidence for operations.
  • Combine private OpenAI-compatible servers (vLLM, SGLang, on-prem GPUs) with commercial and frontier APIs behind one caller token, model-group contract, runtime license, and usage database.

Where It Runs​

Operators run the router on-prem, in a customer cloud or VPC, or hybrid. The product is self-managed: you keep keys, licenses, and traces. Applications do not call providers directly.

Request Flow​

The router does not guarantee model quality or provider availability. Operators must validate each active provider/model/API combination and retain a rollback path. See Architecture, Platforms, And Limitations.

Start Here​

  1. Follow the Local Quickstart for a ten-minute laptop path (bootstrap, license, caller, one Chat request).
  2. Choose Docker Compose, binary, or Kubernetes for a durable deployment.
  3. Generate or renew the self-managed runtime-policy license as in Self-Managed Licensing.
  4. Discover allowed groups with /v1/models.
  5. Use the API Quickstart against the running endpoint, then the exact Chat, Responses, Messages, tool, image, or streaming shapes your clients use.
  6. Review security boundaries, observability, and the upgrade guide before production traffic.

Questions and bugs use the repository's public issue forms. Suspected vulnerabilities must use GitHub private vulnerability reporting; do not place secrets or customer content in public issues.