AI Gateway

Nantian Gateway can handle AI traffic in the data plane so applications can route through one Kubernetes Gateway while the platform controls providers, credentials, token quotas, and observability.

What The AI Gateway Does

The AI Gateway feature is built around two Kubernetes resources:

Resource	Purpose
`AIService`	Defines one AI provider/model target, including provider format, model name, auth, timeout, retry, and observability settings.
`TokenPolicy`	Attaches token and request limits to local Gateway API targets such as an `HTTPRoute`.

The runtime includes modules for OpenAI, Anthropic, and Ollama format handling, token accounting, rate limiting, prompt and PII processing, semantic cache, fallback, A/B testing, cost tracking, and multitenant context. Treat those as runtime modules unless a stable CRD field explicitly exposes the behavior you want to configure.

Capabilities

The AI Gateway provides the following capabilities, each configurable via CRD:

Model Routing

Route requests to different LLM models based on complexity classification, request attributes, or header-based selection. Read more

Semantic Cache

Cache LLM responses for similar prompts using embedding similarity to reduce redundant API calls and costs. Read more

Prompt Guard

Detect and block prompt injection attempts and suspicious keyword patterns before requests reach the model. Read more

Content Safety

Filter violent, hateful, self-harm, exploitation, and illegal content from both prompts and model responses. Read more

PII Masking

Detect and mask personally identifiable information such as email addresses, phone numbers, and credentials in prompts and responses. Read more

A/B Testing

Split traffic between model variants using weighted selection to compare performance, cost, and quality across providers. Read more

Fallback Chains

Configure backup models in sequence so requests cascade to fallback providers when the primary model fails or times out. Read more

Cost Tracking

Monitor token usage and estimated cost per model, per tenant, and per route for chargeback and budget analysis. Read more

Multi-Tenant

Map multiple API keys to tenants, each with their own model allowlists, rate limits, and usage quotas. Read more

Token Policies

Enforce per-minute and per-hour token rate limits on AI routes, with configurable burst multipliers and rejection strategies. Read more

Langfuse Integration

Send traces, generations, and scores to Langfuse for LLM observability and debugging of model performance. Read more

Enablement

Start from Experimental Features and make sure the required CRDs exist:

kubectl get crd aiservices.gateway.nantian.dev
kubectl get crd tokenpolicies.gateway.nantian.dev

With Helm, the minimal AI Gateway values are:

featureMode: experimental
controlplane:
  config:
    features:
      enableAiGateway: true

Configure An AI Provider

AIService describes the provider and model target. The provider can use formats such as OpenAI, Anthropic, or Ollama, depending on the backend you route to.

apiVersion: gateway.nantian.dev/v1alpha1
kind: AIService
metadata:
  name: openai-gpt4o
  namespace: nantian-demo
spec:
  provider: openai
  format: openai
  model: gpt-4o
  auth:
    type: bearer
    secret: openai-api-key
    key: token
    header: Authorization
  timeout: 60s
  retry:
    maxRetries: 2
    backoff: 500ms

The control plane translates the resource into runtime configuration distributed to data plane instances. Keep provider credentials in Kubernetes Secrets and avoid embedding tokens in route manifests.

Enforce Token Quotas

AI traffic often needs token-aware limits instead of request-only limits. TokenPolicy can attach limits to a route:

apiVersion: gateway.nantian.dev/v1alpha1
kind: TokenPolicy
metadata:
  name: ai-route-quota
  namespace: nantian-demo
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: ai-route
  tokensPerMinute: 100000
  tokensPerHour: 5000000
  requestsPerMinute: 1000
  scope: route
  burst: 1.5
  onLimit: reject

Use request limits for traffic shape, token limits for cost control, and a clear onLimit behavior so callers get predictable failure modes.

Observe AI Traffic

AIService can carry Langfuse and OpenTelemetry settings. Nantian Gateway does not bundle those backends; point the fields at systems you operate.

apiVersion: gateway.nantian.dev/v1alpha1
kind: AIService
metadata:
  name: observed-openai
  namespace: nantian-demo
spec:
  provider: openai
  format: openai
  model: gpt-4o
  observability:
    langfuse:
      host: https://langfuse.example.com
      publicKey: langfuse-public
      secretKey: langfuse-secret
    otel:
      endpoint: http://otel-collector.observability.svc:4317
      serviceName: nantian-ai-gateway

Combine AI-specific telemetry with the standard metrics and troubleshooting workflows.

Production Checklist

Enable experimental mode only in clusters where you can tolerate API changes.
Install and verify AIService and TokenPolicy CRDs before applying manifests.
Store provider credentials in Kubernetes Secrets.
Start with request and token limits that fail closed.
Watch control plane and data plane logs after rollout.
Confirm metrics and traces before routing production AI traffic.