Observability

Nantian Gateway emits structured logs, Prometheus metrics, and optional OpenTelemetry traces from both the control plane and data plane. The data plane also supports Sentry error tracking. This page covers the configuration of each observability signal.

Logging

Both components use structured logging with JSON as the default format. Structured logs integrate directly with log aggregation systems such as Loki, Elasticsearch, and Datadog.

Control Plane Logging (Go)

The control plane uses Go’s log/slog package:

Parameter	Type	Default	Description
log.level	string	info	Minimum log level: debug, info, warn, or error
log.format	string	json	Output format: json or text
log.add_source	bool	false	Include source file and line number in log entries

The debug level includes reconciliation details, configuration snapshot diffs, and gRPC stream events. This level generates substantially more output than info and should not remain enabled in production.

When log.add_source is enabled, each log entry includes the Go source file and line number. This adds a small overhead per log call but significantly aids debugging.

Each JSON log entry contains the fields time, level, msg, and context-specific attributes. Use the text format for local development where logs are read directly rather than processed by aggregation tools.

Data Plane Logging (Rust)

The data plane uses Rust’s tracing framework with fine-grained per-module control:

Parameter	Type	Default	Description
log.level	string	info,nantian_core::connectors=off	Tracing filter directives
log.format	string	json	Output format: json or text
log.add_source	bool	false	Include source location
log.include_target	bool	false	Include the tracing target (Rust module path)
log.include_thread_ids	bool	false	Include thread IDs
log.include_thread_names	bool	false	Include thread names
log.non_blocking	bool	true	Use non-blocking log output
log.non_blocking_buffered_lines	int	65536	Ring buffer capacity
log.drop_when_full	bool	true	Drop logs when the buffer is full

The level field accepts tracing filter syntax for per-module granularity. For example, info,hyper=warn,tower=debug sets the global level to info, reduces hyper to warn, and increases tower to debug. The default suppresses verbose output from nantian_core::connectors.

Non-blocking logging is essential for a high-throughput proxy. A slow log sink (congested disk or network) can stall proxy threads when using blocking output. The ring buffer absorbs bursts, and drop_when_full ensures the proxy continues operating if the buffer overflows.

Metrics

Both planes expose Prometheus metrics on configurable endpoints.

Control Plane Metrics

The control plane exposes an HTTP metrics endpoint at the configured path (default: /metrics). Metrics include:

• Reconciliation counters — count of Kubernetes resource events processed, errors encountered, and configuration snapshots generated
• gRPC stream metrics — active connections, messages sent and received, and connection duration
• Go runtime metrics — goroutine count, memory allocation, and GC statistics

Data Plane Metrics

The data plane exposes Prometheus metrics on its admin listener at /metrics. There is no separate metrics server to enable or configure — the endpoint shares the admin address:

Parameter	Type	Default	Description
admin_addr	string	"127.0.0.1:19080"	Admin listener address; serves /metrics alongside health endpoints

In Kubernetes, the Helm chart exposes this through the dataplane metrics Service on port 19080.

Key data plane metrics:

• Request counters — total requests, grouped by HTTP method, status code, and route
• Latency histograms — request duration percentiles (p50, p95, p99)
• Connection metrics — active connections, connection rate, and connection duration
• Upstream metrics — backend connection pool hits/misses, backend latency, and backend errors
• AI gateway metrics — token counts, provider latency, and rate limit enforcement

For a complete metric reference, see Metrics Reference.

Tracing

OpenTelemetry tracing is optional and is configured under log.open_telemetry in the data plane configuration:

Parameter	Type	Default	Description
log.open_telemetry.enabled	bool	false	Enable OpenTelemetry tracing
log.open_telemetry.endpoint	string	""	OTLP collector endpoint
log.open_telemetry.protocol	string	"grpc"	OTLP transport (grpc or http)
log.open_telemetry.timeoutMs	int	3000	Export timeout in milliseconds
log.open_telemetry.insecure	bool	false	Disable TLS when connecting to the collector
log.open_telemetry.sampleRatio	float	1.0	Sampling ratio (0.0 to 1.0)
log.open_telemetry.serviceName	string	"nantian-dataplane"	Service name in trace data
log.open_telemetry.serviceNamespace	string	""	Service namespace in trace data

When enabled, the data plane propagates W3C trace-context headers and exports spans to the configured OTLP collector. Traces cover the request lifecycle: route matching, header transformation, upstream request, and response.

sampleRatio controls the fraction of requests that generate traces (default 1.0 traces all requests). For high-traffic production, lower this to reduce trace volume and storage cost.

Sentry Error Tracking

The data plane supports Sentry error tracking for full APM: error/panic capture, performance tracing, breadcrumbs, and release tracking.

Parameter	Type	Default	Description
log.sentry.enabled	bool	false	Enable Sentry error tracking
log.sentry.dsn	string	""	Sentry DSN (Data Source Name)
log.sentry.environment	string	""	Environment name (production, staging, development)
log.sentry.sample_rate	float	1.0	Event sampling rate (0.0 to 1.0)
log.sentry.traces_sample_rate	float	0.01	Performance transaction sampling rate (0.0 to 1.0)
log.sentry.attach_stacktrace	bool	true	Attach stack traces to error events
log.sentry.send_default_pii	bool	false	Send personally identifiable information
log.sentry.debug	bool	false	Enable Sentry SDK debug logging

Sentry is disabled by default. To enable, set log.sentry.enabled to true and provide a valid log.sentry.dsn. The DSN is available from your Sentry project settings.

When enabled, the data plane captures:

• Errors and panics — all Rust panics and unhandled errors are captured and reported
• Performance traces — transactions and spans from request processing
• Breadcrumbs — context events leading up to errors
• Release tracking — correlates events with the deployed data plane version

Sentry is an additional consumer of tracing events and does not replace existing OpenTelemetry tracing.

Prometheus Scraping

To configure Prometheus to scrape the data plane metrics:

scrape_configs:
  - job_name: nantian-gw-dataplane
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [nantian-gw]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
        action: keep
        regex: nantian-gw-dataplane
      - source_labels: [__meta_kubernetes_pod_container_port_number]
        action: keep
        regex: "19080"

Next Steps

• Metrics Reference — complete reference of all available metrics
• Grafana Dashboard — pre-built dashboards for visualization
• Alerting Rules — recommended alerting configurations