Performance Tuning

This page covers configuration options that affect throughput, latency, and resource usage in the data plane. The default values are appropriate for moderate traffic volumes (hundreds to low thousands of requests per second). For higher throughput scenarios or resource-constrained environments, the parameters described here enable tradeoffs between performance, memory consumption, and operational safety.

Threading and Concurrency

The data plane uses an async runtime built on Tokio. The following settings control how work is distributed across threads:

Parameter	Type	Default	Description
runtime_tuning.http_capacity.workerThreads	int	0	Worker thread count (0 = CPU count)
runtime_tuning.http_capacity.acceptConcurrency	int	16	Concurrent TCP accept operations

Worker Threads

The default value of 0 causes the runtime to create one worker thread per available CPU core. This configuration is optimal for CPU-bound workloads where the proxy performs significant processing: TLS termination, header manipulation, and Wasm filter execution all benefit from parallel CPU access.

For I/O-bound workloads where most time is spent waiting for upstream responses, reducing the worker count frees CPU resources for other processes. Set this to a value lower than the CPU count when the proxy primarily forwards requests with minimal transformation.

Accept Concurrency

acceptConcurrency controls how many threads can simultaneously invoke accept() on the listener socket. The default of 16 handles connection bursts effectively. Increase this value for workloads with very high connection rates (tens of thousands of new connections per second). Decrease if profiling reveals accept-related contention.

Connection Pools

Reusing connections to upstream services avoids the overhead of TCP handshakes and TLS negotiation on every request. Connection pooling is critical for latency-sensitive workloads.

Parameter	Type	Default	Description
runtime_tuning.http_capacity.upstreamKeepalivePoolSize	int	32768	Maximum idle connections in the upstream pool

The upstream keepalive pool maintains idle connections to backend services. When the data plane needs to forward a request, it checks the pool for an existing connection before establishing a new one. Each idle connection consumes a small amount of memory (several kilobytes for socket buffers), so the memory cost scales linearly with pool size.

Pool sizing guideline

A useful starting point for pool sizing: poolSize = number_of_backends × average_concurrency_per_backend × 0.5. The 0.5 factor accounts for the fact that not all connections are idle simultaneously. Monitor the upstream_keepalive_pool_hits and upstream_keepalive_pool_misses metrics to verify or adjust the configuration.

HTTP Response Cache

The data plane can cache upstream responses. The entry-size limit controls the largest single response that may be stored:

Parameter	Type	Default	Description
runtime_tuning.http_cache.maxEntrySizeMb	int	16	Maximum size (MiB) of a single cached response entry

Responses larger than maxEntrySizeMb bypass the cache and stream directly to the client. Raise this value for workloads that serve large, cacheable payloads; lower it to bound per-entry memory use.

Timeouts

Timeout configuration prevents resource leaks and ensures the proxy does not hold connections indefinitely. These are flat keys directly under runtime_tuning (there is no nested timeout object), and all values are in milliseconds:

Parameter	Type	Default	Description
runtime_tuning.downstreamReadTimeoutMs	int	60000	Maximum time to read the request from the client
runtime_tuning.upstream_connection_timeout_ms	int	5000	Maximum time to establish a connection to the upstream
runtime_tuning.upstream_read_timeout_ms	int	30000	Maximum time to read the response from the upstream
runtime_tuning.upstream_idle_timeout_ms	int	60000	Maximum idle time before closing an upstream keepalive connection

Lower upstream_connection_timeout_ms to fail fast when backends are unreachable; raise upstream_read_timeout_ms for slow or long-running upstream endpoints.

TLS

The proxy runtime does not expose TLS session-cache tuning knobs. TLS protocol version bounds (runtime.tls_min_version / runtime.tls_max_version) and certificate/mTLS settings are covered in TLS Configuration.

Profiling and Monitoring

Before adjusting tuning parameters, establish a performance baseline using the available metrics:

1. Latency distribution — track p50, p95, and p99 response times via the data plane metrics endpoint
2. Error rates — monitor HTTP 5xx responses and connection errors
3. Resource usage — observe CPU and memory consumption at current traffic levels
4. Connection pool efficiency — track upstream_keepalive_pool_hits vs upstream_keepalive_pool_misses

Make parameter changes incrementally, measuring the impact on each metric before proceeding to the next adjustment. Document the baseline and each change for future reference.

Memory Allocator

The data plane links against jemalloc for production builds, which provides better memory fragmentation behavior and lower tail latency compared to the system allocator. The allocator is selected at compile time via the --features jemalloc flag.

For workloads with very large numbers of concurrent connections, jemalloc’s thread-cache and background thread features reduce allocation contention. Monitor nantian_gw_dataplane_container_memory_working_set_bytes to verify the allocator is performing well under your traffic profile.

Buffer and I/O Tuning

For high-throughput deployments, consider adjusting these environment-level knobs:

Parameter	Type	Default	Description
runtime_tuning.http_capacity.bufferPoolSize	int	1024	Number of reusable buffer slabs
runtime_tuning.http_capacity.maxHeaderSize	int	65536	Max header bytes per request (64 KiB)
runtime_tuning.http_capacity.maxBodySize	int	10485760	Max body bytes per request (10 MiB)

Increase bufferPoolSize for workloads with large, bursty request volumes to reduce per-request allocation overhead. Reduce maxBodySize if your upstreams only accept small payloads — this limits the memory window for malformed requests.

Next Steps

• Observability — configure metrics, logs, and traces for monitoring
• Metrics Reference — detailed metric descriptions
• Production Deployment — production resource sizing guidance