Building a Production-Ready Kubernetes Performance Testing Framework

Building a Kubernetes cluster is easy; proving it's production-ready is hard. How do you know if your control plane can scale? Is your storage actually delivering the IOPS promised by the vendor?

To answer these questions and ensure the cluster is ready for production, I researched and gathered information on setting it up. Then, I built a performance test framework to do it.

In this post, I'll walk you through the Objective, the Test Flow, and the Requirements needed to set up this framework.

TL;DR 👉 github.com/nh4ttruong/k8s-perf-tests

🎯 The Objective

The goal is to ensure the Kubernetes cluster meets specific performance requirements across all layers:

• Control Plane: API Server handles expected request volume.

• etcd: Database meets throughput and latency requirements.

• Network: Pod-to-pod communication achieves expected bandwidth and latency.

• DNS: CoreDNS handles query rate from services.

• Storage: Persistent Volume meets IOPS and throughput requirements.

• Ingress: Load balancer handles external traffic.

🔄 The Test Flow

We don't test randomly. We execute in a specific order, moving from the infrastructure layer up to the application layer. If the foundation (etcd) is shaky, testing the Ingress is pointless.

1. etcd → Database performance (foundation for everything)

2. API Server → Control plane capacity

3. Network (CNI) → Pod networking performance

4. CoreDNS → Service discovery latency

5. Storage → Persistent volume I/O

6. Ingress → External traffic handling

🛠 Components & Tools

Here is the stack we use to validate each component. Click on the component name to view the source code and detailed documentation:

Test types

For each component in the following posts, we will look at three types of tests:

1. Smoke: Validate configuration (1-5 min).

2. Load: Measure performance at expected load (15-60 min, 70-100% capacity).

3. Stress: Find the breaking point (10-30 min, 150-200% capacity).

Quick Start

# Clone repository
git clone https://github.com/nh4ttruong/k8s-perf-test.git
cd k8s-perf-test

# 1. etcd smoke test (if you have access)
./etcd/scripts/etcdctl/smoke.sh

# 2. API Server smoke test
kube-burner init -c kube-burner/api-server/smoke.yaml

# 3. Network smoke test
k8s-netperf --config network/smoke_1.yaml --local

# 4. DNS smoke test
kubectl apply -f coredns/
kubectl logs -f -l app=dnsperf -n coredns-perf-test

# 5. Storage smoke test
kubectl create namespace storage-perf-test
kubectl apply -f storage/smoke.yaml -n storage-perf-test

Evaluation Criteria

Kubernetes SLIs/SLOs

The Kubernetes project defines Service Level Indicators (SLIs) and Service Level Objectives (SLOs) for a properly functioning cluster. These criteria are used for performance evaluation.

Reference: Kubernetes Scalability SLIs/SLOs

API Server SLOs

Pod Startup SLOs

Target Metrics by Component

Specific evaluation criteria for each component:

Control Plane

Data Plane

Storage

Ingress

Result Evaluation

Pass/Fail Criteria

Pre-Production Checklist

• API Server P99 latency < 500ms under expected load

• etcd write latency P99 < 25ms

• No etcd leader election during 24h test

• Pod startup time P99 < 5s (image cached)

• DNS query latency P99 < 10ms

• Storage IOPS meets workload requirements

• Network throughput meets inter-zone requirements

• Ingress handles expected peak traffic

References

• Kubernetes Scalability SLIs/SLOs - Official SLO definitions

• Kubernetes Scalability Thresholds - Cluster size limits

• SIG Scalability - Scalability working group

• kube-burner Documentation

• etcd Performance

• etcd Tuning

• k8s-netperf

• fio Documentation

• CoreDNS

• NGINX Ingress Controller

---

In the next post, we start with the heart of the cluster: etcd component → Kubernetes etcd Performance Benchmarks