Benchmarking LLM Inference with Production Agent Traces
https://medium.com/inference-perf/benchmarking-llm-inference-with-production-agent-traces-f47f7f994aff
Most load-testing tools, however, were designed for a simpler world. They assume requests are independent, stateless, and interchangeable. But an AI agent doesn't work that way. This gap was the motivation behind OpenTelemetry (OTel) Trace Replay, a new capability in Inference Perf, the Kubernetes SIG tool for GenAI inference benchmarking.
https://medium.com/inference-perf/benchmarking-llm-inference-with-production-agent-traces-f47f7f994aff