TensorBench measures AI GPU costs from throughput, power and utilization
The Apache 2.0 benchmark turns Llama 3 8B INT8 throughput and hardware telemetry into a cost-per-million-tokens estimate, with important hardware and scoring limits.
2 min read
Tensor Machines has released TensorBench, an Apache 2.0 infrastructure benchmark that connects accelerator throughput and power use to an estimated cost per million generated tokens. Instead of ranking GPUs from peak specifications alone, the suite runs compute, memory, inference, fine-tuning, convolution, FFT and collective-communication workloads while collecting GPU and platform telemetry.
The current headline score comes from one phase: Llama 3 8B INT8 inference. TensorBench measures generated tokens per hour and power above idle, then combines them with GPU ownership or rental cost, reserved power capacity, electricity price and expected utilization. It reports a per-GPU score and uses the median across GPUs. Raw logs, telemetry and the inputs behind the calculation remain available locally for inspection.
The repository currently provides profiles for NVIDIA V100 16GB and 32GB, H100 80GB SXM5 and H200 141GB systems, plus several NVIDIA DGX and HGX platforms and DigitalOcean droplets. Running the full benchmark takes roughly 90 minutes after setup. It requires Ubuntu 22.04, sudo access, internet access, Hugging Face approval for the Llama models and a spare drive of at least 500GB. The setup warns before formatting that drive.
Tensor Machines reports that identical GPU models differed by nearly 15% in token throughput on one tested workload, while another comparison found 36% more power produced 21% more FP16 compute throughput. Those are company-run early findings, not an independent leaderboard, and the publisher says they are specific to the tested workloads and operating conditions.
The limitations are substantial but explicit. The cost score currently represents only Llama 3 8B INT8, not the suite’s other models or training tests. Non-GPU node power is estimated from platform design power, and default cost assumptions may not match a particular operator’s contracts or location. Unsupported hardware requires a manually completed profile. TensorBench is therefore best read as a reproducible framework for local comparisons, not a universal GPU ranking.
Sources: TensorBench repository and methodology · Tensor Machines launch announcement · Unsplash image license
Conversation
0 approved commentsNo comments yet. Start a thoughtful conversation.
Join the conversation.
Sign in with ChatGPT to leave a comment or rate this article.