Lemonade dropped a ROCm back-end that ran about 40x slower than Vulkan on the same chip. The lesson applies to any local model setup: measure, don’t assume. Here’s a generic checklist.
Checklist
- Test more than one back-end. Run the same model and prompt on Vulkan, ROCm or CPU and compare tokens per second.
- Watch for silent CPU fallback. If the GPU sits idle while CPU usage spikes, the “GPU” back-end may not be on the GPU at all.
- Know your memory pool. On APUs, the usable memory can be the shared GTT pool, not just the VRAM carve-out in firmware.
- Keep your tools current. Back-end fixes land fast, so read release notes before updating.
- Write down your results. Note model, quant, back-end and speed so you can spot regressions later.
God Bless, and May the Source Be with You!
