A 35B Hybrid-Attention Mixture-of-Experts Model on a 6GB 2011 GPU: Hand-Written 4-bit CUDA Inference for Fermi

We report end-to-end inference of $\mathbf{\text{Qwen3.6-35B-A3B}}$ – a 35-billion-parameter, $\sim$3B-active Mixture-of-Experts (MoE) model with a hybrid gated-delta-net / full-attention backbone – on a $\mathbf{\text{2011}}\mathbf{\text{NVIDIA}}\mathbf{\text{Tesla}}\mathbf{\text{C2075}}$ (Fermi, compute capability \smtwenty, 6,GB), a GPU that predates tensor cores, native FP16 arithmetic, the $\mathtt{\text{DP4A}}$ integer dot-product instruction, and support in every modern CUDA toolchain. Because the 4-bit model ($\approx$10.5,GB) is roughly twice the device memory, we adopt a \emph{hybrid} execution strategy: the GPU performs batched prompt \emph{prefill} with expert weights streamed layer-by-layer from host RAM, while \emph{decode} runs on the host CPU using a hand-written W4A8 integer GEMV built on the SSSE3 $\mathtt{\text{pmaddubsw}}$ instruction. The entire engine – GEMM, hybrid-attention recurrence, MoE routing, and a from-scratch vision tower – is written by hand for \smtwenty{} and compiled with the legacy CUDA 8.0 toolchain. On a 947-token prompt we reduce prefill latency from 57.2,s to 37.5,s ($-34\mathrm{\%}$) through expert pinning, single-pass prefill, and NUMA interleaving, and we raise decode throughput from 2.8 to 8.6,\tps{} ($\approx 3\times$) with the integer-SIMD kernel. A position-indexed snapshot cache for the recurrent (gated-delta-net) state restores prefix reuse on a recurrent architecture, cutting a repeated 78,s prefill to 0.5,s. We also report a set of \emph{negative} results – offloading the language-model head to the idle GPU, hyper-threading, and three GPU-kernel rewrites all fail to help – % which together pin down the practical floor of this hardware. Our aim is not a speed record but a careful account of what it takes, and where the walls are, to run a contemporary frontier-class MoE on fourteen-year-old silicon.