№ 0458Reddit post★ Pick
Muse Glimmer 30B stretched to 512K context
Extended Muse Glimmer 30B from 128K to 512K context by changing one config value, then spent ~70h of DGX Spark compute testing it: 100% needle-in-a-haystack and multi-hop retrieval up to 512K.
mr_il found that Glimmer only applies RoPE on 2,048-token sliding-window layers while its full-attention layers have no position encoding, so raising max_position_embeddings is enough. Counting across the whole context degraded from 95% at 32K to 22% at 512K; LongBench v2 and LongCodeQA held steady. Scripts and reports are open on GitHub.
Muse Glimmer 30B with 512k context
My fun weekend project was to try to make the new Muse Glimmer 30B work with a longer context, deciding to go for 512k first. I had expected the usual YaRN shenanigans and maybe a LoRA. I couldn't have been wrong more. Upon closer look, Glimmer turned out to be rather unusual architecturally. The thing that make long-context adaptations painful in other models, full attention layers with token position encoding, it simply not there. Instead, only 2048 tokens-wide SWA layers have RoPE, and full GQA attention layers have no position encoding at all. It appears the model is trained to work with long-distance token relationships inferred from the context and SWA layers. It's a rather bold architecture bet, but it seems Meta managed to pull it off. As a result, the model architecture appears to be uniquely suited for context extension by simple mechanical means. To change model context length from stock 128k to, say, 512k, you need only to change “max_position_embeddings” config setting from 131072 to 524288. What confuses other models, like Qwen3.5 family, Glimmer just takes into its stride. I spent close to 70h of compute on DGX Spark to test stock model with extended context on a
ChatForm
Tgmlabs