Your #1 source of news on xAI's Grok

Structurally, it looks quite similar to the Llama 2 tokenizer (BPE w/ byte-fallback), with a vocabulary size of 2¹⁷ =…

Grok's declared MMLU score of 73% beats Llama 2 70B’s 68.9% and Mixtral 8x7B’s 70.6%. At 314 billion parameters, xAI’s Grok-1 is significantly larger than today’s leading open-source model.

@xai's Grok-1 is a Mixture-of-Experts…

https://huggingface.co/xai-org/grok-1

314B, mixture of expert (2 out of 8 active). Even the active parameters only (86B) is more than the biggest Llama. Can’t wait to see the benchmarking results and what people build with it.

It was their first model done on a brand new stack coded in JAX and Rust by a new and tiny team

Were any of your first models better?

StableLM alpha wasn’t great, now they are the best performing models of their size

https://github.com/xai-org/grok

Looking quickly:
- model nicely written
- partition rules for sharding follow the old style of t5x
- they used haiku but it wouldn't be too hard to update to flax
- they use shard_map on the MoE layers for…