Output
Index of public work
1OpenLanguageModel
- Kind
- Software library
- Release
- v2.2
- License
- MIT
- Language
- Python 3.10–3.12, PyTorch
- Documentation
- openlanguagemodel.github.io
A transparent PyTorch framework for building, training, and experimenting with language models, with the architecture left visible. Every component is an ordinary torch.nn.Module; data loading, mixed precision, single-node multi-GPU training, checkpointing, and logging are already wired into a clean path. It has carried more than thirty training runs at scales from 100M to 1B parameters.
1.1Architecture as blocks
The library separates what happens from how it is wired. Components such as attention, normalization, and feed-forward layers say what happens; Block, Residual, and Repeat say how the components are composed. An architectural experiment then becomes a local edit. Listing 1 is a complete Llama 3 block, written as it is described.
from olm.nn.structure import Blockfrom olm.nn.structure.combinators import Residualfrom olm.nn.attention import GroupedQueryAttentionfrom olm.nn.feedforward import SwiGLUFFNfrom olm.nn.norms import RMSNormllama3_block = Block([ Residual(Block([ RMSNorm(embed_dim, eps=1e-5), GroupedQueryAttention( embed_dim, num_heads, num_kv_heads, max_seq_len, dropout=dropout, rope_theta=rope_theta, use_bias=False, ), ])), Residual(Block([ RMSNorm(embed_dim, eps=1e-5), SwiGLUFFN( embed_dim, hidden_dim=intermediate_size, dropout=dropout, bias=False, ), ])),])1.2Reference architectures
The presets in olm.models are readable implementations of familiar model families, written in the same blocks (Table 1).
| Family | Presets | Number of presets |
|---|---|---|
| GPT-2 | GPT2 · GPT2Medium · GPT2Large · GPT2XL | 4 |
| Llama 2 | Llama2_7B · Llama2_13B · Llama2_70B | 3 |
| Llama 3.x | Llama3_1_8B · Llama3_1_70B · Llama3_1_405B · Llama3_2_1B · Llama3_2_3B | 5 |
| Qwen 2.5 | Qwen2_5_0_5B · Qwen2_5_1_5B · Qwen2_5_3B · Qwen2_5_7B · Qwen2_5_14B · Qwen2_5_32B · Qwen2_5_72B | 7 |
| Phi-3 | Phi3_5_Mini · Phi3_Small | 2 |
| Phi-4 | Phi4_14B | 1 |
| Gemma 2 | Gemma2_2B · Gemma2_9B · Gemma2_27B | 3 |
| OLMo | OLMo_7B | 1 |
| OPT | OPT125M | 1 |
1.3Releases
- v1.0Foundation: core architectures, streaming data, single-GPU training.
- v1.1Flash and SDPA attention, RoPE and ALiBi variants, experiment logging, API documentation.
- v2.0DDP and FSDP training; readable mixture-of-experts routing.
- v2.1AutoTrainer, hardware-aware training setup, distributed and attention stability fixes.
- v2.2Stability fixes, a new website, and a refined API reference.current
- v3.0Further training: SFT, LoRA, DPO, PPO and RLHF, GRPO-style RLVR, evaluation recipes.planned
- v4.0Multi-node training and cluster support.planned
1.4Citing
@software{openlanguagemodel,
title = {{OpenLanguageModel}: A Modular {PyTorch} Library for
Transformer Language Models},
author = {{OpenLanguageModel contributors}},
year = {2026},
version = {2.2},
license = {MIT},
url = {https://github.com/openlanguagemodel/openlanguagemodel}
}2Models and records
Trained models, datasets, and the records of our experiments are published on Hugging Face. Code and systems are published on GitHub.
