FAIRC

Output

FAIRC · Output2026

Output

Index of public work

1OpenLanguageModel

Kind
Software library
Release
v2.2
License
MIT
Language
Python 3.10–3.12, PyTorch

A transparent PyTorch framework for building, training, and experimenting with language models, with the architecture left visible. Every component is an ordinary torch.nn.Module; data loading, mixed precision, single-node multi-GPU training, checkpointing, and logging are already wired into a clean path. It has carried more than thirty training runs at scales from 100M to 1B parameters.

1.1Architecture as blocks

The library separates what happens from how it is wired. Components such as attention, normalization, and feed-forward layers say what happens; Block, Residual, and Repeat say how the components are composed. An architectural experiment then becomes a local edit. Listing 1 is a complete Llama 3 block, written as it is described.

Listing 1. A Llama 3 transformer block in OpenLanguageModel: RMSNorm, grouped-query attention, and a SwiGLU feed-forward layer, each inside an explicit residual.
from olm.nn.structure import Blockfrom olm.nn.structure.combinators import Residualfrom olm.nn.attention import GroupedQueryAttentionfrom olm.nn.feedforward import SwiGLUFFNfrom olm.nn.norms import RMSNormllama3_block = Block([    Residual(Block([        RMSNorm(embed_dim, eps=1e-5),        GroupedQueryAttention(            embed_dim,            num_heads,            num_kv_heads,            max_seq_len,            dropout=dropout,            rope_theta=rope_theta,            use_bias=False,        ),    ])),    Residual(Block([        RMSNorm(embed_dim, eps=1e-5),        SwiGLUFFN(            embed_dim,            hidden_dim=intermediate_size,            dropout=dropout,            bias=False,        ),    ])),])

1.2Reference architectures

The presets in olm.models are readable implementations of familiar model families, written in the same blocks (Table 1).

Table 1. Model families implemented in olm.models: 9 families, 27 named presets.
FamilyPresetsNumber of presets
GPT-2GPT2 · GPT2Medium · GPT2Large · GPT2XL4
Llama 2Llama2_7B · Llama2_13B · Llama2_70B3
Llama 3.xLlama3_1_8B · Llama3_1_70B · Llama3_1_405B · Llama3_2_1B · Llama3_2_3B5
Qwen 2.5Qwen2_5_0_5B · Qwen2_5_1_5B · Qwen2_5_3B · Qwen2_5_7B · Qwen2_5_14B · Qwen2_5_32B · Qwen2_5_72B7
Phi-3Phi3_5_Mini · Phi3_Small2
Phi-4Phi4_14B1
Gemma 2Gemma2_2B · Gemma2_9B · Gemma2_27B3
OLMoOLMo_7B1
OPTOPT125M1

1.3Releases

  1. v1.0Foundation: core architectures, streaming data, single-GPU training.
  2. v1.1Flash and SDPA attention, RoPE and ALiBi variants, experiment logging, API documentation.
  3. v2.0DDP and FSDP training; readable mixture-of-experts routing.
  4. v2.1AutoTrainer, hardware-aware training setup, distributed and attention stability fixes.
  5. v2.2Stability fixes, a new website, and a refined API reference.current
  6. v3.0Further training: SFT, LoRA, DPO, PPO and RLHF, GRPO-style RLVR, evaluation recipes.planned
  7. v4.0Multi-node training and cluster support.planned

1.4Citing

@software{openlanguagemodel,
  title   = {{OpenLanguageModel}: A Modular {PyTorch} Library for
             Transformer Language Models},
  author  = {{OpenLanguageModel contributors}},
  year    = {2026},
  version = {2.2},
  license = {MIT},
  url     = {https://github.com/openlanguagemodel/openlanguagemodel}
}
BibTeX

2Models and records

Trained models, datasets, and the records of our experiments are published on Hugging Face. Code and systems are published on GitHub.

huggingface.co/FAIRC github.com/FAIRC-Research