Upbringing

The question

What changes when a base language model is taught to follow instructions, learn preferences, or optimize a programmatic reward? Upbringing makes those changes inspectable across checkpoints, including the cases where a better score hides a narrower improvement.

Three experiments, three kinds of feedback

The first experiment runs a reduced-scale version of Ai2’s OLMo 2 post-training recipe: supervised fine-tuning followed by direct preference optimization. The second uses LoRA DPO and simulated labelers with known preferences, so the learned behavior can be compared with the rule behind the labels. The third trains Qwen2.5-3B on Countdown arithmetic with GRPO and a programmatic verifier.

These are separate training experiments, not one model moving through all three stages. Checkpoint sweeps, reference releases, fixed prompts and saved rollouts connect training decisions to observed behavior.

Training changed behavior—and exposed tradeoffs

On OLMo 2 1B, instruction-following accuracy rose while the preference-training step reduced arithmetic performance relative to the SFT checkpoint. Evaluating Ai2’s released models provided a check on the local harness; it did not establish a reproduction of Ai2’s training results.

Reward improved reliability within a limited task set

For Countdown, the base model scored 1.3% per sampled attempt; the final checkpoint scored 55.7%. The final puzzle-bootstrap 95% interval was 49.7–61.7%. Every evaluated checkpoint received the same 256 puzzles and 32 samples per puzzle, with a separate greedy evaluation.

The important shape is below the headline: pass@32 reached 55.1% at step 50 and finished at 56.2%, while per-attempt accuracy kept improving. Within this experiment, training increasingly concentrated probability on solutions it could already find early. Thirty-two samples per puzzle cannot settle whether the model acquired genuinely new capabilities.

The grader was part of the experiment

An early run kept generating beyond its answer. The model often continued the conversation with invented questions and additional answers; the grader read the last answer block. Re-scoring the one retained batch found different extracted answers in 166 of 256 rollouts. Both graders awarded one correct answer, but to different rollouts.

The repair combined a generation stop at the closing answer tag with first-answer extraction and regression checks. The evidence establishes a grading vulnerability with a measured reward effect in one stopped run. It does not demonstrate that training taught the model to exploit it.

Inspect the work

The public repository includes training and evaluation entry points, correctness tests, the retained grader-bug batch and result tables. Final SFT, DPO and GRPO checkpoints are published on Hugging Face. The interactive project lets visitors compare responses and preference-learning traces; visitors’ own judgments are fit locally rather than used for live model training.

Models and training mixtures come from Ai2 and Qwen; Countdown data and prompt conventions build on TinyZero. The contribution is the implemented experiments, evaluation and analysis, not the base models or training algorithms.