🧠 MiniAxion1.5-3M
Emergent reasoning in a 2.7M parameter model. A tiny Portuguese-first language model that learns how to think before it learns how to be correct.
🚀 Overview
MiniAxion1.5-3M is an ultra-compact (~2.7M parameters) GPT-style language model designed to investigate reasoning emergence at extreme small scale.
Unlike typical small models optimized for fluency, MiniAxion is explicitly trained to produce:
Structured reasoning traces Step-by-step thinking () Deterministic answer formatting
It operates primarily in Portuguese, making it a rare example of a non-English reasoning-first nano model.
⚡ Why This Model Is Interesting
Most models follow this trajectory:
Language → Knowledge → Reasoning
MiniAxion flips part of that:
Structure → Reasoning format → (still learning correctness)
💡 Key insight:
The model demonstrates that reasoning structure can emerge independently of reasoning accuracy.
🧪 Evaluation Task Performance Task Accuracy Addition 10% Subtraction 10% Multiplication 0% Even/Odd 100% Comparison 5% Sequence Completion 0% Word Problems (Addition) 10% Word Problems (Subtraction) 0% Word Problems (Multiplication) 10% True/False 100% Chat/Greetings 100%
🧠 Reasoning Behavior Metrics Metric Score Thinking Rate 100% Step Format 100% Answer Completion 100%
✔ The model always thinks ✔ The model always structures reasoning ✔ The model always produces an answer
📊 Interpretation
MiniAxion exhibits a clear dissociation:
✅ What it learned Reasoning format Step-by-step decomposition Logical task patterns (parity, boolean) ❌ What it did NOT learn Arithmetic correctness Numerical reasoning Multi-step computation
🔬 Core Finding
Reasoning ≠ Correctness
MiniAxion shows that:
Models can internalize thinking patterns Without actually learning how to solve problems
This makes it a strong candidate for studying:
Emergent reasoning Tiny Recursive Models (TRMs) Reasoning distillation
🏗️ Architecture Type: GPT-style Transformer Parameters: ~2.7M Objective: Next-token prediction Language: Portuguese (primary) Specialization: Structured reasoning traces
🧠 Training Strategy
The model was trained with a reasoning-first approach:
Portuguese language grounding Structured reasoning data () Emphasis on: Deterministic formats Multi-step thinking Explicit reasoning tokens
🚫 No RLHF 🚫 No instruction tuning at scale 🚫 No large model distillation (yet)
⚠️ Limitations
- Arithmetic Collapse
Near-random performance in:
Addition
Subtraction
Multiplication
→ Indicates lack of numerical representation learning
Strong dependence on:
Prompt format
Token patterns
Seen reasoning templates
🔮 Future Work
This model is just the beginning.
📈 Scaling
5M / 10M / 20M versions
Track emergence of correctness
🧪 Distillation
Inject reasoning from larger models
Improve accuracy without scaling params
🔁 Self-Play / Synthetic Data
Generate reasoning loops
Reinforce correct chains
🧩 Hybrid Reasoning
Combine symbolic + neural learning
Fix arithmetic weakness
🧾 Example Output
Identifico os números Tento somar os valores Ajusto o resultado 74✔ Perfect reasoning structure ❌ Incorrect answer
💡 Takeaway
MiniAxion1.5-3M proves something important:
Even a 2.7M model can learn to simulate thinking before it learns to actually think correctly.
🤝 Use Cases
Research on emergent reasoning
Tiny model experimentation (CPU-friendly)
Educational demos of:
Chain-of-Thought
Reasoning failure modes
Base model for:
Distillation
NRM experiments
- Downloads last month
- -