LLM parameters are the weights and biases that a model learns during training, according to Google’s machine learning glossary. A large language model is, at a minimum, a language model having a very high number of parameters. That count is why one figure stands in for a model’s size.
The term covers the learned values inside the network, not the sampling controls a user sets at request time. Hyperparameters are excluded too. They are the values that you, or a hyperparameter tuning service, supply to the model rather than values it learns, and learning rate is one of them. The count that follows a model’s name compresses billions of learned values into one number.
Key Takeaways
- GPT-3 is an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, per Brown and colleagues.
- Chinchilla uses the same compute budget as Gopher but with 70 billion parameters and 4x more data, and it uniformly and significantly outperforms Gopher at 280 billion parameters, according to Hoffmann and colleagues.
- The compute-optimal result came from training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens.
- In Mixtral, each token has access to 47 billion parameters but only uses 13 billion active parameters during inference, per Jiang and colleagues.
- DeepSeek-V3 carries 671 billion total parameters with 37 billion activated for each token, per DeepSeek-AI.
- Meta released the Llama 3.1 collection on July 23, 2024, at three sizes: 8 billion, 70 billion, and 405 billion parameters, all sharing one optimized transformer architecture, according to Meta’s model card.
How Do LLM Parameters Work?
Parameters are the weights and biases that a model learns during training. Four stages carry them from untrained values to the stored array a released model ships with. The sequence is the same across large language model architectures.
1. The network starts with untrained values
Before training, the weights and biases carry no information about language. A mixing desk makes the shape concrete: billions of faders, none of them yet set to anything useful.
2. Training adjusts each value by gradient descent
A parameter update is the operation of adjusting a model’s parameters during training, typically within a single iteration of gradient descent. Repeat that operation across a training run, and every parameter moves.
Hyperparameters sit outside that loop. They are the values that you, or a hyperparameter tuning service, supply to the model, and learning rate is the standard example. An oven holds the difference: a hyperparameter is the temperature the cook chooses, and a parameter is what the dough becomes.
3. The finished values are frozen into the released model
Llama 3.1 shipped as a static model trained on an offline dataset, with a knowledge cutoff of December 2023, per Meta’s model card. Nothing in the stored array moves when a user sends a prompt. The faders are locked before release.
4. Inference loads the stored values
A large language model is, at a minimum, a language model having a very high number of parameters. The size of that frozen array is what any deployment has to accommodate, which is where the published count starts doing real work.
Why the published count is a weak proxy
The number that follows a model’s name compresses all four stages into one figure. Three documented mechanisms sever the link between that figure and what a model can do.
Compute-optimal training breaks it at the training-budget level. Hoffmann and colleagues found that model size and the number of training tokens should be scaled equally. They tested it with Chinchilla, which uses the same compute budget as Gopher but with 70 billion parameters and 4x more data.
Chinchilla uniformly and significantly outperforms Gopher at 280 billion parameters, GPT-3 at 175 billion, Jurassic-1 at 178 billion, and Megatron-Turing NLG at 530 billion. Four larger models lost to a smaller one trained on more data.
Sparse activation breaks it at inference time. In Mixtral, a router network selects two experts at each layer. Each token has access to 47 billion parameters but only uses 13 billion active parameters during inference. DeepSeek-V3 widens the same split to 671 billion total parameters with 37 billion activated for each token.
Precision breaks it at the storage layer. Quantization lowers the memory requirements of loading and using a model by storing the weights in a lower precision while trying to preserve as much accuracy as possible, according to Hugging Face’s Transformers documentation. An identical count therefore occupies very different memory depending on how it ships.
| Property | Parameter | Hyperparameter |
|---|---|---|
| Where the value comes from | Learned by the model during training | Supplied to the model by you or a hyperparameter tuning service |
| Worked example | The bias and all the weights in a linear regression model | Learning rate |
| How it changes | Adjusted during training, typically within a single iteration of gradient descent | Set before the training run rather than learned |
Source: Google Machine Learning Glossary
Reading a count correctly starts with separating it from the other large numbers labs publish alongside it.
Parameters vs Tokens vs Context Window
Three figures travel together in every model announcement and measure different things. The Llama 3.1 model card publishes parameter sizes of 8 billion, 70 billion, and 405 billion alongside a training data token count of more than 15 trillion and a context length of 128,000 tokens. Parameters are learned and then frozen, training tokens are consumed once during the training run, and context length caps what a running model can read at a time.
| Figure | What it measures | When it is fixed |
|---|---|---|
| Parameters | The weights and biases the model learned | Frozen when the training run ends |
| Training tokens | The volume of text consumed during training | Fixed once training completes |
| Context length | How much text the model can take as input at once | Applies at run time, on every request |
Sources: Google Machine Learning Glossary, Meta Llama 3.1 Model Card
What is the difference between parameters and tokens?
Parameters are values the model owns, and tokens are the text it processes. Compute-optimal training ties the two together: model size and the number of training tokens should be scaled equally, so every doubling of model size should double the number of training tokens. DeepSeek-V3 was pre-trained on 14.8 trillion tokens.
Why Does Parameter Count Matter?
Three jobs sit behind the figure:
- It tracks capacity, the complexity of problems a model can learn.
- It anchors hardware planning, because every stored weight has to be loaded.
- It is now a legal disclosure item in the European Union.
Capacity is the complexity of problems that a model can learn, and a model’s capacity typically increases with the number of model parameters, per Google’s machine learning glossary. That relationship is why the count became shorthand for capability in the first place. GPT-3 was trained with 175 billion parameters, 10x more than any previous non-sparse language model. That jump was the headline of its release.
Kaplan and colleagues put a shape on it: Loss scales as a power law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude, and larger models are significantly more sample-efficient. Both results describe loss rather than downstream skill.
The EU AI Act turns the figure into a disclosure item. Annex XI requires the technical documentation for a general-purpose AI model to include the architecture and number of parameters, alongside the intended tasks, the acceptable use policies, the date of release, the modality and format of inputs and outputs, and the licence.
Recital 98 adds a threshold reading. It treats models with at least a billion parameters and trained with a large amount of data using self-supervision at scale as displaying significant generality.
Our AI benchmark coverage shows capability rankings shifting roughly every six months while public perception lags well behind them, and the parameter count is what that lag attaches itself to. It survives as the headline because it is the one property every lab publishes in the same unit. The accelerators that hold those weights follow their own supply curve, tracked in AI chip shipment and revenue data.
How much memory does a large model need?
The honest answer depends on precision, not on the count alone. Weights are typically stored in full-precision (fp32) floating-point representations, while half-precision types such as fp16 or bf16 are increasingly popular given the large size of models today, per Hugging Face’s Transformers documentation. One count maps to several different memory bills.
Total Parameters, Active Parameters, and Stored Size
Sparse architectures quote two counts, and the gap between them runs to an order of magnitude on current releases.
Mixtral is a Sparse Mixture of Experts language model in which each layer is composed of 8 feedforward blocks, the experts, and a router network selects two of them to process each token. Each token only sees two experts, so it has access to 47 billion parameters but only uses 13 billion active parameters during inference.
DeepSeek-V3 publishes 671 billion total parameters with 37 billion activated for each token. That is the wider of the two splits, from the model tracked in DeepSeek adoption and usage data.
Stored size adds another axis. Some quantization methods can reduce the precision of the weights as far as integer representations like int8 or int4. The count is unchanged in every one of those cases; only the bytes behind it move.
| Count type | What it includes | Example model | Parameters (billions) |
|---|---|---|---|
| Total parameters | Every value stored in the released model | DeepSeek-V3 | 671 |
| Activated per token | The subset that runs for a given token | DeepSeek-V3 | 37 |
| Accessible per token | The values a token can reach across the layers | Mixtral | 47 |
| Active during inference | The values a token actually uses | Mixtral | 13 |
| Published family sizes | The sizes one architecture ships at | Llama 3.1 | 8 / 70 / 405 |
Sources: DeepSeek-V3 Technical Report, Mixtral of Experts, Meta Llama 3.1 Model Card
What are active parameters?
Active parameters are the subset that runs for a given token, rather than the subset stored. In Mixtral, the selected experts can be different at each timestep, so what the model holds and what it spends per token are separate figures. A headline total describes what has to be resident; the activated count describes what has to be computed.
Pros, Cons, and Risks of Ranking Models by Parameter Count
Advantages
- Consistent publication. Annex XI of the EU AI Act requires the architecture and number of parameters in the technical documentation for a general-purpose AI model.
- Memory planning. Quantization lowers the memory requirements of loading and using a model by storing the weights in a lower precision.
- Correlation with capacity. A model’s capacity, the complexity of problems it can learn, typically increases with the number of model parameters.
- A measured relationship. Loss scales as a power law with model size, dataset size, and the amount of compute used for training, across more than seven orders of magnitude.
Trade-offs and Risks
- Undertrained models. Hoffmann and colleagues found current large language models significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant.
- Overstated compute. In Mixtral, each token has access to 47 billion parameters but only uses 13 billion active parameters during inference.
- Precision-blind estimates. Weights are typically stored in full precision, while half-precision types are increasingly popular given the large size of models today.
- Rankings that are not like for like. Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
Chinchilla’s parameter count is 0.25 of Gopher’s. Reliability sits on a different axis again, measured in LLM hallucination rate data.
| Reading the count as… | Why it misleads | What to check instead |
|---|---|---|
| A capability score | A smaller model trained on more data beat larger ones under the same compute budget | The training token count and the compute budget |
| Inference cost | Sparse models activate only part of the total for each token | The activated-per-token figure |
| A fixed memory bill | The same weights can be stored at a lower precision | The precision the weights ship in |
| A like-for-like ranking | Dense and sparse designs count parameters on different bases | The architecture first, then the active count |
Sources: Training Compute-Optimal Large Language Models, Mixtral of Experts, Hugging Face Transformers Documentation
Those cautions are easier to judge against counts that were actually published.
Real-World Applications of Parameter Counts
The figure does concrete work in three places: model releases, regulatory filings, and research budgeting.
Model families published at several sizes
Meta released the Llama 3.1 collection on July 23, 2024, as pretrained and instruction-tuned generative models at 8 billion, 70 billion, and 405 billion, all built on one optimized transformer architecture. Holding the architecture fixed and moving only the count is what makes a within-family comparison mean anything.
Sizes move with every release cycle, which is why current figures belong in a running model release tracker rather than in a definition.
Regulatory disclosure
Annex XI of the EU AI Act lists the architecture and number of parameters among the elements a general-purpose AI model provider must document. A number that began as engineering shorthand now has a compliance surface attached to it.
Research budgeting
Hoffmann and colleagues trained over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens to locate the compute-optimal point, in work submitted in March 2022. Sweeping the size axis, rather than picking one size and defending it, produced the correction.
Scenario: Reading a model card before deployment
- Start with the headline total. DeepSeek-V3 publishes 671 billion total parameters.
- Look for a second count on the same page. The same report lists 37 billion activated for each token.
- Check the precision the weights ship in. Quantization lowers the memory requirements of loading and using a model by storing the weights in a lower precision.
- Read the context length as a separate figure. The Llama 3.1 card lists a context length of 128,000 tokens next to its parameter sizes.
All of which leads back to the question every model announcement provokes.
Do More Parameters Always Mean a Better Model?
Not reliably. Under a fixed compute budget, model size and the number of training tokens should be scaled equally, and Chinchilla at 70 billion parameters uniformly and significantly outperformed Gopher at 280 billion on a large range of downstream evaluation tasks. Sparse designs complicate it further, since in Mixtral each token only uses 13 billion active parameters of the 47 billion it can access.
The scaling relationship that does hold describes loss, which scales as a power law with model size, dataset size, and the amount of compute used for training. A larger count generally correlates with more capacity and helps predict a hardware bill. On the published evidence, it does not determine which model answers a given question better.
Conclusion
Parameters are the weights and biases that a model learns during training. A model’s capacity typically increases with the number of model parameters. That relationship stops working as a ranking once the training budget varies. Chinchilla, at 70 billion parameters and 4x more data, uniformly and significantly outperformed Gopher at 280 billion.
The practical reading is that one figure now describes storage more reliably than skill. A model card publishing a total, an activated count, a token budget, and a precision answers four different questions. The headline count answers only the narrowest of them.
