VaultGemma 1B: Google’s Privacy-First 1B-Parameter LLM Trained with Differential Privacy

A milestone for privacy-preserving LLMs

Google AI Research and DeepMind announced VaultGemma 1B, the largest open-weight large language model trained from scratch with differential privacy (DP). This release demonstrates that rigorous privacy guarantees can be applied during full pretraining, not just during fine-tuning, and that such models can still be practically useful.

Why differential privacy matters for LLMs

Large language models trained on web-scale corpora can memorize and regurgitate sensitive or personally identifiable information. Differential privacy provides a mathematical bound that limits the influence of any single training example on the model’s outputs. By enforcing DP throughout pretraining, VaultGemma reduces the risk of extraction attacks and data leakage at the foundation of the model.

Architecture and training data

VaultGemma mirrors the architectural design of earlier Gemma family models with optimizations for private training:

• Size: 1 billion parameters, 26 layers
• Transformer: decoder-only
• Activations: GeGLU with feedforward dimension 13,824
• Attention: Multi-Query Attention (MQA) and global span of 1,024 tokens
• Normalization: RMSNorm in pre-norm configuration
• Tokenizer: SentencePiece with a 256K vocabulary

To reduce compute under DP constraints, the sequence length was reduced to 1,024 tokens, enabling larger effective batch sizes.

VaultGemma was trained on the same 13 trillion-token corpus used for Gemma 2, composed mainly of English web documents, code, and scientific text. The dataset underwent filtering to remove unsafe content, reduce personal information exposure, and prevent contamination with evaluation data.

How differential privacy was applied

The team used DP-SGD with per-example gradient clipping and Gaussian noise addition, implemented on top of JAX Privacy with several scaling optimizations:

• Vectorized per-example clipping for parallel efficiency
• Gradient accumulation to simulate large DP batches
• Truncated Poisson Subsampling for efficient on-the-fly sampling

The model achieved a formal DP guarantee of approximately (ε ≤ 2.0, δ ≤ 1.1e−10) at the sequence level (1,024 tokens), providing a measurable privacy bound for training.

New scaling laws for private training

Training under DP changes the tradeoffs between model size, batch noise, and compute. The VaultGemma team developed DP-specific scaling laws that include:

• Quadratic fits for optimal learning rate modeling across runs
• Parametric extrapolation of loss values to reduce dependence on intermediate checkpoints
• Semi-parametric fits to generalize across model size, step counts, and noise-to-batch ratios

These methods allowed accurate loss prediction and efficient resource allocation on TPUv6e clusters.

Training setup and results

VaultGemma was trained on 2,048 TPUv6e chips with GSPMD partitioning and MegaScale XLA. Key training specs:

• Batch size: ~518K tokens
• Training iterations: 100,000
• Noise multiplier: 0.614

The achieved loss was within 1% of the DP scaling law predictions, validating the approach.

Performance and tradeoffs

On academic benchmarks VaultGemma trails non-private counterparts but demonstrates meaningful utility:

• ARC-C: 26.45 vs 38.31 (Gemma-3 1B)
• PIQA: 68.0 vs 70.51 (GPT-2 1.5B)
• TriviaQA (5-shot): 11.24 vs 39.75 (Gemma-3 1B)

These results indicate that current DP-trained models are comparable to non-private models from several years ago. Crucially, memorization tests showed no detectable training-data leakage in VaultGemma, unlike in non-private Gemma releases.

What this means for the community

VaultGemma 1B proves that end-to-end private pretraining at scale is feasible. While a utility gap remains relative to non-private models, the open release of the model and technical details gives researchers and practitioners a foundation for building more capable private models and refining DP training strategies.

For the full technical report and resources see: https://services.google.com/fh/files/blogs/vaultgemma_tech_report.pdf

You can also find model artifacts on Hugging Face and code, notebooks, and tutorials on the authors’ GitHub pages.