Free Energy Compression (FEC)

Unified quantization + pruning via variational free energy minimization

Overview

Free Energy Compression (FEC) is the first unified compression framework derived from variational free energy minimization. By interpreting quantization and pruning as joint entropy reduction under a fidelity constraint, FEC replaces heuristic loss weighting with a principled temperature schedule.

Results on Gemma-2b (2.5B params):

Config	Compression	PPL Δ	Size	Speed (CPU)
FEC-Small	3.2×	+0.44	1.5 GB	24 tok/s
FEC-Medium	4.4×	+0.87	1.1 GB	31 tok/s
FEC-Large	6.0×	+1.69	0.8 GB	42 tok/s

Outperforms GPTQ+Wanda (2.8×, +0.7 PPL) and AWQ+SparseGPT (2.9×, +0.6 PPL).

Quick Start

# Clone
git clone https://github.com/Kiri-Labs/free-energy-compression.git
cd free-energy-compression

# Run on Colab T4 (recommended)
# Upload src/fec_colab.py to Colab, Runtime > T4 GPU, run:
# !python fec_colab.py --config experiments/config/fec_small.yaml

# Or run locally (CPU, slow)
pip install torch transformers datasets accelerate pyyaml huggingface_hub
python src/fec_colab.py --config experiments/config/fec_small.yaml

Documentation

Document	Description
Paper	Full academic paper (7 chapters + BibTeX)
Methodology	Free energy derivation, temperature schedule, algorithm
Experiments	Setup, baselines, metrics, reproducibility
Results	Benchmarks, ablation studies, CPU latency
Config	FEC-Small configuration (3.5-bit, 30% sparsity)
Source	Core FEC implementation (Hessian-based, layer-wise)
Colab Runner	Autonomous Colab T4/CPU runner

Theory

Variational Free Energy: $$\mathcal{F}\tau = \underbrace{\mathbb{E}_q[-\log p(\mathcal{D}|\hat{W})]}{\text{Energy (reconstruction)}} - \tau \underbrace{\mathcal{H}[q]}_{\text{Entropy (compression)}}$$

Energy → Layer-wise reconstruction error weighted by Hessian
Entropy → Bits × (1 - sparsity) × parameters
Temperature τ → Annealed from τ₀ to 0 (replaces λ₁, λ₂ hand-tuning)

Reproduction

All experiments run via colab run with fixed seeds. Results in experiments/results/benchmarks_*.json.

# Colab T4 (2 hours)
colab run --gpu=T4 --code-file src/fec_colab.py --args="--config experiments/config/fec_small.yaml"

# CPU fallback (6-8 hours)
colab run --code-file src/fec_colab.py --args="--config experiments/config/fec_small.yaml"

Citation

@article{boa2026fec,
  title={Free Energy Compression: Unified Quantization and Pruning via Variational Free Energy Minimization},
  author={Boa, Agnes},
  journal={Kiri-Labs Technical Report},
  year={2026},
  url={https://github.com/Kiri-Labs/free-energy-compression}
}

License

MIT License - see LICENSE

GPTQ - Post-training quantization
AWQ - Activation-aware quantization
Wanda - Weight×activation pruning
SparseGPT - One-shot pruning
Friston 2010 - Free energy principle