Installing Every NixOS Package: A Journey Through the Depths of Nixpkgs
Share this article
Four years ago, a developer installed every Arch Linux package (with some clever exclusions for conflicts). Now, with the power of NixOS's purely functional package management, they've embarked on an even more ambitious experiment: installing every package in Nixpkgs. The journey reveals not only the sheer scale of the Nix ecosystem but also the remarkable engineering that makes such an undertaking possible.
The experiment begins with a simple question: how many packages are there in Nixpkgs, and can they all coexist in a single system? With search.nixos.org boasting over 120,000 packages—nearly ten times the 12,232 packages in the previous Arch experiment—the challenge is significant. Unlike Arch, NixOS allows conflicting packages or multiple versions of the same package to coexist in the Nix store, activated into the environment via symlinks.
The First Attempt: A Naive Approach
The initial approach is straightforward: include every package in the pkgs attribute set in the system configuration.
environment.systemPackages = builtins.attrValues pkgs;
However, this approach immediately encounters the "horrors that scuttle in the nooks and crannies of pkgs." The author encounters evaluation errors like pkgs.AAAAAASomeThingsFailToEvaluate and complex stack traces from packages like pkgs.coqPackages_8_11.hierarchy-builder. Lazy evaluation, normally a Nix strength, reveals the hidden complexity of the package set.
The Second Attempt: Top-Level Packages Only
Realizing the full package set is too complex, the developer simplifies the goal to installing only top-level packages (like pkgs.fish but not pkgs.kdePackages.dolphin). They filter out packages marked as broken, insecure, or proprietary:
environment.systemPackages = builtins.attrValues pkgs
|> builtins.map (
x:
if
(builtins.tryEval x).success
&& builtins.isAttrs x
&& x ? "drvPath"
&& (builtins.tryEval x.value.drvPath).success
then
x
else
[ ]
)
|> lib.lists.flatten;
This approach fails with build errors for packages like liquidfun, which aren't marked as broken but still fail to build due to missing dependencies. The solution? Leverage Hydra, Nix's CI system, which tracks which packages fail to build for each update to nixos-unstable.
The Third Attempt: Recursive Discovery
With Hydra data, the developer creates a complex Nix expression to recursively traverse the entire pkgs attribute set. This "monstrosity" of a function handles various edge cases and avoids infinite recursion:
let
pkgs = import <nixpkgs> { };
lib = pkgs.lib;
getpkgs =
y: a: b:
builtins.map (
x:
if
(builtins.tryEval x.value).success
&& builtins.isAttrs x.value
&& !lib.strings.hasPrefix "pkgs" x.name # Ignore pkgs.pkgs*
&& !lib.strings.hasPrefix "__" x.name # Bad stuff
&& x.name != y.name # Definitely infinite recursion
&& x.name != "lib" # Ignore pkgs.lib, pkgs.agdaPackages.lib, and other evil stuff
&& x.name != "override" # Doesn't contain packages
&& x.name != "buildPackages" # Another copy of pkgs
&& x.name != "targetPackages" # Yet another copy of pkgs
&& x.name != "formats" # Ignore the pkgs.formats library
&& x.name != "tests" # Ignore tests
&& x.name != "nixosTests" # Ignore more tests
&& x.name != "scope" # Ignore pkgs.haskell.packages.ghc910.buildHaskellPackages.generateOptparseApplicativeCompletions.scope which contains another copy of pkgs
&& x.name != "_cuda" # Proprietary garbage
&& x.name != "vmTools" # Not a VM
&& x.name != "ghc902Binary" # Broken
&& x.name != "coqPackages_8_11" # Broken
&& x.name != "coqPackages_8_12" # Broken
&& x.name != "pypyPackages" # Broken
&& x.name != "pypy2Packages" # Broken
&& x.name != "pypy27Packages" # Broken
then
(
if x.value ? "drvPath" then
(
if (builtins.tryEval x.value.drvPath).success then
b
+ "|"
+ x.name
+ " "
+ (if x.value ? "pname" then x.value.pname else "unnamed")
+ " "
+ x.value.outPath
else
[ ]
)
else if a > 10 then
abort "Probably infinite loop?"
else
builtins.trace a
<| builtins.trace x.name
<| builtins.trace b
<| getpkgs x (a + 1)
# For some stupid reason x.name can contain . so use | as the separator instead
<| b + "|" + x.name
)
else
[ ]
)
<| lib.attrsToList y.value;
in
lib.strings.concatStringsSep "
"
<| lib.lists.flatten
<| getpkgs {
name = "pkgs";
value = pkgs;
} 0 "pkgs"
This script takes 10 minutes and 48 GB of RAM to generate a 623,946-line file with every Nix package. After deduplication, the count drops to 281,260 unique packages.
Filtering for the Cache
Rather than attempting to build every package (which would take months), the developer shifts focus to installing only packages available in the official cache.nixos.org binary cache. They create an Idris program to query which packages are available:
import Curl
open Curl
def N := 1024
def main : IO Unit := do
try
let mut lines := (← IO.FS.readFile "dedup").splitOn "
"
let h ← IO.FS.Handle.mk "cached" IO.FS.Mode.write
while 0 < lines.length do
IO.println s!"{lines.length} lines remaining"
let batch := lines.take N
lines := lines.drop N
let curlM ← curl_multi_init
let resps ← batch.filterMapM fun line ↦ do
match line.splitOn " " with
| [_, _, store] =>
let hash := store.stripPrefix "/nix/store/" |>.takeWhile (· ≠ '-')
let resp ← IO.mkRef { : IO.FS.Stream.Buffer}
let curl ← curl_easy_init
curl_set_option curl <| CurlOption.URL s!"https://cache.nixos.org/{hash}.narinfo"
curl_set_option curl <| CurlOption.NOBODY 1
curl_set_option curl <| CurlOption.HEADERDATA resp
curl_set_option curl <| CurlOption.HEADERFUNCTION writeBytes
curl_multi_add_handle curlM curl
return some (resp, line)
| _ => return none
while 0 < (← curl_multi_perform curlM) do
curl_multi_poll curlM 100
let good_lines ← resps.filterMapM fun (resp, line) ↦ do
let bytes := (← resp.get).data
return if _ : bytes.size > 9 then
-- Check if bytes starts with "HTTP/2 200"
if bytes[7] = 50 ∧ bytes[8] = 48 ∧ bytes[9] = 48 then
some line
else
none
else
none
-- Don't print out extra newline if empty
if 0 < good_lines.length then
h.putStrLn <| "
".intercalate good_lines
h.flush
catch e => IO.println s!"error: {e}"
After running this program (which makes 281,260 network requests), the list is reduced to 71,480 packages available in the binary cache.
The Build Environment Challenge
Even with packages pre-built, the system configuration faces issues with pkgs.buildEnv, which merges all packages into a single environment. The solution involves modifying the system-path.nix module to ignore single file outputs and manually excluding packages that cause directory conflicts:
match pname with
| "hypothesis" -- Cannot build '/nix/store/4k1n7lbmkihdf1jisahqrjl3k4cbmhgf-hypothesis-6.136.9-doc.drv'
| "gci" -- pkgs.buildEnv error: not a directory: `/nix/store/dv8q5wfm717q4qqk6pps6dyiysqqf22c-gci-0.13.7/bin/internal'
| "openjdk-minimal-jre" -- pkgs.buildEnv error: not a directory: `/nix/store/zmcc5baw10kbhkh86b95rhdq467s9cy1-openjdk-minimal-jre-11.0.29+7/lib/modules'
| "olvid" -- pkgs.buildEnv error: not a directory: `/nix/store/xlmxn65dw2nvxbpgvfkb4nivnk7gd2n2-olvid-2.5.1/lib/modules'
| "resources" -- pkgs.buildEnv error: not a directory: `/nix/store/36ywzl3hqg57ha341af8wpvgbwga90b8-resources-1.9.0/bin/resources'
| "temurin-bin" -- pkgs.buildEnv error: not a directory: `/nix/store/6v1bb758sn4fh2i27msxni2sy07ypqrr-temurin-bin-11.0.28/lib/modules'
| "temurin-jre-bin" -- pkgs.buildEnv error: not a directory: `/nix/store/kj0hj48hm4psjrpsa2lsd5lhnax6v9p6-temurin-jre-bin-21.0.8/lib/modules'
| "semeru-bin" -- pkgs.buildEnv error: not a directory: `/nix/store/chn2lx7gnvd3ay5x8gmnn774gw1yafv0-semeru-bin-21.0.3/lib/modules'
| "semeru-jre-bin" -- pkgs.buildEnv error: not a directory: `/nix/store/hbazadpm1x0a2nkg686796387d14539r-semeru-jre-bin-21.0.3/lib/modules'
| "zulu-ca-jdk" -- pkgs.buildEnv error: not a directory: `/nix/store/z16qj0jgm9vffy8m6533rxjzw904f7c1-zulu-ca-jdk-21.0.8/lib/modules'
| "graalvm-ce" -- pkgs.buildEnv error: not a directory: `/nix/store/1ywfss0i4rpdiyzydvqd43ahsjvq2jk6-graalvm-ce-25.0.0/lib/modules'
| "pax" -- pkgs.buildEnv error: not a directory: `/nix/store/n0w0kkr796jyiq16kff4cq4vfrnb9n9i-pax-20240817/bin/pax
| "discord-haskell" -- pkgs.buildEnv error: not a directory: `/nix/store/hrx6j0ld4dly5g75dalipa7s36pjndk9-discord-haskell-1.18.0/bin/cache
=> none
| _ =>
if path.endsWith "bsd|source" then
-- Either pkgs|openbsd|source or pkgs|netbsd|source which both clobber shadow's /sbin/nologin and excluding shadow may break the system
none
else
match path.splitOn "|" with
| .nil => none
| part :: parts =>
-- Quote each part because it might contain weird characters
some <| part ++ ".\"" ++ "\".".intercalate parts ++ "\""
The Moment of Truth
After several iterations of fixing build environment issues, the final nixos-rebuild switch encounters an unexpected error:
env: 'php': No such file or directory
error: your NixOS configuration path seems to be missing essential files.
To avoid corrupting your current NixOS installation, the activation will abort.
Even NixOS itself is terrified of this configuration! The developer proceeds with NIXOS_REBUILD_I_UNDERSTAND_THE_CONSEQUENCES_PLEASE_BREAK_MY_SYSTEM, which changes their shell from Fish to a Go Fish game, making the system unloginable. Fortunately, NixOS allows rolling back to a previous generation, and after switching the shell to Bash, the system finally builds successfully.
The Results
The final system boasts 69,700 packages—five times more than the previous Arch experiment. The Nix store is 1.34 TB with 744,954 paths, but thanks to btrfs compression, it only occupies 671 GB on disk. Surprisingly, the system boots quickly and functions normally, with only minor oddities like ls using the 9base implementation rather than coreutils.
root@nixos
----------
OS: NixOS 26.05 (Yarara) x86_64
Host: MS-7C94 (1.0)
Kernel: Linux 6.17.9-300.fc43.x86_64
Uptime: 2 days, 23 hours, 22 mins
Packages: 69700 (nix-system)
Shell: fish 4.2.1
Display (DELL U2515H): 2560x1440 in 25", 60 Hz [External]
Terminal: xterm-256color
CPU: AMD Ryzen 9 3950X (32) @ 4.76 GHz
GPU: AMD Radeon RX 7900 XTX [Discrete]
Memory: 5.81 GiB / 62.70 GiB (9%)
Swap: 1.43 GiB / 8.00 GiB (18%)
Disk (/): 644.04 GiB / 1.82 TiB (35%) - btrfs
Local IP (enp42s0): 10.187.0.152/21
Locale: en_US.UTF-8
This experiment demonstrates not only the remarkable scale of the Nix ecosystem but also the robustness of its package management system. The ability to install and manage such a vast collection of packages without system failure highlights the unique advantages of Nix's purely functional approach to package management.
As the author notes, "Honestly, the total size of Nixpkgs seems smaller than I expected, and the total network usage of this experiment was about the same as a few people using NixOS for a year." This speaks to the efficiency of Nix's binary caching and deduplication mechanisms.
The journey through installing every NixOS package reveals the hidden complexity and beauty of the Nix ecosystem—a testament to the power of functional programming principles applied to system configuration management.