Distcc - Crushing Compilation Times
Table of Contents
Distcc - Crushing Compilation Times⌗
Ever tried to compile something big on a slow PC like a Raspberry Pi? If you have, you know the struggle.
You’ve thought about cross-compiling, but it’s a chore.
DistCC is the answer – let powerful machines handle the heavy lifting while you stay productive on your Pi.
No more hassle, just faster compilation times.
Ok, I lied a bit. It’s not without hassle, but once you set this up, you don’t need to manage it anymore until you upgrade the toolchain But I found an easy way to easily have the same toolchains on all machines so keeping everything updated is really easy.
Few terms we need to clarify⌗
- Host
- This is the system that you run the compilation on and that is the target of the compilation
- Slave
- All the machines that the Host connects to and sends all the work to
Setting up the Host⌗
Because I’ve been setting up this for my Raspberry Pi 3, and most servers/machines run it (or something based on it), I’ll assume you’re running Debian.
But everything I’ll be describing here can easily be used on other distros. Just look it up on your distro’s wiki.
Installing the tools⌗
First of all we need to make sure that you have all the basic tools for compilation installed:
sudo apt install build-essential
Now we need to install distcc
:
sudo apt install distcc -y
Configuring DistCC⌗
The Host machine is really easy to configure.
All you need to do is to open /etc/distcc/hosts
and add the IP addresses of the Slave machines
(This will open the file in your $EDITOR
)
sudo -e /etc/distcc/hosts
Now remove (or comment out) the 127.0.0.1
line
(unless you want to compile on the Host as well - I’ll talk about this later)
and add the IP addresses of the other machines followed by /n
where n
is the number of threads you want to run on that machine.
The file should then look something like this:
192.168.10.221:3632/11
192.168.10.23:3632/4
The 3632
port is the default one. You don’t need to change it,
unless you want to run a more complicated setup.
It that’s your case fell free to leave a comment below and I’ll add to this post.
Slaves⌗
Now that our Host is done, we need to setup our Slaves. To make our jobs easier we’ll be using Podman and Distrobox to run our DistCC daemon.
If you use the same OS on all machines you don’t need to follow the container setup but I still advise it in case the OS versions are different.
(Eg. Raspberry Pi is now Debian 11 while Debian 12 is the latest one available as of writing this).
Setting up podman⌗
This one should be easy as podman
should already be in your
distribution’s repository:
sudo apt install podman -y
Setting up distrobox⌗
This one is a bit more tricky as distrobox may not be present in your repositories.
But the instalation of distrobox is pretty straightforward so it should take only a while.
curl -s https://raw.githubusercontent.com/89luca89/distrobox/main/install | sudo sh
Setting up the container⌗
The container setup is a bit more unconventional from the normal way you use Distrobox.
We need to have systemd
installed to start the distcc.service
and we need to have it running as root to have the DistCC port accessible.
To setup the container use the same OS and version as the Host
*(use cat /etc/os-release
if you’re not sure)
(Replace debian:bullseye
with a docker image for your OS and version
and change the name so that you know what is the container for)
distrobox create -i debian:bullseye --root --init --additional-packages "systemd libpam-systemd" -n debian11
Let it download the image if needed and enter the container: This may take a long time depending on your SSD/HDD and internet speeds.
distrobox enter --root debian11
If you want to start DistCC again, eg. after rebooting, you can run this command:
(change debian11
to your container’s name)
distrobox enter --root -n debian11 -- /usr/bin/sudo systemctl start distcc
Setting up the toolchain and DistCC⌗
(inside the Distrobox container)
Now we just need to install the same toolchain as the Host machine and setup distcc
.
What package to intall⌗
If you’re unsure about what packages to install look at your Host’s architecture:
uname -m
This command will output something like aarch64
and that is what you’re looking for.
Now we just need to install the correct GCC
and G++
on the Slaves:
(Replace aarch64
with the Host’s architecture)
sudo apt install build-essential distcc g++-aarch64-linux-gnu gcc-aarch64-linux-gnu
Now you should be able to run
aarch64-linux-gnu-gcc --version
and have the same version on the Host and the Slave.
Configuring DistCC⌗
This one is a bit more tricky but it’s still nothing hard:
sudo -e /etc/distcc/clients.allow
Here you need to put in your network adresses that you want to allow.
You can use specific adresses like 192.168.10.4
or you can just add 192.168.10.0/24
to allow all devices on 192.168.10.x
address the use of this machine’s power.
My file looks like this:
(Change it to your IP range)
192.168.10.0/24
Now the last file to make this work:
sudo -e /etc/default/distcc
Here we need to modify multiple things. (I’ll post the whole file at the end)
First change
STARTDISTCC="false"
to
STARTDISTCC="true"
so that the service can start.
Then we need to change
ALLOWEDNETS="127.0.0.1"
To our network’s address range
ALLOWEDNETS="192.168.10.0/24"
We also need to change
LISTENER="127.0.0.1"
to
LISTENER="0.0.0.0"
so that DistCC listens on all interfaces
and the last thing we need to change is
JOBS=""
to
JOBS="4"
to limit the number of jobs DistCC will accept at one time.
Here’s the whole file
STARTDISTCC="true"
ALLOWEDNETS="192.168.10.0/24"
LISTENER="0.0.0.0"
NICE="10"
JOBS="4"
ZEROCONF="false"
Now we just need to start the service:
sudo systemctl start distcc
Compiling⌗
Now we finally come back to our Host machine.
When you want to compile something using DistCC you need to add it’s path before any other paths so that it uses the DistCC binary instead of the compiler:
export PATH=/usr/lib/distcc:$PATH
Now we just need to get a project that we want to compile - my is Doxygen.
wget "https://github.com/doxygen/doxygen/releases/download/Release_1_9_8/doxygen-1.9.8.src.tar.gz"
tar -xvf doxygen-1.9.8.src.tar.gz
cd doxygen-1.9.8
mkdir build && cd build
Just to make sure that we’ll be using the correct compiler run
which aarch64-linux-gnu-gcc aarch64-linux-gnu-g++
it should lead to /usr/lib/distcc
like so: /usr/lib/distcc/aarch64-linux-gnu-gcc
Now we need to manually choose the compiler because if we’re cross compiling
and we’re using gcc
the Slaves will use their native gcc
instead of the one
needed for the target machine.
(The -G "Unix Makefiles" -Dbuild_doc=NO
are for building Doxygen,
use the ones for your project instead.)
CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ cmake -G "Unix Makefiles" -Dbuild_doc=NO ..
Look at the CMake output and make sure that it says
-- Check for working C compiler: /usr/lib/distcc/aarch64-linux-gnu-gcc
-- Check for working CXX compiler: /usr/lib/distcc/aarch64-linux-gnu-g++
If it doesn’t lead to /usr/lib/distcc
make sure to run
export PATH=/usr/lib/distcc:$PATH
as mentioned above.
Now you can just run
(Don’t forget to change -jx
to the number of threads you have on all your
Slave machines combined)
make -jx
and enjoy way more compiling performance than you did before!
PS: I hope that you enjoyed my guide. I haven’t found guides that easily described the setup needed to make DistCC work properly on different OSes, OS versions and architectures. I’ve always ended up with DistCC using the wrong compiler on basically all the Slave machines.
Thanks to experimenting for more than 2 days straight I’ve figured out that using Distrobox is the easiest way to get proper toolchains setup without needing to build your own custom ones and then messing around with the DistCC daemon to pick the right one.
Especially for the Raspberry Pi it can be quite hard to get the correct toolchain as the old ones available from RaspberryPi toolchain repository are deprecated and using different
GCC
versions does sometimes result in compilation errors.I’d really appreciate if you’d leave some feedback for me in the comments as I haven’t really done guides before and I’d love to make them, and my posts, easier to read and follow.
Thanks for reading and I hope that I helped you save some time while waiting for compilations to finish.