Distcc - Crushing Compilation Times⌗

Ever tried to compile something big on a slow PC like a Raspberry Pi? If you have, you know the struggle.

You’ve thought about cross-compiling, but it’s a chore.

DistCC is the answer – let powerful machines handle the heavy lifting while you stay productive on your Pi.

No more hassle, just faster compilation times.

Ok, I lied a bit. It’s not without hassle, but once you set this up, you don’t need to manage it anymore until you upgrade the toolchain But I found an easy way to easily have the same toolchains on all machines so keeping everything updated is really easy.

Few terms we need to clarify⌗

• Host
  • This is the system that you run the compilation on and that is the target of the compilation
• Slave
  • All the machines that the Host connects to and sends all the work to

Setting up the Host⌗

Because I’ve been setting up this for my Raspberry Pi 3, and most servers/machines run it (or something based on it), I’ll assume you’re running Debian.

But everything I’ll be describing here can easily be used on other distros. Just look it up on your distro’s wiki.

Installing the tools⌗

First of all we need to make sure that you have all the basic tools for compilation installed:

sudo apt install build-essential

Now we need to install distcc:

sudo apt install distcc -y

Configuring DistCC⌗

The Host machine is really easy to configure.

All you need to do is to open /etc/distcc/hosts and add the IP addresses of the Slave machines

(This will open the file in your $EDITOR)

sudo -e /etc/distcc/hosts

Now remove (or comment out) the 127.0.0.1 line (unless you want to compile on the Host as well - I’ll talk about this later) and add the IP addresses of the other machines followed by /n where n is the number of threads you want to run on that machine.

The file should then look something like this:

192.168.10.221:3632/11
192.168.10.23:3632/4

The 3632 port is the default one. You don’t need to change it, unless you want to run a more complicated setup. It that’s your case fell free to leave a comment below and I’ll add to this post.

Slaves⌗

Now that our Host is done, we need to setup our Slaves. To make our jobs easier we’ll be using Podman and Distrobox to run our DistCC daemon.

If you use the same OS on all machines you don’t need to follow the container setup but I still advise it in case the OS versions are different.

(Eg. Raspberry Pi is now Debian 11 while Debian 12 is the latest one available as of writing this).

Setting up podman⌗

This one should be easy as podman should already be in your distribution’s repository:

sudo apt install podman -y

Setting up distrobox⌗

This one is a bit more tricky as distrobox may not be present in your repositories.

But the instalation of distrobox is pretty straightforward so it should take only a while.

curl -s https://raw.githubusercontent.com/89luca89/distrobox/main/install | sudo sh

Setting up the container⌗

The container setup is a bit more unconventional from the normal way you use Distrobox.

We need to have systemd installed to start the distcc.service and we need to have it running as root to have the DistCC port accessible.

To setup the container use the same OS and version as the Host *(use cat /etc/os-release if you’re not sure)

(Replace debian:bullseye with a docker image for your OS and version and change the name so that you know what is the container for)

distrobox create -i debian:bullseye --root --init --additional-packages "systemd libpam-systemd" -n debian11

Let it download the image if needed and enter the container: This may take a long time depending on your SSD/HDD and internet speeds.

distrobox enter --root debian11

If you want to start DistCC again, eg. after rebooting, you can run this command:

(change debian11 to your container’s name)

distrobox enter --root -n debian11 -- /usr/bin/sudo systemctl start distcc

Setting up the toolchain and DistCC⌗

(inside the Distrobox container)

Now we just need to install the same toolchain as the Host machine and setup distcc.

What package to intall⌗

If you’re unsure about what packages to install look at your Host’s architecture:

uname -m

This command will output something like aarch64 and that is what you’re looking for.

Now we just need to install the correct GCC and G++ on the Slaves:

(Replace aarch64 with the Host’s architecture)

sudo apt install build-essential distcc g++-aarch64-linux-gnu gcc-aarch64-linux-gnu

Now you should be able to run

aarch64-linux-gnu-gcc --version

and have the same version on the Host and the Slave.

Configuring DistCC⌗

This one is a bit more tricky but it’s still nothing hard:

sudo -e /etc/distcc/clients.allow

Here you need to put in your network adresses that you want to allow. You can use specific adresses like 192.168.10.4 or you can just add 192.168.10.0/24 to allow all devices on 192.168.10.x address the use of this machine’s power.

My file looks like this:

(Change it to your IP range)

192.168.10.0/24

Now the last file to make this work:

sudo -e /etc/default/distcc

Here we need to modify multiple things. (I’ll post the whole file at the end)

First change

STARTDISTCC="false"

to

STARTDISTCC="true"

so that the service can start.

Then we need to change

ALLOWEDNETS="127.0.0.1"

To our network’s address range

ALLOWEDNETS="192.168.10.0/24"

We also need to change

LISTENER="127.0.0.1"

to

LISTENER="0.0.0.0"

so that DistCC listens on all interfaces

and the last thing we need to change is

JOBS=""

to

JOBS="4"

to limit the number of jobs DistCC will accept at one time.

Here’s the whole file

STARTDISTCC="true"

ALLOWEDNETS="192.168.10.0/24"

LISTENER="0.0.0.0"

NICE="10"

JOBS="4"

ZEROCONF="false"

Now we just need to start the service:

sudo systemctl start distcc

Compiling⌗

Now we finally come back to our Host machine.

When you want to compile something using DistCC you need to add it’s path before any other paths so that it uses the DistCC binary instead of the compiler:

export PATH=/usr/lib/distcc:$PATH

Now we just need to get a project that we want to compile - my is Doxygen.

wget "https://github.com/doxygen/doxygen/releases/download/Release_1_9_8/doxygen-1.9.8.src.tar.gz"
tar -xvf doxygen-1.9.8.src.tar.gz
cd doxygen-1.9.8
mkdir build && cd build

Just to make sure that we’ll be using the correct compiler run

which aarch64-linux-gnu-gcc aarch64-linux-gnu-g++

it should lead to /usr/lib/distcc like so: /usr/lib/distcc/aarch64-linux-gnu-gcc

Now we need to manually choose the compiler because if we’re cross compiling and we’re using gcc the Slaves will use their native gcc instead of the one needed for the target machine.

(The -G "Unix Makefiles" -Dbuild_doc=NO are for building Doxygen, use the ones for your project instead.)

CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ cmake -G "Unix Makefiles" -Dbuild_doc=NO ..

Look at the CMake output and make sure that it says

-- Check for working C compiler: /usr/lib/distcc/aarch64-linux-gnu-gcc
-- Check for working CXX compiler: /usr/lib/distcc/aarch64-linux-gnu-g++

If it doesn’t lead to /usr/lib/distcc make sure to run export PATH=/usr/lib/distcc:$PATH as mentioned above.

Now you can just run

(Don’t forget to change -jx to the number of threads you have on all your Slave machines combined)

make -jx

and enjoy way more compiling performance than you did before!

PS: I hope that you enjoyed my guide. I haven’t found guides that easily described the setup needed to make DistCC work properly on different OSes, OS versions and architectures. I’ve always ended up with DistCC using the wrong compiler on basically all the Slave machines.

Thanks to experimenting for more than 2 days straight I’ve figured out that using Distrobox is the easiest way to get proper toolchains setup without needing to build your own custom ones and then messing around with the DistCC daemon to pick the right one.

Especially for the Raspberry Pi it can be quite hard to get the correct toolchain as the old ones available from RaspberryPi toolchain repository are deprecated and using different GCC versions does sometimes result in compilation errors.

I’d really appreciate if you’d leave some feedback for me in the comments as I haven’t really done guides before and I’d love to make them, and my posts, easier to read and follow.

Thanks for reading and I hope that I helped you save some time while waiting for compilations to finish.