Latest 10 recent news (see index)

September 29, 2024

Goodbye, Python 2! Hello, New Buildbot!

At long last, Void is saying goodbye to Python 2. Python ended support for Python 2 in 2020, but Void still had over 200 packages that depended on it. Since then, Void contributors have updated, patched, or removed these packages. For the moment, Python 2 will remain in the repositories as python2 (along with python2-setuptools and python2-pip). python is now a metapackage that will soon point to python3.

One of the biggest blockers for this project was some of Void’s own infrastructure: our buildbot, which builds all packages for delivery to users. For a long time, we were stuck on buildbot 0.8.12 (released on 21 April 2015 and using Python 2), because it was complex to get working, had many moving parts, and was fairly fragile. To update it to a modern version would require significant time and effort.

Now, we move into the future: we’ve upgraded our buildbot to version 4.0, and it is now being managed via our orchestration system, Nomad, to improve reliability, observability, and reproducibility in deployment. Check out the 2023 Infrastructure Week series of blog posts for more info about how and why Void uses Nomad.

Visit the new buildbot dashboard at build.voidlinux.org and watch your packages build!

July 12, 2024

Welcome a New Contributor!

The Void project is pleased to welcome aboard another new member, @tranzystorekk.

Interested in seeing your name in a future update here? Read our Contributing Page and find a place to help out! New members are invited from the community of contributors.

March 14, 2024

March 2024 Image Release (and Raspberry Pi 5 support)

We’re pleased to announce that the 20240314 image set has been promoted to current and is now generally available.

You can find the new images on our downloads page and on our many mirrors.

Some highlights of this release:

• A keymap selector is now shown in LightDM on XFCE images (@classabbyamp in #354)
• The chrony NTP daemon is now enabled by default in live images (@classabbyamp in abbd636)
• Raspberry Pi images can now be installed on non-SD card storage without manual configuration on models that support booting from USB or NVMe (@classabbyamp in #361)
• Raspberry Pi images now default to a /boot partition of 256MiB instead of 64MiB (@classabbyamp in #368)

rpi-aarch64* PLATFORMFSes and images now support the Raspberry Pi 5. After installation, the kernel can be switched to the Raspberry Pi 5-specific rpi5-kernel.

You may verify the authenticity of the images by following the instructions on the downloads page, and using the following minisign key information:

untrusted comment: minisign public key A3FCFCCA9D356F86
RWSGbzWdyvz8o4nrhY1nbmHLF6QiFH/AQXs1mS/0X+t1x3WwUA16hdc/

January 22, 2024

Changes to xbps-src Masterdir Creation and Use

In an effort to simplify the usage of xbps-src, there has been a small change to how masterdirs (the containers xbps-src uses to build packages) are created and used.

The default masterdir is now called masterdir-<arch>, except when masterdir already exists or when using xbps-src in a container (where it’s still masterdir).

Creation

When creating a masterdir for an alternate architecture or libc, the previous syntax was:

./xbps-src -m <name> binary-bootstrap <arch>

Now, the <arch> should be specified using the new -A (host architecture) flag:

./xbps-src -A <arch> binary-bootstrap

This will create a new masterdir called masterdir-<arch> in the root of your void-packages repository checkout.

Arbitrarily-named masterdirs can still be created with -m <name>.

Usage

Instead of specifying the alternative masterdir directly, you can now use the -A (host architecture) flag to use the masterdir-<arch> masterdir:

./xbps-src -A <arch> pkg <pkgname>

Arbitrarily-named masterdirs can still be used with -m <name>.

January 22, 2024

Welcome New Contributors!

The Void project is pleased to welcome aboard 2 new members.

Joining us to work on packages are @oreo639 and @cinerea0.

Interested in seeing your name in a future update here? Read our Contributing Page and find a place to help out! New members are invited from the community of contributors.

January 04, 2024

glibc 2.38 Update Issues and Solutions

With the update to glibc 2.38, libcrypt.so.1 is no longer provided by glibc. Libcrypt is an important library for several core system packages that use cryptographic functions, including pam. The library has changed versions, and the legacy version is still available for precompiled or proprietary applications. The new version is available on Void as libxcrypt and the legacy version is libxcrypt-compat.

With this change, some kinds of partial upgrades can leave PAM unable to function. This breaks tools like sudo, doas, and su, as well as breaking authentication to your system. Symptoms include messages like “PAM authentication error: Module is unknown”. If this has happened to you, you can either:

• add init=/bin/sh to your kernel command-line in the bootloader and downgrade glibc,
• or mount the system’s root partition in a live environment, chroot into it, and install libxcrypt-compat

Either of these steps should allow you to access your system as normal and run a full update.

To ensure the disastrous partial upgrade (described above) cannot happen, glibc-2.38_3 now depends on libxcrypt-compat. With this change, it is safe to perform partial upgrades that include glibc 2.38.

October 24, 2023

Changes to Repository Sync

Void is a complex system, and over time we make changes to reduce this complexity, or shift it to easier to manage components. Recently through the fantastic work of one of our maintainers classabbyamp our repository sync system has been dramatically improved.

Previously our system was based on a series of host managed rsyncs running on either snooze or cron based timers. These syncs would push files to a central location to be signed and then distributed. This central location is sometimes referred to as the “shadow repo” since its not directly available to end users to synchronize from, and we don’t usually allow anyone outside Void to have access to it.

As you might have noticed from the Fastly Overview the packages take a long path from builders to repos. What is not obvious from the graph shown is that the shadow repo previously lived on the musl builder, meaning that packages would get built there, copied to the glibc builder, then copied back to the musl builder and finally copied to a mirror. So many copies! To streamline this process, the shadow mirror is now just the glibc server, since that’s where the packages have to wind up for architectural reasons anyway. This means we were able to cut out 2 rsyncs and reclaim a large amount of space on the musl builder, making the entire process less fragile and more streamlined.

But just removing rsyncs isn’t all that was done. To improve the time it takes for packages to make it to users, we’ve also switched the builders from using a time based sync to using lsyncd to take more active management of the synchronization process. In addition to moving to a more sustainable sync process, the entire process was moved up into our Nomad managed environment. Nomad allows us to more easily update services, monitor them for long term trends, and to make it clearer where services are deployed.

In addition to fork-lifting the sync processes, we also forklifted void-updates, xlocate, xq-api (package search), and the generation of the docs-site into Nomad. These changes represent some of the very last services that were not part of our modernized container orchestrated infrastructure.

Visually, this is what the difference looks like. Here’s before:

And here’s what the sync looks like now, note that there aren’t any cycles for syncs now:

If you run a downstream mirror we need your help! If your mirror has existed for long enough, its possible that you were still synchronizing from alpha.de.repo.voidlinux.org, which has been a dead servername for several years now. Since moving around sync traffic is key to our ability to keep the lights on, we’ve provisioned a new dedicated DNS record for mirrors to talk to. The new repo-sync.voidlinux.org is the preferred origin point for all sync traffic and using it means that we can transparently move the sync origin during maintenance rather than causing an rsync hang on your sync job. Please check where you’re mirroring from and update accordingly.

October 07, 2023

Changes to Python 3 and pip on Void

Happy Pythonmas! It’s October, which means it’s Python 3 update season. This year, along with the usual large set of updates for Python packages, a safety feature for pip, the Python package manager, has been activated. To ensure that Python packages installed via XBPS and those installed via pip don’t interfere with one another, the system-wide Python environment has been marked as “externally managed”.

If you try to use pip3 or pip3 --user outside of a Python virtual environment, you may see this error that provides guidance on how to deploy a virtual environment suitable for use with pip:

This system-wide Python installation is managed by the Void Linux package
manager, XBPS. Installation of Python packages from other sources is not
normally allowed.

To install a Python package not offered by Void Linux, consider using a virtual
environment, e.g.:

  python3 -m venv /path/to/venv
  /path/to/venv/pip install <package>

Appending the flag --system-site-packages to the first command will give the
virtual environment access to any Python package installed via XBPS.

Invoking python, pip, and executables installed by pip in /path/to/venv/bin
should automatically use the virtual environment. Alternatively, source its
activation script to add the environment to the command search path for a shell:

  . /path/to/venv/activate

After activation, running

  deactivate

will remove the environment from the search path without destroying it.

The XBPS package python3-pipx provides pipx, a convenient tool to automatically
manage virtual environments for individual Python applications.

You can read more about this change on Python’s website in PEP 668.

To simplify the use of Void-based containers, all Void container images tagged 20231003R1 or later will explicitly ignore the “externally managed” marker. Containers based on these images will still be able to use pip to install Python packages in the container-wide environment.

Here Be Dragons

If you really want to be able to install packages with pip in the system- or user-wide Python environment, there are several options, but beware: this can cause hard-to-debug issues with Python applications, or issues when updating with XBPS.

1. Use pip’s --break-system-packages flag. This only applies to the current invocation.
2. Set pip’s configuration: pip3 config set install.break-system-packages True. This will apply to all future invocations.
3. Add a noextract=/usr/lib/python*/EXTERNALLY-MANAGED rule to your XBPS configuration and re-install the python3 package. This will apply to all future invocations.

August 30, 2023

Changes to Container Images

To simplify the container experience, we’ve revamped the way Void’s OCI container images are built and tagged.

In short:

• Architectures are now encoded using the platform in the manifest instead of in tags
• Different libcs and flavors of images are now separate images instead of tags
• The mini flavor is no longer built, as they did not work as intended

You can check out the available images on the Download page or on Github.

If you’re interested in the technical details, you can take a look at the pull request for these changes.

To migrate your current containers:

July 21, 2023

Infrastructure Week - Day 5: Making Distributed Infrastructure Work for Distributed Teams

Void runs a distributed team of maintainers and contributors. Making infrastructure work for any team is a confluence of goals, user experience choices, and hard requirements. Making infrastructure work for a distributed team adds on the complexity of accessing everything securely over the open internet, and doing so in a way that is still convenient and easy to setup. After all, a light switch is difficult to use is likely to lead to lights being left on.

We take several design criteria into mind when designing new systems and services that make Void work. We also periodically re-evaluate systems that have been built to ensure that they still follow good design practices in a way that we are able to maintain, and that does what we want. Lets dive in to some of these design practices.

No Maintainer Facing VPNs

VPNs, or Virtual Private Networks are ways of interconnecting systems such that the network in between appears to vanish beneath a layer of abstraction. WireGuard, OpenVPN, and IPSec are examples of VPNs. OpenVPN and IPSec, a client program handles encryption and decryption of traffic on a tunnel or tap device that translates packets into and out of the kernel network stack. If you work in a field that involves using a computer for your job, your employer may make use of a VPN to grant your device connectivity to their corporate network environment without you having to be physically present in a building. VPN technologies can also be used to make multiple physical sites appear to all be on the same network.

Void uses WireGuard to provide machine-to-machine connectivity for our fleet, but only within our fleet. Maintainers always access services without a VPN. Why do we do this, and how do we do it? First the why. We operate in this way because corporate VPNs are often cumbersome, require split horizon DNS (where you get different DNS answers depending on where you resolve from) and require careful planning to make sure no subnet overlap occurs between the VPN, the network you are connecting to, and your local network. If there were an overlap, the kernel would be unable to determine where to send the packets since it has multiple routes for the same subnets. There are cases where this is a valid network topology (ECMP), but that is not what is being discussed here. We also have no reason to use a VPN. Most of the use cases that still require a VPN have to do with transporting arbitrary TCP streams across a network, but this is unnecessary. For Void, all our services are either HTTP based or are transported over SSH.

For almost all our systems that we interact with daily, either a web interface or HTTP-based API is provided. For the devspace file hosting system, maintainers can use SFTP via SSH. Both HTTP and SSH have robust, extremely well tested authentication and encryption options. When designing a system for secure access, defense in depth is important, but so is trust that the cryptographic primitives you have selected actually work. We trust that HTTPS works, and so there is no need to wrap the connection in an additional layer of encryption. The same goes for SSH, which we use exclusively public-key authentication for. This choice is sometimes challenging to maintain, since it means that we need to ensure highly available HTTP proxies and secure, easily maintained SSH key implementations, we have found it works well for us. In addition to the static files that all our tier 1 mirrors serve, the mirrors are additionally capable of acting as proxies. This allows us to terminate the externally trusted TLS session at a webserver running nginx, and then pass the traffic over our internal encrypted fabric to the destination service.

For SSH we simply make use of AuthorizedKeysCommand to summon keys from NetAuth allowing authorized maintainers to log onto servers or ssh-enabled services wherever their keys are validated. For the devspace service which has a broader ACL than our base hardware, we can enhance its separation by running an SFTP server distinct from the host sshd. This allows us to ensure that it is impossible for a key validated for devspace to inadvertently authorize a shell login to the underlying host.

For all other services, we make use of the service level authentication as and when required. We use combinations of Native NetAuth, LDAP proxies, and PAM helpers to make all access seamless for maintainers via our single sign on system. Removing the barrier of a VPN also means that during an outage, there’s one less component we need to troubleshoot and debug, and one less place for systems to break.

Use of Composable Systems

Distributed systems are often made up of complex, interdependent sub-assemblies. This level of complexity is fine for dedicated teams who are paid to maintain systems day in and day out, but is difficult to pull off with an all-volunteer team that works on Void in their free time. Distributed systems are also best understood on a whiteboard, and this doesn’t lend itself well to making a change on a laptop from a train, or reviewing a delta from a tablet between other tasks. While substantive changes are almost always made from a full terminal, the ratio of substantive changes to items requiring only quick verification is significant, and its important to maintain a level of understand-ability.

In order to maintain the level of understand-ability of the infrastructure at a level that permits a reasonable time investment, we make use of composable systems. Composable systems can best be thought of as infrastructure built out of common sub-assemblies. Think Lego blocks for servers. This allows us to have a common base library of components, for example webservers, synchronization primitives, and timers, and then build these into complex systems through joining their functionality together.

We primarily use containers to achieve this composeability. Each container performs a single task or a well defined sub-process in a larger workflow. For example we can look at the workflow required to serve https://man.voidlinux.org/. In this workflow, a task runs periodically to extract all man pages from all packages, then another process runs to copy those files to the mirrors, and finally a process runs to produce an HTTP response to a given man page request. Notice there that its an HTTP response, but the man site is served securely over HTTPS. This is because across all of our web-based services we make use of common infrastructure such as load balancers and our internal network. This allows applications to focus on their individual functions without needing to think about the complexity of serving an encrypted connection to the outside world.

By designing our systems this way, we also gain another neat feature: local testing. Since applications can be broken down into smaller building blocks, we can take just the single building block under scrutiny and run it locally. Likewise, we can upgrade individual components of the system to determine if they improve or worsen a problem. With some clever configuration, we can even upgrade half of a system that’s highly available and compare the old and new implementations side by side to see if we like one over the other. This composability enables us to configure complex systems as individual, understandable components.

Its worth clarifying though that this is not necessarily a microservices architecture. We don’t really have any services that could be defined as microservices in the conventional sense. Instead this architecture should be thought of as the Unix Philosophy as applied to infrastructure components. Each component has a single well understood goal and that’s all it does. Other goals are accomplished by other services.

We assemble all our various composed services into the service suite that Void provides via our orchestration system (Nomad) and our load balancers (nginx) which allow us to present the various disparate systems as though they were one to the outside world, while still maintaining them as separate service “verticals” side by side each other internally.

Everything in Git

Void’s packages repo is a large git repo with hundreds of contributors and many maintainers. This package bazaar contains all manner of different software that is updated, verified, and accepted by a team that spans the globe. Our infrastructure is no different, but involves fewer people. We make use of two key systems to enable our Infrastructure as Code (IaC) approach.

The first of these tools is Ansible. Ansible is a configuration management utility written in python which can programatically SSH into machines, template files, install and remove packages and more. Ansible takes its instructions as collections of YAML files called roles that are assembled into playbooks (composeability!). These roles come from either the main void-infrastructure repo, or as individual modules from the void-ansible-roles organization on GitHub. Since this is code checked into Git, we can use ansible-lint to ensure that the code is consistent and lint-free. We can then review the changes as a diff, and work on various features on branches just like changes to void-packages. The ability to review what changed is also a powerful debugging tool to allow us to see if a configuration delta led to or resolved a problem, and if we’ve encountered any similar kind of change in the past.

The second tool we use regularly is Terraform. Whereas Ansible configures servers, Terraform configures services. We can apply Terraform to almost any service that has an API as most popular services that Void consumes have terraform providers. We use Terraform to manage our policy files that are loaded into Nomad, Consul and Vault, we use it to provision and deprovision machines on DigitalOcean, Google and AWS, and we use it to update our DNS records as services change. Just like Ansible, Terraform has a linter, a robust module system for code re-use, and a really convenient system for producing a diff between what the files say the service should be doing and what it actually is doing.

Perhaps the most important use of Terraform for us is the formalized onboarding and offboarding process for maintainers. When a new maintainer is proposed and has been accepted through discussion within the Void team, we’ll privately reach out to them to ask if they want to join the project. Given that a candidate accepts the offer to join the group of pkg-committers, the action that formally brings them on to the team is a patch applied to the Terraform that manages our GitHub organization and its members. We can then log approvals, welcome the new contributor to our team with suitable emoji, and grant access all in one convenient place.

Infrastructure as Code allows our distributed team to easily maintain our complex systems with a written record that we can refer back to. The ability to defer changes to an asynchronous review is imperative to manage the workflows of a distributed team.

Good Lines of Communication

Of course, all the infrastructure in the world doesn’t help if the people using it can’t effectively communicate. To make sure this issue doesn’t occur for Void, we have multiple forms of communication with different features. For real-time discussions and even some slower ones, we make use of IRC on Libera.chat. Though many communities appear to be moving away from synchronous text, we find that it works well for us. IRC is a great protocol that allows each member of the team to connect using the interface that they believe is the best for them, as well as to allow our automated systems to connect in as well.

For conversations that need more time or are generally going to be longer we make use of email or a group-scoped discussion on GitHub. This allows for threaded messaging and a topic that can persist for days or weeks if needed. Maintaining a long running thread can help us tease apart complicated issues or ensure everyone’s voice is heard. Long time users of Void may remember our forum, which has since been supplanted by a subreddit and most recently GitHub Discussions. These threaded message boards are also examples of places that we converse and exchange status information, but in a more social context.

For discussion that needs to pertain directly to our infrastructure, we open tickets against the infrastructure repo. This provides an extremely clear place to report issues, discuss fixes, and collate information relating to ongoing work. It also allows us to leverage GitHub’s commit message parsing to automatically resolve a discussion thread once a fix has been applied by closing the issue. For really large changes, we can also use GitHub projects, though in recent years we have not made use of this particular organization system for issues (we use tags).

No matter where we converse though, its always important to make sure we converse clearly and concisely. Void’s team speaks a variety of languages, though we mostly converse in English which is not known for its intuitive clarity. When making hazardous changes, we often push changes to a central location and ask for explicit review of dangerous parts, and call out clearly what the concerns are and what requires review. In this way we ensure that all of Void’s various services stay up, and our team members stay informed.

This post was authored by maldridge who runs most of the day to day operations of the Void fleet. On behalf of the entire Void team, I hope you have enjoyed this week’s dive into the infrastructure that makes Void happen, and have learned some new things. We’re always working to improve systems and make them easier to maintain or provide more useful features, so if you want to contribute, join us in IRC. Feel free to ask questions about this post or any of our others this week on GitHub Discussions or in IRC.