diff --git a/README.md b/README.md index bd7fed87901..5d8c7a35752 100644 --- a/README.md +++ b/README.md @@ -1,511 +1,5 @@ -## The XBPS source packages collection +# Custom Void Packages -This repository contains the XBPS source packages collection to build binary packages -for the Void Linux distribution. +This repo holds my custom package templates for void linux -The included `xbps-src` script will fetch and compile the sources, and install its -files into a `fake destdir` to generate XBPS binary packages that can be installed -or queried through the `xbps-install(1)` and `xbps-query(1)` utilities, respectively. - -See [Contributing](./CONTRIBUTING.md) for a general overview of how to contribute and the -[Manual](./Manual.md) for details of how to create source packages. - -### Table of Contents - -- [Requirements](#requirements) -- [Quick start](#quick-start) -- [chroot methods](#chroot-methods) -- [Install the bootstrap packages](#install-bootstrap) -- [Configuration](#configuration) -- [Directory hierarchy](#directory-hierarchy) -- [Building packages](#building-packages) -- [Package build options](#build-options) -- [Sharing and signing your local repositories](#sharing-and-signing) -- [Rebuilding and overwriting existing local packages](#rebuilding) -- [Enabling distcc for distributed compilation](#distcc) -- [Distfiles mirrors](#distfiles-mirrors) -- [Cross compiling packages for a target architecture](#cross-compiling) -- [Using xbps-src in a foreign Linux distribution](#foreign) -- [Remaking the masterdir](#remaking-masterdir) -- [Keeping your masterdir uptodate](#updating-masterdir) -- [Building 32bit packages on x86_64](#building-32bit) -- [Building packages natively for the musl C library](#building-for-musl) -- [Building void base-system from scratch](#building-base-system) - -### Requirements - -- GNU bash -- xbps >= 0.56 -- git(1) - unless configured to not, see etc/defaults.conf -- common POSIX utilities included by default in almost all UNIX systems -- curl(1) - required by `xbps-src update-check` - -For bootstrapping additionally: -- flock(1) - util-linux -- bsdtar or GNU tar (in that order of preference) -- install(1) - GNU coreutils -- objcopy(1), objdump(1), strip(1): binutils - -`xbps-src` requires [a utility to chroot](#chroot-methods) and bind mount existing directories -into a `masterdir` that is used as its main `chroot` directory. `xbps-src` supports -multiple utilities to accomplish this task. - -> NOTE: `xbps-src` does not allow building as root anymore. Use one of the chroot -methods. - - -### Quick start - -Clone the `void-packages` git repository and install the bootstrap packages: - -``` -$ git clone https://github.com/void-linux/void-packages.git -$ cd void-packages -$ ./xbps-src binary-bootstrap -``` - -Build a package by specifying the `pkg` target and the package name: - -``` -$ ./xbps-src pkg -``` - -Use `./xbps-src -h` to list all available targets and options. - -To build packages marked as 'restricted', modify `etc/conf`: - -``` -$ echo XBPS_ALLOW_RESTRICTED=yes >> etc/conf -``` - -Once built, the package will be available in `hostdir/binpkgs` or an appropriate subdirectory (e.g. `hostdir/binpkgs/nonfree`). To install the package: - -``` -# xbps-install --repository hostdir/binpkgs -``` - -Alternatively, packages can be installed with the `xi` utility, from the `xtools` package. `xi` takes the repository of the current working directory into account. - -``` -$ xi -``` - - -### chroot methods - -#### xbps-uunshare(1) (default) - -XBPS utility that uses `user_namespaces(7)` (part of xbps, default without `-t` flag). - -This utility requires these Linux kernel options: - -- CONFIG\_NAMESPACES -- CONFIG\_IPC\_NS -- CONFIG\_UTS\_NS -- CONFIG\_USER\_NS - -This is the default method, and if your system does not support any of the required kernel -options it will fail with `EINVAL (Invalid argument)`. - -#### xbps-uchroot(1) - -XBPS utility that uses `namespaces` and must be `setgid` (part of xbps). - -> NOTE: This is the only method that implements functionality of `xbps-src -t`, therefore the -flag ignores the choice made in configuration files and enables `xbps-uchroot`. - -This utility requires these Linux kernel options: - -- CONFIG\_NAMESPACES -- CONFIG\_IPC\_NS -- CONFIG\_PID\_NS -- CONFIG\_UTS\_NS - -Your user must be added to a special group to be able to use `xbps-uchroot(1)` and the -executable must be `setgid`: - - # chown root: xbps-uchroot - # chmod 4750 xbps-uchroot - # usermod -a -G - -> NOTE: by default in void you shouldn't do this manually, your user must be a member of -the `xbuilder` group. - -To enable it: - - $ cd void-packages - $ echo XBPS_CHROOT_CMD=uchroot >> etc/conf - -If for some reason it's erroring out as `ERROR clone (Operation not permitted)`, check that -your user is a member of the required `group` and that `xbps-uchroot(1)` utility has the -proper permissions and owner/group as explained above. - -#### bwrap(1) - -bubblewrap, sandboxing tool for unprivileged users that uses -user namespaces or setuid. -See . - -#### ethereal - -Destroys host system it runs on. Only useful for one-shot containers, i.e docker (used with CI). - - -### Install the bootstrap packages - -There is a set of packages that makes up the initial build container, called the `bootstrap`. -These packages are installed into the `masterdir` in order to create the container. - -The primary and recommended way to set up this container is using the `binary-bootstrap` -command. This will use pre-existing binary packages, either from remote `xbps` repositories -or from your local repository. - -There is also the `bootstrap` command, which will build all necessary `bootstrap` packages from -scratch. This is usually not recommended, since those packages are built using your host system's -toolchain and are neither fully featured nor reproducible (your host system may influence the -build) and thus should only be used as a stage 0 for bootstrapping new Void systems. - -If you still choose to use `bootstrap`, use the resulting stage 0 container to rebuild all -`bootstrap` packages again, then use `binary-bootstrap` (stage 1) and rebuild the `bootstrap` -packages once more (to gain stage 2, and then use `binary-bootstrap` again). Once you've done -that, you will have a `bootstrap` set equivalent to using `binary-bootstrap` in the first place. - -Also keep in mind that a full source `bootstrap` is time consuming and will require having an -assortment of utilities installed in your host system, such as `binutils`, `gcc`, `perl`, -`texinfo` and others. - -### Configuration - -The `etc/defaults.conf` file contains the possible settings that can be overridden -through the `etc/conf` configuration file for the `xbps-src` utility; if that file -does not exist, will try to read configuration settings from `$XDG_CONFIG_HOME/xbps-src.conf`, `~/.config/xbps-src.conf`, `~/.xbps-src.conf`. - -If you want to customize default `CFLAGS`, `CXXFLAGS` and `LDFLAGS`, don't override -those defined in `etc/defaults.conf`, set them on `etc/conf` instead i.e: - - $ echo 'XBPS_CFLAGS="your flags here"' >> etc/conf - $ echo 'XBPS_LDFLAGS="your flags here"' >> etc/conf - -Native and cross compiler/linker flags are set per architecture in `common/build-profiles` -and `common/cross-profiles` respectively. Ideally those settings are good enough by default, -and there's no need to set your own unless you know what you are doing. - -#### Virtual packages - -The `etc/defaults.virtual` file contains the default replacements for virtual packages, -used as dependencies in the source packages tree. - -If you want to customize those replacements, copy `etc/defaults.virtual` to `etc/virtual` -and edit it accordingly to your needs. - - -### Directory hierarchy - -The following directory hierarchy is used with a default configuration file: - - /void-packages - |- common - |- etc - |- srcpkgs - | |- xbps - | |- template - | - |- hostdir - | |- binpkgs ... - | |- ccache ... - | |- distcc- ... - | |- repocache ... - | |- sources ... - | - |- masterdir- - | |- builddir -> ... - | |- destdir -> ... - | |- host -> bind mounted from - | |- void-packages -> bind mounted from - - -The description of these directories is as follows: - - - `masterdir-`: master directory to be used as rootfs to build/install packages. - - `builddir`: to unpack package source tarballs and where packages are built. - - `destdir`: to install packages, aka **fake destdir**. - - `hostdir/ccache`: to store ccache data if the `XBPS_CCACHE` option is enabled. - - `hostdir/distcc-`: to store distcc data if the `XBPS_DISTCC` option is enabled. - - `hostdir/repocache`: to store binary packages from remote repositories. - - `hostdir/sources`: to store package sources. - - `hostdir/binpkgs`: local repository to store generated binary packages. - - -### Building packages - -The simplest form of building package is accomplished by running the `pkg` target in `xbps-src`: - -``` -$ cd void-packages -$ ./xbps-src pkg -``` - -When the package and its required dependencies are built, the binary packages will be created -and registered in the default local repository at `hostdir/binpkgs`; the path to this local repository can be added to -any xbps configuration file (see xbps.d(5)) or by explicitly appending them via cmdline, i.e: - - $ xbps-install --repository=hostdir/binpkgs ... - $ xbps-query --repository=hostdir/binpkgs ... - -By default **xbps-src** will try to resolve package dependencies in this order: - - - If a dependency exists in the local repository, use it (`hostdir/binpkgs`). - - If a dependency exists in a remote repository, use it. - - If a dependency exists in a source package, use it. - -It is possible to avoid using remote repositories completely by using the `-N` flag. - -> The default local repository may contain multiple *sub-repositories*: `debug`, `multilib`, etc. - - -### Package build options - -The supported build options for a source package can be shown with `xbps-src show-options`: - - $ ./xbps-src show-options foo - -Build options can be enabled with the `-o` flag of `xbps-src`: - - $ ./xbps-src -o option,option1 pkg foo - -Build options can be disabled by prefixing them with `~`: - - $ ./xbps-src -o ~option,~option1 pkg foo - -Both ways can be used together to enable and/or disable multiple options -at the same time with `xbps-src`: - - $ ./xbps-src -o option,~option1,~option2 pkg foo - -The build options can also be shown for binary packages via `xbps-query(1)`: - - $ xbps-query -R --property=build-options foo - -> NOTE: if you build a package with a custom option, and that package is available -in an official void repository, an update will ignore those options. Put that package -on `hold` mode via `xbps-pkgdb(1)`, i.e `xbps-pkgdb -m hold foo` to ignore updates -with `xbps-install -u`. Once the package is on `hold`, the only way to update it -is by declaring it explicitly: `xbps-install -u foo`. - -Permanent global package build options can be set via `XBPS_PKG_OPTIONS` variable in the -`etc/conf` configuration file. Per package build options can be set via -`XBPS_PKG_OPTIONS_`. - -> NOTE: if `pkgname` contains `dashes`, those should be replaced by `underscores` -i.e `XBPS_PKG_OPTIONS_xorg_server=opt`. - -The list of supported package build options and its description is defined in the -`common/options.description` file or in the `template` file. - - -### Sharing and signing your local repositories - -To share a local repository remotely it's mandatory to sign it and the binary packages -stored on it. This is accomplished with the `xbps-rindex(1)` utility. - -First a RSA key must be created with `openssl(1)` or `ssh-keygen(1)`: - - $ openssl genrsa -des3 -out privkey.pem 4096 - -or - - $ ssh-keygen -t rsa -b 4096 -m PEM -f privkey.pem - -> Only RSA keys in PEM format are currently accepted by xbps. - -Once the RSA private key is ready you can use it to initialize the repository metadata: - - $ xbps-rindex --sign --signedby "I'm Groot" --privkey privkey.pem $PWD/hostdir/binpkgs - -And then make a signature per package: - - $ xbps-rindex --sign-pkg --privkey privkey.pem $PWD/hostdir/binpkgs/*.xbps - -> If --privkey is unset, it defaults to `~/.ssh/id_rsa`. - -If the RSA key was protected with a passphrase you'll have to type it, or alternatively set -it via the `XBPS_PASSPHRASE` environment variable. - -Once the binary packages have been signed, check if the repository contains the appropriate `hex fingerprint`: - - $ xbps-query --repository=hostdir/binpkgs -vL - ... - -Each time a binary package is created, a package signature must be created with `--sign-pkg`. - -> It is not possible to sign a repository with multiple RSA keys. - -If packages in `hostdir/binpkgs` are signed, the key in `.plist` format (as imported by xbps) can be placed -in `etc/repo-keys/` to prevent xbps-src from prompting to import that key. - - -### Rebuilding and overwriting existing local packages - -Packages are overwritten on every build to make getting package with changed build options easy. -To make xbps-src skip build and preserve first package build with given version and revision, -same as in official void repository, set `XBPS_PRESERVE_PKGS=yes` in `etc/conf` file. - -Reinstalling a package in your target `rootdir` can be easily done too: - - $ xbps-install --repository=/path/to/local/repo -yf xbps-0.25_1 - -Using `-f` flag twice will overwrite configuration files. - -> Please note that the `package expression` must be properly defined to explicitly pick up -the package from the desired repository. - - -### Enabling distcc for distributed compilation - -Setup the workers (machines that will compile the code): - - # xbps-install -Sy distcc - -Modify the configuration to allow your local network machines to use distcc (e.g. `192.168.2.0/24`): - - # echo "192.168.2.0/24" >> /etc/distcc/clients.allow - -Enable and start the `distccd` service: - - # ln -s /etc/sv/distccd /var/service - -Install distcc on the host (machine that executes xbps-src) as well. -Unless you want to use the host as worker from other machines, there is no need -to modify the configuration. - -On the host you can now enable distcc in the `void-packages/etc/conf` file: - - XBPS_DISTCC=yes - XBPS_DISTCC_HOSTS="localhost/2 --localslots_cpp=24 192.168.2.101/9 192.168.2.102/2" - XBPS_MAKEJOBS=16 - -The example values assume a localhost CPU with 4 cores of which at most 2 are used for compiler jobs. -The number of slots for preprocessor jobs is set to 24 in order to have enough preprocessed data for other CPUs to compile. -The worker 192.168.2.101 has a CPU with 8 cores and the /9 for the number of jobs is a saturating choice. -The worker 192.168.2.102 is set to run at most 2 compile jobs to keep its load low, even if its CPU has 4 cores. -The XBPS_MAKEJOBS setting is increased to 16 to account for the possible parallelism (2 + 9 + 2 + some slack). - - -### Distfiles mirror(s) - -In etc/conf you may optionally define a mirror or a list of mirrors to search for distfiles. - - $ echo 'XBPS_DISTFILES_MIRROR="ftp://192.168.100.5/gentoo/distfiles"' >> etc/conf - -If more than one mirror is to be searched, you can either specify multiple URLs separated -with blanks, or add to the variable like this - - $ echo 'XBPS_DISTFILES_MIRROR+=" https://sources.voidlinux.org/"' >> etc/conf - -Make sure to put the blank after the first double quote in this case. - -The mirrors are searched in order for the distfiles to build a package until the -checksum of the downloaded file matches the one specified in the template. - -Ultimately, if no mirror carries the distfile, or in case all downloads failed the -checksum verification, the original download location is used. - -If you use `uchroot` for your XBPS_CHROOT_CMD, you may also specify a local path -using the `file://` prefix or simply an absolute path on your build host (e.g. /mnt/distfiles). -Mirror locations specified this way are bind mounted inside the chroot environment -under $XBPS_MASTERDIR and searched for distfiles just the same as remote locations. - - -### Cross compiling packages for a target architecture - -Currently `xbps-src` can cross build packages for some target architectures with a cross compiler. -The supported target is shown with `./xbps-src -h`. - -If a source package has been adapted to be **cross buildable** `xbps-src` will automatically build the binary package(s) with a simple command: - - $ ./xbps-src -a pkg - -If the build for whatever reason fails, might be a new build issue or simply because it hasn't been adapted to be **cross compiled**. - - -### Using xbps-src in a foreign Linux distribution - -xbps-src can be used in any recent Linux distribution matching the CPU architecture. - -To use xbps-src in your Linux distribution use the following instructions. Let's start downloading the xbps static binaries: - - $ wget http://repo-default.voidlinux.org/static/xbps-static-latest.-musl.tar.xz - $ mkdir ~/XBPS - $ tar xvf xbps-static-latest.-musl.tar.xz -C ~/XBPS - $ export PATH=~/XBPS/usr/bin:$PATH - -If `xbps-uunshare` does not work because of lack of `user_namespaces(7)` support, -try other [chroot methods](#chroot-methods). - -Clone the `void-packages` git repository: - - $ git clone https://github.com/void-linux/void-packages.git - -and `xbps-src` should be fully functional; just start the `bootstrap` process, i.e: - - $ ./xbps-src binary-bootstrap - -The default masterdir is created in the current working directory, i.e. `void-packages/masterdir-`, where `` for the default masterdir is is the native xbps architecture. - - -### Remaking the masterdir - -If for some reason you must update xbps-src and the `bootstrap-update` target is not enough, it's possible to recreate a masterdir with two simple commands (please note that `zap` keeps your `ccache/distcc/host` directories intact): - - $ ./xbps-src zap - $ ./xbps-src binary-bootstrap - - -### Keeping your masterdir uptodate - -Sometimes the bootstrap packages must be updated to the latest available version in repositories, this is accomplished with the `bootstrap-update` target: - - $ ./xbps-src bootstrap-update - - -### Building 32bit packages on x86_64 - -Two ways are available to build 32bit packages on x86\_64: - - - native mode with a 32bit masterdir (recommended, used in official repository) - - cross compilation mode to i686 [target](#cross-compiling) - -The canonical mode (native) needs a new x86 `masterdir`: - - $ ./xbps-src -A i686 binary-bootstrap - $ ./xbps-src -A i686 ... - - -### Building packages natively for the musl C library - -The canonical way of building packages for same architecture but different C library is through a dedicated masterdir by using the host architecture flag `-A`. -To build for x86_64-musl on glibc x86_64 system, prepare a new masterdir with the musl packages: - - $ ./xbps-src -A x86_64-musl binary-bootstrap - -This will create and bootstrap a new masterdir called `masterdir-x86_64-musl` that will be used when `-A x86_64-musl` is specified. -Your new masterdir is now ready to build packages natively for the musl C library: - - $ ./xbps-src -A x86_64-musl pkg ... - - -### Building void base-system from scratch - -To rebuild all packages in `base-system` for your native architecture: - - $ ./xbps-src -N pkg base-system - -It's also possible to cross compile everything from scratch: - - $ ./xbps-src -a -N pkg base-system - -Once the build has finished, you can specify the path to the local repository to `void-mklive`, i.e: - - # cd void-mklive - # make - # ./mklive.sh ... -r /path/to/hostdir/binpkgs +Check the [Manual](./Manual.md) for details of how to create source packages.