## page was renamed from KernelMaintenance ## page was renamed from KernelMaintenance <> ||<>|| This is meant as a reference for the kernel team. == Getting the source == Ubuntu kernel source is available via git. More information can be obtained from KernelTeam/KernelGitGuide. == Build system breakdown == This requires some knowledge of the Debian build system (including the ''debian/rules'' file, etc). During gutsy, the build system was rewritten from the ground up, mainly to remove the use of ''kernel-package'' as a build tool, since it was getting too heavy and restrictive for our needs. We still use ''kernel-wedge'' for udeb creation. === Flavours === Flavours are how we define different types of kernels on the same architecture. Each flavour can be targeted to a specific CPU, or provide some functionality for a specific purpose. For example, the amd64 (x86_64) architecture has these flavours: * generic - The default install kernel. * server - Optimized for server installations. When building on the amd64 architecture, these are the kernels that are built. In reality, they only differ in slight variations of kernel config options. === Configs === Each flavour is built using a pre-defined configuration file (used by the kernel build system). Each architecture contains it's config files in ''debian./config//'' (or ''debian/config//'' in the case of Ubuntu Hardy). The files are split for each flavour in ''debian./config//config.'', and common options are shared in ''debian./config//config.common.''. When the kernels are built, the common and flavour specific config files are concatenated together in the build directory. === debian/control === The ''debian/control'' file is generated from ''debian/control.stub'' by the kernel-wedge command. This will be actioned automatically during a 'fakeroot debian/rules clean'. The ''debian/control.stub'' file is generated from ''debian/control.stub.in'' by a ''debian/rules'' target. This is populated by files defining the flavours in ''debian/control.d/vars.*''. The ''vars'' files define package descriptions for each flavour. === debian/changelog === The changelog file is where we track all changes made to the repository. This is a standard ''debian/changelog''. However, changes are not added to it manually. Instead, it is done automatically at release time using ''git log''. See below for process. === ABI === The ABI is probably one of the most important features of the build system. In the kernel package naming, the ABI is the numerical component immediately following the kernel version. E.g. {{{ 2.6.26-1-generic }}} The ABI in the above example is '''1'''. The ABI is represented in the ''/changelog'' as: {{{ linux (2.6.26-1.1) intrepid; urgency=low }}} The minor version after the ABI is an ever increasing revision. It basically denotes how many uploads have been made for this kernel version. The kernel ABI is derived from the ''Modules.symver'' file produced during the kernel build. Each exported function is given a hash defining the function's arguments and return value, making it easy to detect changes which can break modules built against the kernel (such as our own ''linux-restricted-modules'' package). The ABI files for the previous build are located in ''/abi//''. The layout is similar to the ''/config/'' directory in that each architecture is a subdirectory, and each flavour is a file with the ABI for that flavour in it. When the kernel build process finishes, it checks each flavour against the prev ABI. If the previous and current ABI's are the same version (e.g. 6), and the ABI has changed, the build will fail. This signifies that an ABI bump is needed (see below). As of now, a single ABI change in any flavour results in an ABI bump for the entire build. There is no per-flavour or per-architecture ABI number. This is for sanity reasons. ==== Overriding ABI check failures ==== When creating files below ensure the file has some content (a 1 is common) to ensure that the file is not lost when generating source packages when there is an orig.tar.Z. 1. '''No ABI check for all flavours and architectures''' Run the build with '''skipabi=true''' as an argument 1. '''No ABI check for all flavours of a specific architecture''' Create the file "''.../abi///ignore''" 1. '''No ABI check for one flavour of a specific architecture''' Create the file "''.../abi///.ignore''" 1. '''Ignore specific symbols for all flavours of all architectures in all versions''' Create the file "''.../abi/perm-blacklist''" and write each symbol to ignore into one line. If all symbols of a specific module should be ignored, then that line contains the module name prefixed by "''M: ''" (note the space between the colon and the module name). 1. '''Ignore specific symbols for all flavours of all architectures in one version''' Create a file "''.../abi//blacklist''" with the same semantics as the ''perm-blacklist'' file. === Module lists === Module lists are tracked similarly to the ABI, and are kept in ''/abi///.modules''. The file is a basename sorted list, sans .ko extension, of all the modules in a particular flavour. This is checked regardless of ABI, since it's meant to track release-to-release to avoid modules going missing by mistake. ==== Overriding module check failures ==== When creating files below ensure the file has some content (a 1 is common) to ensure that the file is not lost when generating source packages when there is an orig.tar.Z. 1. '''No module checks for all flavours and architectures''' Run the build with '''skipmodule=true''' as an argument 1. '''No module checks for all flavours of a specific architecture''' Create the file "''.../abi///ignore.modules''" 1. '''No module check for one flavour of a specific architecture''' Create the file "''.../abi///.ignore.modules''" 1. '''Ignore specific modules for all flavours of all architectures''' Create a file "''.../abi//modules.ignore''" and place each module to ignore into a separate line of that file. === GIT Hooks === On kernel.ubuntu.com there is a git repo called kteam-tools. It also contains some git hooks that we use to ensure policy compliance for commits to our tree. If you are doing a lot of work with the Ubuntu Linux kernel tree, you should put these scripts into the `.git/hooks/` directory of your working tree. === Commit templates === <> == Development cycle == This section describes all the steps involved in a common development cycle. That is, the cycles from one upload (release) of the Ubuntu kernel to the next. === Updating ABI and modules lists === To start a new release, you need to download the ABI and module files from the previous release. To do this, use: {{{ bash debian/scripts/misc/getabis 2.6.26 1.1 }}} In the above example, `2.6.26` is the kernel version, and `1.1` is the last uploaded revision. This will download all the previous deb's, and extract the ABI files, and module lists. You may find it easier to do this from a DC machine where the archive is closer (downloading all the kernel deb's is a bandwidth eater). === Starting a new release === Immediately after an upload you should run: {{{ debian/rules startnewrelease }}} This will generate a new ''debian/changelog'' entry for the next revision number, and modify appropriate other files under the ''debian/'' directory. === Add new code === Now you can rebase your tree, pull from other branches or apply patches that are going into this update. === Updating configs(optional) === As new drivers are added and merges are done with upstream, it becomes necessary to update our configs in ''debian./config/''. This is easily done for all architectures with one simple command: {{{ debian/rules updateconfigs }}} This runs '''silentoldconfig''' for all flavours, so if any possible config options are unanswered, it will ask you. Alternatively, if you know of a new config option, you can alleviate answering the same question for it on all flavours by appending it to the ''debian./config//config'' files, and running the above command. Once the configs are updated, you simply commit the new changes: {{{ git add debian./config/ git commit -s -e -F debian/commit-templates/update-configs }}} === Build === Now build the kernel for all archs and flavours === Bumping the ABI === If the build fails because of the ABI checker, this is a sign that the ABI needs to be bumped. If it becomes necessary to increase the ABI, you simply need to edit ''debian/changelog'' and increase the ABI in the version. For example, if the current changelog looks like: {{{ linux (2.6.26-1.1) UNRELEASED; urgency=low }}} And an ABI bump is required, you would change the above line to look like: {{{ linux (2.6.26-2.1) UNRELEASED; urgency=low }}} Once this is complete and the changelog is committed the following commands should be run to regenerate the control files: {{{ git clean -f -d fakeroot debian/rules clean }}} === Submitting Patches === Patches need to be ACK'd by other members of the Ubuntu kernel team before being pushed. <> Refer to https://wiki.ubuntu.com/KernelTeam/KernelPatches and https://wiki.ubuntu.com/KernelTeam/KernelGitGuide for some additional information. === Upstream syncing === During an Ubuntu development cycles, the kernel is constantly synced against upstream Linux GIT head. This involves merging conflicts, possibly deprecating local patches, and performing test builds. For stable releases, we cherry pick and sometimes outright sync to the 2.6.x.y GIT tree for that particular release. During development cycle, when we are following upstream git very closely, we will use ''git rebase'' to stay synced to upstream. Once upstream version has been released, we no longer rebase. This may cause times where you need to force a GIT pull (because the '''master''' of our tree is now rebased). === Third party drivers === During gutsy, feisty and hardy, third-party drivers were kept in a package called linux-ubuntu-modules. They are now kept in a subdirectory `ubuntu/` in the main kernel tree. Modules added by Ubuntu have to meet some basic criteria: * Modules added to the main tree must be open source. * Must be stable, and pass code review. * Must support hardware and/or features relevant to a good portion of users. It is preferable that these drivers are maintained upstream, and be scheduled for inclusion. In some cases, we keep drivers that do not meet these last criteria. However, this is mainly due to being grandfathered in and our policy of not losing support for hardware. === Preparing an upload === Once the tree is ready for upload, follow these steps to complete the package for uploading: * Run ''git status'' to be sure you have no outstanding commits, or extraneous files in your local tree. * It may also be a good idea to run ''git ls-files --others'' to check for extraneous files. * Create the changelog entries, with the following commands: {{{ debian/rules insertchanges git add debian/changelog git commit -s -m "UBUNTU: Ubuntu-2.6.20-6.11" }}} * Tag the release. The '''-m''' option is the message for the tag commit, and matches the actual tag name. {{{ git tag -s -m Ubuntu-2.6.20-6.11 Ubuntu-2.6.20-6.11 }}} * Push the changes and tag to our repo. * If the tree has a .orig.tar.gz in the repo, download that to the same directory as the git tree. * Ensure the tree is clean and the control files are populated: {{{ fakeroot debian/rules clean }}} * Run dpkg-buildpackage: {{{ dpkg-buildpackage -S -rfakeroot -I.git -I.gitignore -i'\.git.*' }}} or to build _with_ source use: {{{ dpkg-buildpackage -S -rfakeroot -I.git -I.gitignore -i'\.git.*' -sa }}} * Upload using whatever tools you wish (''dupload'' for example). * Submit your patch for upstream inclusion when appropriate, e.g., quirk updates. It should be noted that for development releases, a new tarball is generated at each upload. However at release time, a .orig.tar.gz is used, which is based on the upstream tarball of the released kernel we are using. === Announce Kernel uploads === <> === Pushing to our repo === The kernel team should push changes to our repo (see KernelGitGuide for public repo). For normal pushes, use the following command: {{{ git push [] master }}} For a release the release tag should be pushed as well well with: {{{ git push [] master }}} '''NOTE''': Each release tree is someone's responsibility. If it's not yours, it's probably best to email patches to kernel-team@lists.ubuntu.com rather than committing yourself. === Security and Proposed Updates === On occasion, security patches are applied to stable kernels. For these kernels, you will set the archive in ''debian/changelog'' to '''-security''', where '''''' is some release like ''intrepid''. The security team generally provides patches in the form of CVE's, with GIT SHA URL's to an upstream patch that fixes the problem. We usually use will fetch from the upstream GIT repo, and ''git cherry-pick'' to get the fix into the local tree. The easiest way to fetch the remote repo's objects is to create a remote file such as ''.git/remotes/upstream-linux-2.6.20'', containing: {{{ URL: git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.20.y.git Pull: master:upstream-linux-2.6.20 }}} Then to pull the objects, and cherry pick an SHA, do this: {{{ git fetch upstream-linux-2.6.20 git cherry-pick -s -e -x aa1812b3e344dbb2c7f30e1b95661cbd36d0cd2b }}} The '''-e''' option will bring up an editor to add any Ubuntu related info to the commit (perhaps a bug number, or maybe even add the CVE number). The '''-x''' will keep the original GIT SHA in the commit message to track where it came from. The '''-s''' option applies your Signed-off-by so that we can track who applied the commit. Security uploads should be build tested prior to upload (this is the case for any upload, but especially so for security and proposed updates). You and one other person should boot test the resulting kernel. If possible (especially for critical code paths), attempt to reproduce the bug via supplied reproducers (usually from the security team) to verify it actually fixes the bug. Make sure that there are no regressions as well. After upload, notify the security team that you have made the upload and include a copy of the changelog. == Performing builds == === Grab the Pre-requisite Tools === {{{ For Hardy and older: $ sudo apt-get install linux-kernel-devel devscripts fakeroot libncurses5-dev ccache For Intrepid and newer: $ sudo apt-get install build-essential devscripts fakeroot libncurses5-dev ccache $ sudo apt-get install kernel-wedge makedumpfile xmlto docbook-utils gs transfig sharutils $ sudo apt-get build-dep linux }}} A good way to ensure that you have all of the tools installed that are required to build a package is to use 'debuild -b'. If you are missing something, it will fail very quickly. === Pre-upload test builds === A set of scripts to help farm out builds across all supported architectures is described in https://wiki.ubuntu.com/KernelTeam/KernelBuildScripts === Quick builds === Aside from the standard ''dpkg-buildpackage'' build method, there are several easier ways to do build for developers. When I actually add new code, I generally do an out-of-tree build for test, like so: {{{ $ cd ubuntu-intrepid $ mkdir ../build $ cat debian./config/i386/config{,.generic} > ../build/.config $ make O=`pwd`/../build oldconfig $ make O=`pwd`/../build }}} You can add '''-jX''' as needed. One other helpful tool for a developer is to install the ''ccache'' package. This speeds rebuild by a large factor. === Normal full build === To test normal package building, one can use this command from the kernel git repo: {{{ fakeroot debian/rules binary-arch }}} This will build every kernel flavour. === Partial build === You can also do single or multiple specific flavours during the build, using this syntax: {{{ fakeroot debian/rules binary-generic }}} So you can change ''generic'' with any flavour you want to build. You may also want to do: {{{ fakeroot debian/rules binary-headers }}} to get the architecture independent headers package. LUM builds are done using: {{{ fakeroot debian/rules binary-modules- }}} where flavour = generic, ume, rt, xen, etc. === Build options === There are several build options, in the form of environment variables and make variables. Make options are passed on the ''debian/rules'' command line. * Environment * '''CONCURRENCY_LEVEL'''='''X''' where '''X''' is a number to pass using ''make's'' ''-j'' option. Use of this option is generally not recommended, as it will override the value normally selected automatically by the build rule (based on the number of processor cores). Setting the value to '''1''' can be useful to serialize the build in order to help isolate the source of a build or clean failure. * '''AUTOBUILD''' - (can be passed to make as well). Tells the build to create a non-official package with generated ABI. Used for daily builds of the kernel. * Make * '''skipabi=true''' - Used for test only, causes the build to skip the ABI consistency check. * '''skipmodule=true''' - Same as above, but for the module check. * '''NOKERNLOG''' - Used for the '''insertchanges''' or '''printchanges''' target. Causes git-ubuntu-log to not include upstream kernel changes in changelog output. * '''PRINTSHAS''' - Include GIT SHAs in '''printchanges''' and '''insertchanges''' targets. E.g. When trying to avoid "previous or current ABI file missing" and not wanting to do an ABI-bump: {{{ fakeroot debian/rules binary-generic binary-headers skipabi=true skipmodule=true }}} == Post upload == There are several things to do after an upload. These depend greatly on what the upload involved. Usually, for a non-ABI changing upload, you do not need to do anything further. However, for an ABI changing upload, packages need to be rebuilt against the linux-headers. Note that these packages have an extra version whose major matches the ABI of the kernel it is being built against. === linux-restricted-modules === This package contains modules that are proprietary. They are distributed separately so that it is easy for users to not have to use them. For an ABI bump, edit ''debian/rules'' and find '''abi_version'''. Change the value to the new ABI number. Use ''dpkg-buildpackage'' as normal after creating a new changelog entry. The linux-restricted-modules package was not stored in a git tree prior to Intrepid. To get it you would need to use ''apt-get source'' like: {{{ apt-get source linux-restricted-modules-2.6.24-11-generic }}} This will only work from an environment that has the apropriate series deb-src set for ''restricted'', like: {{{ deb-src http://archive.ubuntu.com restricted }}} With Intrepid LRM started to be a git tree on kernel.ubuntu.com and you can clone it to a working directory. If you got the sources, then look for the ''abi-version'' in debian/rules and make sure it corresponds to the abi of your new kernel and you have installed the header packages for all flavours. After that {{{fakeroot dpkg-buildpackage }}} will build the package. For Jaunty, you will need to follow the same instructions for bumping the ABI in the base source, i.e., updating debian/changelog then running: {{{ touch debian/rules.d/control.stub.in fakeroot debian/rules clean }}} === linux-ubuntu-modules === For Hardy there was a separate LUM git tree which was quite similar to LRM. Setting the ABI is the same as in linux-source (via ''debian/changelog''). A quick summary: to checkout, use {{{ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-hardy-lum.git ubuntu-hardy-lum }}} and to build, use {{{fakeroot debian/rules binary-modules-FLAVOUR}}} Please note that the alsa tree for Hardy is in linux-ubuntu-modules, so if you're having sound issues you might want to try building this package in addition to the main kernel package. Starting with Intrepid, additional modules are placed into the ubuntu directory of the main kernel repository. === linux-meta === The linux-meta package provides meta packages for easier user upgrades. Because our kernel packages are named with full ABI and flavour, users normally would have to manually upgrade to a new kernel (if it had an ABI bump). The meta packages always point to the latest kernel, so if users install these packages (and they are installed by default on new systems), then upgrades will always pull in everything they need to be current. For an ABI bump, edit ''debian/changelog'' and increment the ABI number and (since only an ABI bump requires a new linux-meta package) the upload number. === linux-backports-modules === This package is usually empty during development. It is meant for post-release updates of new drivers. It is packages separately from the kernel so that it is easier to provide sometimes untested drivers for new hardware on a stable release. Users do not have this package installed by default. This package is based heavily on ''linux-ubuntu-modules''. == main, proposed and security, OH MY! == We have some differing processes for different types of uploads. Described here for the sanity challenged. === main === No differences for this. Uploading to main (the current development release) is the primary target for work. === proposed === Technically known as `-proposed` (e.g. `edgy-proposed`) is where we can push changes that need testing, and are unsuited for a wide audience of users who expect a rock solid distribution. Kernel changes after release first get uploaded to -proposed and after some time will get moved to the -updates. See the KernelUpdates page for more information. === security === This is self explanatory, or so it would seem. Security patches will get applied to a temporary topic branch based on the last version in -updates. The resulting kernel package is uploaded to a special build environment and will get into the security and updates pocket. After upload the security patches will be merged back into the main tree. ---- CategoryKernel CategoryDocumentation