Bug Triage

Triaging kernel bugs is a day-to-day effort, and can be very time consuming. Luckily, we have a lot of community members willing and able to help with this effort. The kernel team will also begin using a set of Kernel Arsenal scripts to help with the day-to-day triaging efforts. In order to make sure everyone working on kernel bugs follows the same policy, this document will describe how to handle the kernel bug workflow. This will also provide bug reporters with an idea of the life cycle their bug will follow.

Note that beginning with the Karmic development cycle an emphasis is being made to ensure bugs are tested and reported upstream. In relation to this, the Ubuntu kernel team will be transitioning their focus to fixing bugs which have been confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel.

Also note that, as of the Maverick cycle, we will be breaking out our bugs into specific subsystems based on tags. For a list of current tags, please see the Tagging page. For specific information concerning the Bug Review process as it currently stands, please visit the Bug Review page. We have also further defined the levels of triage within our team. For an overview of that, please visit the Levels of Triage page.

New Bugs

Bugs should always come in with a Status of New and an Importance of Undecided. Bugs should automatically have the appropriate debug information attached assuming they were reported using the preferred ubuntu-bug linux method. If this is the case, move the bug to a Confirmed state. If the bug is missing the appropriate debug information, the submitter should be asked to run apport-collect. Note the apport-collect command below only applies to Karmic.

apport-collect -p linux <bug#>

In Jaunty, use

apport-collect -p linux-image-`uname -r` <bug #>

Set the bug to Incomplete. Also tag the bug "needs-kernel-logs". It's also a good idea to subscribe to a bug which you've set to Incomplete. That will ensure you are notified if and when the requested information is provided.

See process-new-bugs.py Kernel Arsenal script [to be linked soon]

Incomplete Bugs

Bugs are typically moved from a New state to an Incomplete state because they are lacking debug information necessary for the Ubuntu Kernel Team to debug the issue. Once all the information has been provided, the bug should be moved from an Incomplete state to a Confirmed state. If the bug was previously tagged "needs-kernel-logs", remove the tag as well once the bug moves to a Confirmed state.

If a bug is in an Incomplete state for more than 120 days and is not updated to provide the requested information, the bug should be expired by setting the status to Invalid and the reason it's being expired should be stated as a comment.

See process-incomplete-bugs.py Kernel Arsenal script [to be linked soon]

Confirmed Bugs

Confirmed bugs should have the appropriate debug information attached. In order for a Confirmed bug to move to a Triaged state, the upstream mainline kernel should be tested as well. This not only helps determine if the bug exists upstream, but also helps determine if a bug might be fixed upstream as well. If the bug exists upstream is also allow additional upstream developers to examine the issue. As mentioned previously, the Ubuntu kernel team will be focusing on bugs which have been confirmed to exist upstream or are fixed upstream but exist in the Ubuntu kernel. If a bug is in a Confirmed state but has not yet tested the upstream mainline kernel, tag the bug "needs-upstream-testing". If a bug has been tested with the upstream kernel, move the bug to a "Triaged" state. If the bug was previously tagged with "needs-upstream-testing", remove the tag once the bug moves to a Triaged state.

See process-confirmed-bugs.py Kernel Arsenal script [to be linked soon]

Triaged Bugs

Once a bug has been tested with the upstream mainline kernel and moved to a Triaged state, the bug should have their Importance set to something other than Undecided. If a bug is Triaged, then there should be enough information to know how important it is. If a bug was tested with the upstream mainline kernel and determined to exist upstream, a bug should also be reported upstream and an upstream bug watch should be set.

Note, you must be a member of the Ubuntu Bug Control team in order to set bugs to Triaged.

In Progress

A bug will move from Triaged to In Progress when a developer has chosen to actively work on the bug. If you are a developer and have set a bug to In Progress, make sure you also assign the bug to yourself. Please keep in mind that a bug marked In Progress for an extended period (greater than a week or two) should be updated periodically with any progress. If you are unable to dedicate time to working on the bug, move the bug back to a Triaged state and unassign yourself from the bug.

Also, do not assign another individual to a bug without their consent first! Doing so just gives bug reporters/subscribers a false sense that someone is actively working on a bug when that may not be the case. Leave it to the discretion of the developer to take ownership of a bug.

Fix Committed

Bugs that are marked Fix Committed are considered fixed by a patch which as been committed to the Ubuntu kernel git repository. This does not mean the fix has been released but it should be expected to be in the next kernel upload. There is no determinate time when a kernel upload will happen, it's up to the discretion of the Ubuntu Kernel Team.

Fix Released

Bugs are moved to a Fix Released state when the fix is readily available in the Ubuntu archive (in the updates pocket). If the Ubuntu kernel developer correctly created the git commit message to include the Launchpad Buglink, the launchpad janitor should automatically move a bug from Fix Committed to Fix Released once the fix is officially available.

Won't Fix

Won't Fix indicates the issue is recognized as a bug but a fix will not be applied, for ex BIOS issues. Some bugs may be marked as Won't Fix for a specific release and Fix Released against another.

Note, you must be a member of the Ubuntu Bug Control team in order to set bugs to Won't Fix.

Invalid

Bugs which are not legitimate bugs are set to Invalid. This can include bugs which failed to provide requested debug information or bugs which are the result of user error.

Caveats

Sometimes kernel bugs are opened to track security vulnerabilities. These bugs usually contain the word "CVE" either in the title or bug description and will most likely have the ubuntu-security team subscribed to the bug. We should try to avoid spamming these bugs with comments to test the latest kernel to verify if the issue still exists. Whether manually posting to a bug or using python-launchpad-bugs to script comments, please take extra care to not cause more unnecessary traffic for the security team to deal with. Thanks.

Kernel Sound Bugs

If this is a sound related Ubuntu kernel bug, run the following to gather and attach important sound debugging information to the bug:

$ apport-collect -p alsa-base <bug#>

Problems in capturing information

Bootloader

If the bug occurs during bootup, you can disable the splash screen in one of the following ways:

• Permanent: Edit /boot/grub/menu.lst and remove splash and quiet kernel parameters to bootloader from the entry for the buggy kernel

• Temporary: Press 'Escape' key at the 3 second pause by Grub bootloader. Then press 'e' (edit) on the buggy kernel entry, followed by 'e' again on the kernel line. Then remove the splash and quiet keywords and press 'b' to boot.

Capturing OOPs

If the bug report involves a crash, it is hoped that a kernel backtrace (aka OOPS, kernel panic) is available. If the machine does not completely lockup from the crash, the backtrace should be available in the dmesg output. If the crash completely locks the system:

• Take a look at https://wiki.ubuntu.com/KernelTeam/CrashdumpRecipe

• If using linux-crashdump (above) is not successful try and see if any backtrace was logged to /var/log/kern.log.0. Please attach this file if anything was captured.

• If unable to log the full backtrace, supply a digital photo of the screen to capture the crash. It most important to capture the beginning of the kernel oops or panic.

• When all fails, try to see if KernelTeam/Netconsole can help out.

In X window mode

Sometimes crashes occur in X, and so terminal access is not available (to capture the kernel backtrace). When this occurs, the user should try to recreate the crash at the console (Ctrl+Alt+F1). If this is not possible, then annotate the bug as such.

DIY Debugging Hints

For a complete list of debugging procedures refer to https://wiki.ubuntu.com/DebuggingProcedures

CategoryKernel CategoryBugSquad