Fix the $@%* bug already (aka "the long-overdue mentoring posts")

I meant to post yesterday, but volunteering and cooking for Thanksgiving were higher priorities. Without further ado...

Recently I dove into the Ask Ubuntu pond, and as always it's wonderful seeing so many open source contributors assisting each other with goodwill. I deal in defect (aka "bug") reports mainly, and some of the Ask Ubuntu posts seem to be disguised as such reports. When I come upon these posts, I try to redirect the posters toward the debugging process so that we can resolve the underlying defects. Certain areas in the Ubuntu software stack are notoriously difficult to troubleshoot, and to that end I'm going to document a recent attempt to resolve the user's defect report so that we have some sort of "institutional knowledge transfer." Please feel free to raise pertinent questions and suggestions in the comments.

Here is a common symptom: "my laptop's speakers don't mute when I insert headphones." We call this class of bug "jack sense" ones, and it's nearly always a sound driver defect, so it belongs in the "alsa-driver (Ubuntu)" component in Launchpad. Because the defect has both hardware (sound controller and codec) and software (alsa-driver) portions, we must obtain sufficient information. For new defect reports, we request that the reporter use either the

ubuntu-bug alsa-base

command or follow a more thorough but tedious debugging procedure. For existing bug reports, we request that the reporter use the

apport-collect bugnumber

command or follow the script in the alternate debugging procedure.

Once we've received the requested debugging information, we need to inspect the sound codec slot's state (aka "codec dump"). Conexant parsers live in sound/pci/hda/patch_conexant.c in the [kernel] source code. Note the Vendor ID, because it refers to the specific function in the parser, so in this case we look at the patch_cxt5066() function. Of particular importance is the node ID associated with the headphone jack. This is referred to in the "codec dump" as having "[Jack] HP" in the description string. In this Conexant variant, we're looking thusly at NID 0x19.

Now, a note regarding the current Conexant parsers: there is no automatic BIOS-based setup, and in this way they diverge pretty spectacularly from the Realtek and IDT/Sigmatel ones; emerging Conexant ones are more like Realtek and IDT/Sigmatel. Because there isn't an "auto" model, all entries have to be hard-coded. In the case of patch_cxt5066(), the default model is CXT5066_LAPTOP ("laptop" model quirk) and uses

spec->init_verbs[0] = cxt5066_init_verbs;

to initialize the codec. cxt5066_init_verbs[] has this snippet corresponding to NID 0x19:

 /* not handling these yet */
 {0x19, AC_VERB_SET_UNSOLICITED_ENABLE, 0},

which means that to properly enable jack sense for the reporter's machine we need to add a function to handle intrinsic unsolicited responses, specifically Presence Detect. Luckily, a patch courtesy of David Henningsson landed earlier this summer adding this very functionality, so we only need to check that the proper NIDs are referenced, and upon confirmation then generate a one-line patch to enable jack sense for the bug reporter's machine.

You may notice astutely that I've worded the above process differently from how it appears chronologically in the bug report. The more one works with a particular kernel subsystem (or any software, really) the more readily one is aware how [often] bugs are resolved. In this case, sound driver development moves rapidly, so it's often in the reporter's interest to be using the latest code. Also, patch tagging and reviewing are important.

UDS-Maverick recap

UDS-M is bittersweet. It is the last summit I attend due to additional work commitments that will remove me from Ubuntu development, but more significantly it is the first summit where the future gardeners, maintainers, and developers of the audio infrastructure in Ubuntu and derivatives gathered and discussed how we can drive user experience improvements. In fact, the phenomenal Amber Graner asked me a few questions to that end, and a few things that I neglected to mention are that:

* Canonical has some real rock stars ramping up enablement and proactive quirking efforts on the ALSA front, so Linux distributions will see significant improvement in vendor support for audio in the 12.04 LTS time frame;

* we might see a streamlined method of switching between PulseAudio and JACK via indicator-sound on the Ubuntu desktop;

* Maverick will see the complete removal of native OSS support from our linux packages with OSS proxy being used instead.

As always, we'll continue to apply stable release updates to supported releases (particularly the most current LTS), to address high profile bug reports against the audio stack, and to maintain helpful presences on the varied mailing lists.

It has been one heck of a five-year ride. Thanks for all the fish.

Pre-Year of the Tiger

Firstly, lots happening on the Ubuntu 10.04 audio front:

* daily builds of stable alsa-driver snapshots (thanks, Brad!);
* massive alsa-lib and alsa-plugins debugging (thanks, David!);
* continued alsa-driver/linux quirking;
* uploads of latest stable alsa*, libsdl1.2 (including direct seeding of the pulse backend in the Ubuntu desktop seed), openal-soft, and pulseaudio source packages;
* re-addition of HDA power down for Sigmatel/IDT in pm-utils-powersave-policy 0.3.

Some very positive Canonical goings-on have occurred, too, but that's the purvey of others. Also importantly, fellow community members are stepping forward to help triage Ubuntu audio bugs. You gals/guys are too numerous to list individually, but rest assured that your efforts (even if you're new to bug triaging!) are much appreciated. Thanks for helping make Lucid rock!

Secondly, John Poelstra recently wrote a fantastic piece on chairing effective meetings. The points are so entirely lucid (bad pun I know) that I can't believe that I hadn't used them!

In fact, the revitalized DistrictOfColumbia LoCo team meetings have begun using them. Brian has been doing a bang-up job posting summaries in lieu of a configured mootbot to keep us accountable. Members of the LoCo team are working on a screencast/video to demonstrate archive-uploadable activities at jams using Bug Hugger, Lernid, and Ground Control. I'll be doing a portion on fixing alsa-driver/linux bugs.

Lucid development: audio for Alpha 1

Things that won't land for Lucid Alpha 1:

Steady progress on Analog Devices HDA suspend/resume additions/fixes in pursuit of powerdown objectives;

Fix for PulseAudio bailing on HDA softmodems.

On the other hand, they will land sometime soonish. In fact, the latter fix is available for testing in Karmic from the ubuntu-audio-dev PPA.

(As always, thanks for understanding that I have a limited amount of free time to do this stuff.)

Flailing against the light, or why bad things happen to good users

Seriously, Craig, I'm a bit disappointed that you didn't stop to ask members of this perceived conspiracy how to disable PulseAudio before flaming away, but as you say, it is most definitely your "right" to flame away. And, as Jordan mentioned in your blog post's comments, it was (and is) hardly MOTU's decision to tie PulseAudio more tightly into the GNOME desktop. That decision was upstream GNOME's. Not Ubuntu's. Not Fedora's. Not openSUSE's. Not Gentoo's. Not Debian's. Not Mandriva's.

In other words, it would take a non-trivial amount of work to remove the PulseAudio dependency from GNOME, and then Ubuntu would get to carry the delta *and* respond to defect reports.

Let's put this in perspective: there are two people working on audio in Ubuntu. One of them is employed by Canonical, and he isn't 100% time on audio. The other one cares enough to trawl through vitriol and drivel to respond to requests, demands, rants, and apathy -- on his lunch breaks and spare time -- to unbreak a resounding mess. If it's insufficiently clear, you're preaching to the choir, and your ultimatum is pathetically misdirected.

Now let's be constructive:

Firstly, PulseAudio *can* be made to go away in Karmic. I've written about it before, but I guess the naysayers are too busy flailing about how much PA sucks. Some caveats with the approach given: you'll need to use one of alsamixer, amixer (both command line tools shipped by default in the alsa-utils package), alsamixer-gui, aumix, and so on. You'll lose live per-application (per-stream) migration. You'll lose per-application volumes. And you'll lose other things, but they're hardly consequential enough to list here.

Secondly, the loss of system sounds is not a PulseAudio issue. It's a window manager issue. There's even a defect report about it.

Thirdly, PulseAudio *has* to use ALSA. PulseAudio is not a replacement for ALSA and never will be, because one of ALSA's primary functions is enable hardware. It's a sound driver. PulseAudio is not a sound driver; it's a program that uses sound drivers. If you're expecting PulseAudio to replace ALSA, go ahead and throw that idea out the window.

Fourthly, where are your defect reports for PulseAudio and ALSA in Karmic?

Lastly, no one is forcing you to upgrade to the latest Ubuntu hotness. Some people still use Dapper; others, Hardy. A great many people don't use Ubuntu at all. If you don't enjoy Ubuntu's direction, why not attend the Lucid developer summit and work with people there to resolve some issues?

Caveats for audio in 9.10

Here follows an update for 9.10 to this month's earlier caveats:

If you use Ubuntu:

• Use Karmic's kernel, not Jaunty's. A lot of you with Realtek and IDT/Sigmatel HDA codecs are being bitten by /boot/grub/grub.cfg not being updated properly, resulting in Jaunty's 2.6.28 kernel being used instead of Karmic's 2.6.31. Jaunty's kernel has pretty subpar performance for PulseAudio, and you should not incorrectly blame PulseAudio.
• Check that slmodemd is not running if you're seeing a dummy/null sink in the volume control applet. Arguably PulseAudio's module-udev-detect should allow the device instead of bailing when detecting it, and an approach is under discussion for future versions. In the meantime, you can either instead load module-detect in /etc/pulse/default.pa (or ~/.pulse/default.pa) or kill slmodemd.
• Install linux-backports-modules-alsa-karmic (then reboot) if you have a very new computer, because that package enables audio on quite a few very new laptop models and contains much improved support for microphone auto-switching.
• Use a temporary alternate channel mapping for PulseAudio if you have an ice17xx-based sound card. Please be aware that the fault lies with alsa-lib not PulseAudio and will be addressed in 10.04 LTS.
• Configure PulseAudio to ignore your sound driver's misreported dB information if you experience "overdriven" sound.
• Attach a verbose PulseAudio runtime log to your PulseAudio bug report.
• Attach an ALSA codec dump to your linux bug report if jack sense (e.g., connecting headphones mutes internal laptop speakers) does not seem to function properly.

As always, thank you for helping improve Ubuntu!

Caveats for audio in Ubuntu Karmic Beta

In the wake of the Ubuntu Karmic Beta release, here are some audio gotchas on my radar:

* Volumes being erroneously set to 0 and/or muted upon reboot/shutdown
** We'll address this in Lucid by altering how alsactl store is called. For Ubuntu, there's arguably no need to actually call alsactl restore; PulseAudio handles volume state on its own. Granted, the approach will take some fleshing out, as we need to handle Kubuntu, Xubuntu, Ubuntu Studio, ...

* Default PulseAudio behaviour causes overdriven audio due to hardware inanity
** Thanks to some wonderful (sarcasm) hardware integration, we can work around this issue by using the patch mentioned in the comment. Please note that this patch will not be turned on for Karmic, because doing so will result in a very poor user experience for many more HDA users whose hardware are not quite so craptastic.

* Internal/External microphone selection is broken for many HDA users
** Well-known upstream, being addressed for Lucid, patches too invasive for Karmic, etc.

* Extremely high CPU usage by PulseAudio with third-party applications (Skype, I'm looking at you)
** Use the PulseAudio-aware beta of Skype

Finally, last post I described populating the alsactl init database. There are a few things of which one should be aware:

* Too quiet or too loud is usually a driver (i.e., linux) bug
** We can work around this in alsa-kernel/linux, so the first place we check is linux, not alsa-utils. In the rare case it is not a driver bug, we resort to alsactl init patches.

* You can clone the upstream alsa-utils git tree and submit patches directly
** (Next post, I'll cover how to do it.)

Further down the rabbit hole

Craig, emotional bug reporting inevitably does a disservice to bug reporters and the people working to fix bugs alike. Reporters tend to omit necessary information; triagers and developers tend to ignore vitriolic (and often misled and/or uninformed) rants. Neither helps advance the state of FOSS. While I empathize with your (and nearly everyone else's) frustration with audio in Jaunty, there's no easy solution to the mess [PDF]. Many people blame Hardy's poor PulseAudio integration, and they are well placed to decry the missing changes to /etc/pulse/default.pa to allow most non-Free applications to play nicely. You specifically call out "functional" testing in the QA process, but are you actively involved in said QA process?

There are a shedload of shortcomings in Ubuntu related to audio. E.g., ubuntu-bug/apport could be taught to use alsa-info.sh (which isn't in Ubuntu Jaunty's alsa-driver source for various reasons but will be in Karmic's) behind the scenes; a standing FeatureFreeze exception similar to the GNOME desktop's could be requested and granted for ALSA-Lib and PulseAudio; linux's generic configuration could be "better" tweaked for desktop interactivity - and those are by no means comprehensive. However, each comes with an opportunity cost. Every single one of us needs to do a better job of working upstream, because throwing a tantrum in front of volunteers accomplishes little.

Call for testers

If you're as annoyed as I am with Jaunty's PulseAudio instability, behold, there is light at the end of the tunnel. If you're running 64-bit, please test the kernel at http://kernel.ubuntu.com/~dtchen/ and let me know how the stability of PulseAudio fares. If there is enough interest, I'll work on building 32-bit.

Edit: 32-bit is now available.

Fainting and the Linux audio stack, or Know Your Enemy

I've been working off and on to draft a Linux audio overview in the vein of Bryce's excellent X Window System architecture docs. Lennart, the brains behind Avahi and PulseAudio, already has a quite-good writeup on the different APIs with well thought-out reasons for choosing whichever one(s) for a task. Given that I've been knee-deep in triaging audio bugs, it has been amusing to watch various (developer) parties argue the merits of their wheels.

Nonetheless, in Linux, it's still a jungle [PNG].

Any user, power or novice, generally approaches bugs in an identical fashion: stuff needs to work now. Without further ado, here is part one of rationale behind guidelines for troubleshooting audio anomalies in Ubuntu (some of which is documented).

My sound is broken. Well, not quite. Generally when one's sound is inaudible, it can be caused by numerous culprits. Prior to Intrepid, one's user had to be in the "audio" group to use sound devices. PulseAudio's use of PolicyKit obviates that requirement, so one's user no longer has to be in that group. However, users experimenting with multiple desktop environments like Xfce, KDE, and GNOME - some of which don't use PulseAudio with a PolicyKit backend - need to be in that group. What am I driving at?

• The user has to know too much about traditional discretionary access control.

Thanks to PulseAudio's integration with PolicyKit - and here, classic abstraction at work - we no longer have to concern ourselves with questions regarding incorrect ownership and/or permissions on /dev/dsp*, /dev/snd/*, etc. Yet that is a common culprit for inaudible applications that we catch repeatedly using a community-maintained troubleshooting script.

Next time, we look at how Linux speaks to our audio hardware.

PulseAudio

If it seems fashionable to post about PulseAudio experiences, I'll refrain from spouting my own mishaps and instead focus on fixing and maintaining the audio stack in Ubuntu (a rather thankless job in any Linux distribution).

Firstly, people claim that PulseAudio brings a lot of fail to the Linux desktop environment. As with many maturing architectures, there are massive growing pains. David F is not the first to note that Jaunty's packages have thrown many desktop users for a loop. It should come as no surprise that there is more breakage on the horizon, and in fact, I documented it twice to, well, near silence on the ubuntu-devel-discuss mailing list. Much more importantly, however, PulseAudio has been instrumental in exposing numerous alsa-kernel (which in Ubuntu affects the linux source package, not the alsa-driver source package) and alsa-lib bugs. We all owe Takashi, Jaroslav, Lennart, and myriad others a round of $beverage for making things bearable^Waudible.

I'll take this opportunity to state again [OpenOffice.org Impress file] (as during my 2008 Ohio LinuxFest presentation) that Linux audio can be hairy, and if it has worked for us directly via ALSA, OSS (including v4.x), etc., we should be pleased but mindful that there is always room for improvement.

Secondly, Jaunty is in development. While it is reasonable to expect that filesystem-munching bugs don't hit us daily, it is also reasonable to expect that running a development branch brings experimental configurations that could munch our filesystems. The PulseAudio configuration currently in Jaunty was delivered precisely to trigger alsa-kernel and alsa-lib bugs so that they can be fixed. Surely we don't want another instance of Hardy's PulseAudio configuration to bite us!

Finally, it is much more productive to Ubuntu developers and maintainers for the Jaunty testing community to file bugs. In the case of Jaunty's PulseAudio, there already exists one "master" bug tracking the most common complaints of:

• my sound is inaudible on GNOME session login
• my sound disappears randomly while using GNOME
• music and/or video applications seem to hang or crash while using GNOME

Rest assured that we are working diligently to resolve the bug (which affects linux, alsa-lib, and pulseaudio).