Fixing a Linux regression defect

This bug is an excellent example of the continual (aka "seemingly endless") process of verifying that hardware works correctly with Ubuntu (and more broadly, current versions of Linux). Briefly, in mid-May 2007, a patch was committed to upstream ALSA adding support for a particular machine. At the time that the patch was merged, support for certain BIOSes was quite suboptimal. Fast forward to late-November 2010, and a user's "sound no longer works," i.e., playback is inaudible through both speakers and headphones jack when using Ubuntu Maverick. In the realm of kernel code, particularly drivers, certain high-profile subsystems move relatively quickly, and given the vast amount of PC hardware, it's nearly impossible to verify that every released version of Ubuntu (much less upstream Linux) doesn't regress some hardware support. It's a difficult and thankless job, but someone has to do it.

Fix the $@%* bug already (aka "the long-overdue mentoring posts")

I meant to post yesterday, but volunteering and cooking for Thanksgiving were higher priorities. Without further ado...

Recently I dove into the Ask Ubuntu pond, and as always it's wonderful seeing so many open source contributors assisting each other with goodwill. I deal in defect (aka "bug") reports mainly, and some of the Ask Ubuntu posts seem to be disguised as such reports. When I come upon these posts, I try to redirect the posters toward the debugging process so that we can resolve the underlying defects. Certain areas in the Ubuntu software stack are notoriously difficult to troubleshoot, and to that end I'm going to document a recent attempt to resolve the user's defect report so that we have some sort of "institutional knowledge transfer." Please feel free to raise pertinent questions and suggestions in the comments.

Here is a common symptom: "my laptop's speakers don't mute when I insert headphones." We call this class of bug "jack sense" ones, and it's nearly always a sound driver defect, so it belongs in the "alsa-driver (Ubuntu)" component in Launchpad. Because the defect has both hardware (sound controller and codec) and software (alsa-driver) portions, we must obtain sufficient information. For new defect reports, we request that the reporter use either the

ubuntu-bug alsa-base

command or follow a more thorough but tedious debugging procedure. For existing bug reports, we request that the reporter use the

apport-collect bugnumber

command or follow the script in the alternate debugging procedure.

Once we've received the requested debugging information, we need to inspect the sound codec slot's state (aka "codec dump"). Conexant parsers live in sound/pci/hda/patch_conexant.c in the [kernel] source code. Note the Vendor ID, because it refers to the specific function in the parser, so in this case we look at the patch_cxt5066() function. Of particular importance is the node ID associated with the headphone jack. This is referred to in the "codec dump" as having "[Jack] HP" in the description string. In this Conexant variant, we're looking thusly at NID 0x19.

Now, a note regarding the current Conexant parsers: there is no automatic BIOS-based setup, and in this way they diverge pretty spectacularly from the Realtek and IDT/Sigmatel ones; emerging Conexant ones are more like Realtek and IDT/Sigmatel. Because there isn't an "auto" model, all entries have to be hard-coded. In the case of patch_cxt5066(), the default model is CXT5066_LAPTOP ("laptop" model quirk) and uses

spec->init_verbs[0] = cxt5066_init_verbs;

to initialize the codec. cxt5066_init_verbs[] has this snippet corresponding to NID 0x19:

 /* not handling these yet */
 {0x19, AC_VERB_SET_UNSOLICITED_ENABLE, 0},

which means that to properly enable jack sense for the reporter's machine we need to add a function to handle intrinsic unsolicited responses, specifically Presence Detect. Luckily, a patch courtesy of David Henningsson landed earlier this summer adding this very functionality, so we only need to check that the proper NIDs are referenced, and upon confirmation then generate a one-line patch to enable jack sense for the bug reporter's machine.

You may notice astutely that I've worded the above process differently from how it appears chronologically in the bug report. The more one works with a particular kernel subsystem (or any software, really) the more readily one is aware how [often] bugs are resolved. In this case, sound driver development moves rapidly, so it's often in the reporter's interest to be using the latest code. Also, patch tagging and reviewing are important.

UDS-Maverick recap

UDS-M is bittersweet. It is the last summit I attend due to additional work commitments that will remove me from Ubuntu development, but more significantly it is the first summit where the future gardeners, maintainers, and developers of the audio infrastructure in Ubuntu and derivatives gathered and discussed how we can drive user experience improvements. In fact, the phenomenal Amber Graner asked me a few questions to that end, and a few things that I neglected to mention are that:

* Canonical has some real rock stars ramping up enablement and proactive quirking efforts on the ALSA front, so Linux distributions will see significant improvement in vendor support for audio in the 12.04 LTS time frame;

* we might see a streamlined method of switching between PulseAudio and JACK via indicator-sound on the Ubuntu desktop;

* Maverick will see the complete removal of native OSS support from our linux packages with OSS proxy being used instead.

As always, we'll continue to apply stable release updates to supported releases (particularly the most current LTS), to address high profile bug reports against the audio stack, and to maintain helpful presences on the varied mailing lists.

It has been one heck of a five-year ride. Thanks for all the fish.

Public service announcement: When in doubt, don't splatter your apport info into an existing ALSA bug report

Folks -

Please, please don't apport-collect -p alsa-base foo (or apport-collect foo in 10.04 LTS) into a bug report that you didn't file unless a member of the audio dev team explicitly requests it. What we end up with are reports that start from the original reporter as "my laptop's internal mic doesn't work" and end with dozens of unrelated comments from other people along the lines of "my laptop's internal speakers are too loud".

Here's the gist: your symptoms may be eerily similar, but your audio hardware often is quite different, which means that resolving the original reporter's bug symptoms should only affect that reporter's hardware. Of course, if you have identical hardware, then your symptoms will be resolved in your own bug report, too. If you didn't file the original bug report, please just file a new one. Thanks for understanding.

Pre-Year of the Tiger

Firstly, lots happening on the Ubuntu 10.04 audio front:

* daily builds of stable alsa-driver snapshots (thanks, Brad!);
* massive alsa-lib and alsa-plugins debugging (thanks, David!);
* continued alsa-driver/linux quirking;
* uploads of latest stable alsa*, libsdl1.2 (including direct seeding of the pulse backend in the Ubuntu desktop seed), openal-soft, and pulseaudio source packages;
* re-addition of HDA power down for Sigmatel/IDT in pm-utils-powersave-policy 0.3.

Some very positive Canonical goings-on have occurred, too, but that's the purvey of others. Also importantly, fellow community members are stepping forward to help triage Ubuntu audio bugs. You gals/guys are too numerous to list individually, but rest assured that your efforts (even if you're new to bug triaging!) are much appreciated. Thanks for helping make Lucid rock!

Secondly, John Poelstra recently wrote a fantastic piece on chairing effective meetings. The points are so entirely lucid (bad pun I know) that I can't believe that I hadn't used them!

In fact, the revitalized DistrictOfColumbia LoCo team meetings have begun using them. Brian has been doing a bang-up job posting summaries in lieu of a configured mootbot to keep us accountable. Members of the LoCo team are working on a screencast/video to demonstrate archive-uploadable activities at jams using Bug Hugger, Lernid, and Ground Control. I'll be doing a portion on fixing alsa-driver/linux bugs.

Heads up: powerdown changes to alsa-base in Lucid/10.04

If you don't normally follow the Ubuntu development lists, it's a good time to begin. Yesterday I uploaded alsa-driver 1.0.22.1 to Lucid which, among other things, contains my extensions to power down various parts of the HDA controller and codec after a configurable idle period. Work is far from complete, and I could use your help to make sure your sound hardware is well-supported in 10.10.

In summary:
"Just to clarify in case it's not already obvious: my having uploaded alsa-driver 1.0.22.1 in no way affects the default Lucid install of the kernelspace ALSA driver (aka alsa-kernel). It remains 1.0.21. Anyone wishing to build modules of 1.0.22.1 can do so using module-assistant and alsa-source. And, until Takashi returns, that approach will be more current than using the daily snap from the 27 Dec.

So, the current state of Lucid contains:
alsa-kernel 1.0.21 (as shipped in linux 2.6.32-9.13-generic)
alsa-lib 1.0.22 + Fix-S24_3LE-softvol-distortion.patch (bdf80)
alsa-plugins 1.0.22"

Lucid development: audio for Alpha 1

Things that won't land for Lucid Alpha 1:

Steady progress on Analog Devices HDA suspend/resume additions/fixes in pursuit of powerdown objectives;

Fix for PulseAudio bailing on HDA softmodems.

On the other hand, they will land sometime soonish. In fact, the latter fix is available for testing in Karmic from the ubuntu-audio-dev PPA.

(As always, thanks for understanding that I have a limited amount of free time to do this stuff.)

How not to configure default settings for sound applications

You're absolutely right, Martin: recordmydesktop did the wrong thing and screwed your sound experience in Ubuntu. Fortunately, it's really straightforward to fix:

--- recordmydesktop-0.3.8.1.orig/src/rmd_types.h
+++ recordmydesktop-0.3.8.1/src/rmd_types.h
@@ -39,7 +39,7 @@
#ifdef HAVE_LIBASOUND
#include

- #define DEFAULT_AUDIO_DEVICE "hw:0,0"
+ #define DEFAULT_AUDIO_DEVICE "default"
#else
#include
#include

Flailing against the light, or why bad things happen to good users

Seriously, Craig, I'm a bit disappointed that you didn't stop to ask members of this perceived conspiracy how to disable PulseAudio before flaming away, but as you say, it is most definitely your "right" to flame away. And, as Jordan mentioned in your blog post's comments, it was (and is) hardly MOTU's decision to tie PulseAudio more tightly into the GNOME desktop. That decision was upstream GNOME's. Not Ubuntu's. Not Fedora's. Not openSUSE's. Not Gentoo's. Not Debian's. Not Mandriva's.

In other words, it would take a non-trivial amount of work to remove the PulseAudio dependency from GNOME, and then Ubuntu would get to carry the delta *and* respond to defect reports.

Let's put this in perspective: there are two people working on audio in Ubuntu. One of them is employed by Canonical, and he isn't 100% time on audio. The other one cares enough to trawl through vitriol and drivel to respond to requests, demands, rants, and apathy -- on his lunch breaks and spare time -- to unbreak a resounding mess. If it's insufficiently clear, you're preaching to the choir, and your ultimatum is pathetically misdirected.

Now let's be constructive:

Firstly, PulseAudio *can* be made to go away in Karmic. I've written about it before, but I guess the naysayers are too busy flailing about how much PA sucks. Some caveats with the approach given: you'll need to use one of alsamixer, amixer (both command line tools shipped by default in the alsa-utils package), alsamixer-gui, aumix, and so on. You'll lose live per-application (per-stream) migration. You'll lose per-application volumes. And you'll lose other things, but they're hardly consequential enough to list here.

Secondly, the loss of system sounds is not a PulseAudio issue. It's a window manager issue. There's even a defect report about it.

Thirdly, PulseAudio *has* to use ALSA. PulseAudio is not a replacement for ALSA and never will be, because one of ALSA's primary functions is enable hardware. It's a sound driver. PulseAudio is not a sound driver; it's a program that uses sound drivers. If you're expecting PulseAudio to replace ALSA, go ahead and throw that idea out the window.

Fourthly, where are your defect reports for PulseAudio and ALSA in Karmic?

Lastly, no one is forcing you to upgrade to the latest Ubuntu hotness. Some people still use Dapper; others, Hardy. A great many people don't use Ubuntu at all. If you don't enjoy Ubuntu's direction, why not attend the Lucid developer summit and work with people there to resolve some issues?

Caveats for audio in 9.10

Here follows an update for 9.10 to this month's earlier caveats:

If you use Ubuntu:

• Use Karmic's kernel, not Jaunty's. A lot of you with Realtek and IDT/Sigmatel HDA codecs are being bitten by /boot/grub/grub.cfg not being updated properly, resulting in Jaunty's 2.6.28 kernel being used instead of Karmic's 2.6.31. Jaunty's kernel has pretty subpar performance for PulseAudio, and you should not incorrectly blame PulseAudio.
• Check that slmodemd is not running if you're seeing a dummy/null sink in the volume control applet. Arguably PulseAudio's module-udev-detect should allow the device instead of bailing when detecting it, and an approach is under discussion for future versions. In the meantime, you can either instead load module-detect in /etc/pulse/default.pa (or ~/.pulse/default.pa) or kill slmodemd.
• Install linux-backports-modules-alsa-karmic (then reboot) if you have a very new computer, because that package enables audio on quite a few very new laptop models and contains much improved support for microphone auto-switching.
• Use a temporary alternate channel mapping for PulseAudio if you have an ice17xx-based sound card. Please be aware that the fault lies with alsa-lib not PulseAudio and will be addressed in 10.04 LTS.
• Configure PulseAudio to ignore your sound driver's misreported dB information if you experience "overdriven" sound.
• Attach a verbose PulseAudio runtime log to your PulseAudio bug report.
• Attach an ALSA codec dump to your linux bug report if jack sense (e.g., connecting headphones mutes internal laptop speakers) does not seem to function properly.

As always, thank you for helping improve Ubuntu!

The perils of saving sound card state; and why apologizing is good.

Bad news: I broke alsa-utils in Karmic.

Good news: I fixed it tonight. Please report your results.

Honestly, I tested the previous upload on Ubuntu, Kubuntu, and Xubuntu before committing to bzr -- even on different hardware! It just goes to show that I make dumb mistakes, so, sorry about that.

Mo' bett' news: many thanks to the awesome Tim Gardner, who imported alsa-driver stable (snapshot from 20091012) into ubuntu-karmic-lbm.git, which means that all you poor saps with eeePCs and HP dvs and Dell Studios will finally have less suck for audio. Granted, it will require installing linux-backports-modules-karmic in a week or so...

In other news (what?! there're more kinds?!), it really is decent to apologize when I make mistakes. This guy is a real gem, that is all.

Caveats for audio in Ubuntu Karmic Beta

In the wake of the Ubuntu Karmic Beta release, here are some audio gotchas on my radar:

* Volumes being erroneously set to 0 and/or muted upon reboot/shutdown
** We'll address this in Lucid by altering how alsactl store is called. For Ubuntu, there's arguably no need to actually call alsactl restore; PulseAudio handles volume state on its own. Granted, the approach will take some fleshing out, as we need to handle Kubuntu, Xubuntu, Ubuntu Studio, ...

* Default PulseAudio behaviour causes overdriven audio due to hardware inanity
** Thanks to some wonderful (sarcasm) hardware integration, we can work around this issue by using the patch mentioned in the comment. Please note that this patch will not be turned on for Karmic, because doing so will result in a very poor user experience for many more HDA users whose hardware are not quite so craptastic.

* Internal/External microphone selection is broken for many HDA users
** Well-known upstream, being addressed for Lucid, patches too invasive for Karmic, etc.

* Extremely high CPU usage by PulseAudio with third-party applications (Skype, I'm looking at you)
** Use the PulseAudio-aware beta of Skype

Finally, last post I described populating the alsactl init database. There are a few things of which one should be aware:

* Too quiet or too loud is usually a driver (i.e., linux) bug
** We can work around this in alsa-kernel/linux, so the first place we check is linux, not alsa-utils. In the rare case it is not a driver bug, we resort to alsactl init patches.

* You can clone the upstream alsa-utils git tree and submit patches directly
** (Next post, I'll cover how to do it.)

Limited liability, or how not to develop Karmic using Karmic

Karmic has been wonderfully jumpy these past few days. I must have some sick fascination with broken systems, as my presentation at this year's Ohio LinuxFest will focus on developing Karmic while running Karmic. Not just that, but this Saturday I'll briefly discuss Linux audio at Atlanta Linux Fest.

Now for the real meat in this post. For Karmic+1, I want Ubuntu to lead the charge in populating alsa-utils's init db.

Currently we do some pretty braindead things in alsa-utils's initscript, namely forcibly iterate over an entire set of static mixer element strings and values and attempt to set all detected audio devices to a specific state. Well, it's rather unlikely that my Logitech USB headset will have a Creative Audigy-specific "Audigy Analog/Digital Output Jack", yes?

Of course, that's just one badness, but the driving factor is to be able to conditionally set volumes on system start (for non-PulseAudio configurations) based on the machine's SSID. This approach largely resolves the "Gee, I have an Audigy X that requires the jack toggle to be muted to get audible analog output but person Y with an Audigy Z requires it to be unmuted" debacle.

So where do you come in? With current HDA codecs, using a static setting of 80% for PCM is pretty fragile. Some hardware sounds really overdriven at that setting, and others are barely audible. We need your SSID (lspci -nv|grep -A1 040[13]) and your preferred volumes! More in the following weeks.

More Karmic plans

The Jaunty development cycle was a hectic, pulsating (no pun intended) ride, and Karmic's looks to be just as vigourous! Here are my objectives for 9.10:

Audio:

Enable power-saving by default for HDA controllers - after fifteen seconds, power down the HDA controller to squeeze some juice for those transpacific flights. I'll be calling for testers shortly, but I'm most interested in resolving anomalies after the controller powers back up. You can already test this today if you're using karmic's kernel or a git snapshot of alsa-driver:
echo options snd-hda-intel power_save=15|sudo tee -a /etc/modprobe.d/test_power_save.conf

Offer snapshots of ALSA drivers - transition alsa-driver stable and unstable snapshots to use DKMS, and build daily debs in an audio-specific PPA (to be established)

Offer snapshots of PulseAudio - transition to weekly snapshots of git HEAD of upstream master, and build debs in above PPA

Streamline troubleshooting - offer GUI method for resetting stream volume and sink

General packaging:

Resume mentoring - already held two classroom tutorials. We need to use #ubuntu-classroom more effectively

Set periodic runs of piuparts/autopkgtest - determine which subset(s) of main and universe and make sense; enhance any existing build harnesses. See also where LDTP fits

Outreach:

Engage middle-schoolers, high-schoolers, itinerant workers in FOSS

Jaunty release party; more Karmic brainstorming

We're having a Jaunty release party in Washington, DC.

I'd also like thoughts on how to make resolving bugs like this one more transparent, err, easier, to the casual user. (Sure, I could have picked any one from thousands on Launchpad.) Matt Z has already contributed apport hooks to facilitate the process, but we have a long way to go still. (Please don't castigate or indulge in overzealousness in the comments.)

Further down the rabbit hole

Craig, emotional bug reporting inevitably does a disservice to bug reporters and the people working to fix bugs alike. Reporters tend to omit necessary information; triagers and developers tend to ignore vitriolic (and often misled and/or uninformed) rants. Neither helps advance the state of FOSS. While I empathize with your (and nearly everyone else's) frustration with audio in Jaunty, there's no easy solution to the mess [PDF]. Many people blame Hardy's poor PulseAudio integration, and they are well placed to decry the missing changes to /etc/pulse/default.pa to allow most non-Free applications to play nicely. You specifically call out "functional" testing in the QA process, but are you actively involved in said QA process?

There are a shedload of shortcomings in Ubuntu related to audio. E.g., ubuntu-bug/apport could be taught to use alsa-info.sh (which isn't in Ubuntu Jaunty's alsa-driver source for various reasons but will be in Karmic's) behind the scenes; a standing FeatureFreeze exception similar to the GNOME desktop's could be requested and granted for ALSA-Lib and PulseAudio; linux's generic configuration could be "better" tweaked for desktop interactivity - and those are by no means comprehensive. However, each comes with an opportunity cost. Every single one of us needs to do a better job of working upstream, because throwing a tantrum in front of volunteers accomplishes little.

Call for testers

If you're as annoyed as I am with Jaunty's PulseAudio instability, behold, there is light at the end of the tunnel. If you're running 64-bit, please test the kernel at http://kernel.ubuntu.com/~dtchen/ and let me know how the stability of PulseAudio fares. If there is enough interest, I'll work on building 32-bit.

Edit: 32-bit is now available.

Fainting and the Linux audio stack, or Know Your Enemy

I've been working off and on to draft a Linux audio overview in the vein of Bryce's excellent X Window System architecture docs. Lennart, the brains behind Avahi and PulseAudio, already has a quite-good writeup on the different APIs with well thought-out reasons for choosing whichever one(s) for a task. Given that I've been knee-deep in triaging audio bugs, it has been amusing to watch various (developer) parties argue the merits of their wheels.

Nonetheless, in Linux, it's still a jungle [PNG].

Any user, power or novice, generally approaches bugs in an identical fashion: stuff needs to work now. Without further ado, here is part one of rationale behind guidelines for troubleshooting audio anomalies in Ubuntu (some of which is documented).

My sound is broken. Well, not quite. Generally when one's sound is inaudible, it can be caused by numerous culprits. Prior to Intrepid, one's user had to be in the "audio" group to use sound devices. PulseAudio's use of PolicyKit obviates that requirement, so one's user no longer has to be in that group. However, users experimenting with multiple desktop environments like Xfce, KDE, and GNOME - some of which don't use PulseAudio with a PolicyKit backend - need to be in that group. What am I driving at?

• The user has to know too much about traditional discretionary access control.

Thanks to PulseAudio's integration with PolicyKit - and here, classic abstraction at work - we no longer have to concern ourselves with questions regarding incorrect ownership and/or permissions on /dev/dsp*, /dev/snd/*, etc. Yet that is a common culprit for inaudible applications that we catch repeatedly using a community-maintained troubleshooting script.

Next time, we look at how Linux speaks to our audio hardware.

PulseAudio

If it seems fashionable to post about PulseAudio experiences, I'll refrain from spouting my own mishaps and instead focus on fixing and maintaining the audio stack in Ubuntu (a rather thankless job in any Linux distribution).

Firstly, people claim that PulseAudio brings a lot of fail to the Linux desktop environment. As with many maturing architectures, there are massive growing pains. David F is not the first to note that Jaunty's packages have thrown many desktop users for a loop. It should come as no surprise that there is more breakage on the horizon, and in fact, I documented it twice to, well, near silence on the ubuntu-devel-discuss mailing list. Much more importantly, however, PulseAudio has been instrumental in exposing numerous alsa-kernel (which in Ubuntu affects the linux source package, not the alsa-driver source package) and alsa-lib bugs. We all owe Takashi, Jaroslav, Lennart, and myriad others a round of $beverage for making things bearable^Waudible.

I'll take this opportunity to state again [OpenOffice.org Impress file] (as during my 2008 Ohio LinuxFest presentation) that Linux audio can be hairy, and if it has worked for us directly via ALSA, OSS (including v4.x), etc., we should be pleased but mindful that there is always room for improvement.

Secondly, Jaunty is in development. While it is reasonable to expect that filesystem-munching bugs don't hit us daily, it is also reasonable to expect that running a development branch brings experimental configurations that could munch our filesystems. The PulseAudio configuration currently in Jaunty was delivered precisely to trigger alsa-kernel and alsa-lib bugs so that they can be fixed. Surely we don't want another instance of Hardy's PulseAudio configuration to bite us!

Finally, it is much more productive to Ubuntu developers and maintainers for the Jaunty testing community to file bugs. In the case of Jaunty's PulseAudio, there already exists one "master" bug tracking the most common complaints of:

• my sound is inaudible on GNOME session login
• my sound disappears randomly while using GNOME
• music and/or video applications seem to hang or crash while using GNOME

Rest assured that we are working diligently to resolve the bug (which affects linux, alsa-lib, and pulseaudio).