Digital audio is so ubiquitous that we rarely stop to think or wonder how the gears turn underneath our all-pervasive apps for entertainment. Today we'll look at one specific piece of the machinery: latency.
Let's say you're making a video of someone's birthday party with an app on your phone. Once the recording starts, you don't care when the app starts writing it to disk
how long it takes for the video to reach the other end and vice versa. It's impossible to have a conversation if the lag (latency) is too high.
Other examples, in order of increasingly stricter latency requirements are: live video streaming, security cameras, augmented reality games such as
, multiplayer video games in general, audio effects apps for live music recording, and many many more.
“But Nirbheek”, you might ask, “why doesn't everyone always ‘immediately’ send/store/show whatever is recorded? Why do people have to worry about latency?” and that's a great question!
. It's also a good primer on analog vs digital audio!
Each operating system has its own set of application APIs for audio, and each has a lower bind on the achievable latency:
GStreamer already has plugins for almost all of these
¹ (plus others that aren't listed here), and on Windows, GStreamer has been using the DirectSound API by default for audio capture and output since the very beginning.
However, the DirectSound API was deprecated in Windows XP, and with Vista, it was removed and replaced with an emulation layer on top of the newly-released WASAPI. As a result, the plugin can't be configured to have less than 200ms of latency, which makes it unsuitable for all the low-latency use-cases mentioned above. The DirectSound API is quite crufty and unnecessarily complex anyway.
GStreamer is rarely used in video games, but it is widely used for live streaming, audio/video calls, and other real-time applications. Worse, the WASAPI GStreamer plugins were effectively untouched and unused since the initial implementation in 2008 and were completely broken
².
This left no way to achieve low-latency audio capture or playback on Windows using GStreamer.
The situation became particularly dire when GStreamer added a new
implementation of the WebRTC spec in this
release cycle. People that try it out on Windows were going to see much higher latencies than they should.
Luckily, I rewrote most of the WASAPI plugin code in January and February, and it should now work well on all versions of Windows from Vista to 10! You can get
binary installers for GStreamer or
build it from source.
Shared and Exclusive WASAPI
WASAPI allows applications to open sound devices in two modes:
shared and
exclusive. As the name suggests,
shared mode allows multiple applications to output to (or capture from) an audio device at the same time, whereas
exclusive mode does not.
Almost all applications should open audio devices in shared mode. It would be quite disastrous if your YouTube videos played without sound because Spotify decided to open your speakers in exclusive mode.
In shared mode, the audio engine has to resample and mix audio streams from all the applications that want to output to that device. This increases latency because it must maintain its own audio ringbuffer for doing all this, from which audio buffers will be periodically written out to the audio device.
In theory, hardware mixing could be used if the sound card supports it, but very few sound cards implement that now since it's so cheap to do in software. On Windows, only high-end audio interfaces used for professional audio implement this.
Another option is to allocate your audio engine buffers directly in the sound card's memory with DMA, but that complicates the implementation and relies on good drivers from hardware manufacturers. Microsoft has tried similar approaches in the past with DirectSound and been burned by it, so it's not a route they took with WASAPI
³.
On the other hand, some applications know they will be the only ones using a device, and for them all this machinery is a hindrance. This is why
exclusive mode exists. In this mode, if the audio driver is implemented correctly, the application's buffers will be directly written out to the sound card, which will yield the lowest possible latency.
Audio latency with WASAPI
So what kind of latencies
can we get with WASAPI?
That depends on the
device period that is being used. The term
device period is a fancy way of saying
buffer size; specifically the buffer size that is used in each call to your application that fetches audio data.
This is the same period with which audio data will be written out to the actual device, so it is the major contributor of latency in the entire machinery.
If you're using the
AudioClient interface in WASAPI to initialize your streams, the default period is 10ms. This means the theoretical
minimum latency you can get in
shared mode would be 10ms (audio engine) + 10ms (driver) = 20ms. In practice, it'll be somewhat higher due to various inefficiencies in the subsystem.
When using
exclusive mode, there's no engine latency, so the same number goes down to ~10ms.
These numbers are decent for most use-cases, but like I explained in my
previous blog post, this is totally insufficient for pro-audio use-cases such as applying live effects to music recordings. You really need latencies that are lower than 10ms there.
Ultra-low latency with WASAPI
Starting with Windows 10, WASAPI removed most of its aforementioned inefficiencies, and introduced a new interface:
AudioClient3. If you initialize your streams with this interface, and if your audio driver is implemented correctly, you can configure a device period of just
2.67ms at 48KHz.
The best part is that this is the period not just in exclusive mode but
also in shared mode, which brings WASAPI almost at-par with JACK and CoreAudio
So that was the good news. Did I mention there's bad news too? Well, now you know.
The first bit is that these numbers are only achievable if you use Microsoft's implementation of the Intel HD Audio standard for consumer drivers. This is fine; you follow
some badly-documented steps and it turns out fine.
Then you realize that if you want to use something more high-end than an Intel HD Audio sound card, unless you use
one of the rare pro-audio interfaces that have drivers that use the new
WaveRT driver model instead of the old
WaveCyclic model, you still see 10ms device periods.
It seems the pro-audio industry made the decision to stick with ASIO since it already provides <5ms latency. They don't care that the API is proprietary, and that most applications can't actually use it because of that. All the apps that are used in the pro-audio world already work with it.
The strange part is that all this information is nowhere on the Internet and seems to lie solely in the minds of the Windows audio driver cabals across the US and Europe. It's surprising and frustrating for someone used to working in the open to see such counterproductive information asymmetry, and
I'm not the only one.
This is where I plug open-source and talk about how Linux has had ultra-low latencies for years since all the audio drivers are open-source, follow the same
ALSA driver model⁴, and are constantly improved. JACK is probably the most well-known low-latency audio engine in existence, and was born on Linux. People are even using Pulseaudio these days to work with <5ms latencies.
But this blog post is about Windows and WASAPI, so let's get back on track.
To be fair, Microsoft is not to blame here. Decades ago they made the decision of not working more closely with the companies that write drivers for their standard hardware components, and they're still paying the price for it. Blue screens of death were the most user-visible consequences, but the current audio situation is an indication that losing control of your platform has more dire consequences.
There is one more bit of bad news. In my testing, I wasn't able to get glitch-free
capture of audio in the source element using the AudioClient3 interface at the minimum configurable latency in shared mode, even with
critical thread priorities unless there was nothing else running on the machine.
As a result, this feature is disabled by default on the source element. This is unfortunate, but not a great loss since the same device period is achievable in exclusive mode without glitches.
Measuring WASAPI latencies
Now that we're back from our detour, the executive summary is that the GStreamer WASAPI
source and
sink elements now use the latest recommended WASAPI interfaces. You should test them out and see how well they work for you!
By default, a device is opened in shared mode with a conservative latency setting. To force the stream into the lowest latency possible, set
low-latency=true. If you're on Windows 10 and want to force-enable/disable the use of the AudioClient3 interface, toggle the
use-audioclient3 property.
To open a device in exclusive mode, set
exclusive=true. This will ignore the
low-latency and
use-audioclient3 properties since they only apply to shared mode streams. When a device is opened in exclusive mode, the stream will always be configured for the lowest possible latency by WASAPI.
To measure the actual latency in each configuration, you can use the new
audiolatency plugin that I wrote to get hard numbers for the total end-to-end latency including the latency added by the GStreamer audio ringbuffers in the source and sink elements, the WASAPI audio engine (capture and render), the audio driver, and so on.
I look forward to hearing what your numbers are on Windows 7, 8.1, and 10 in all these configurations! ;)
1. The only ones missing are AAudio because it's very new and ASIO which is a proprietary API with licensing requirements.
2. It's no secret that although lots of people use GStreamer on Windows, the majority of GStreamer developers work on Linux and macOS. As a result the Windows plugins haven't always gotten a lot of love. It doesn't help that building GStreamer on Windows can be a daunting task . This is actually one of the major reasons why we're moving to Meson, but I've already written about that elsewhere!
3. My knowledge about the history of the decisions behind the Windows Audio API is spotty, so corrections and expansions on this are most welcome!
4. The ALSA drivers in the Linux kernel should not be confused with the ALSA userspace library.
