What is WebRTC? Why does it need ‘Signalling’?

If you’re new to WebRTC, you must’ve heard that it’s a way to do video calls in your browser without needing to install an app. It’s pretty great!

However, it uses a bunch of really arcane terminology because it builds upon older technologies such as RTP, RTCP, SDP, ICE, STUN, etc. To understand what WebRTC Signalling is, you must first understand these foundational technologies.

Readers who are well-versed in this subject might find some of the explanations annoyingly simplistic to read. They will also notice that I am omitting a lot of of detail, leading to potentially misleading statements.

I apologize in advance to these people. I am merely trying to avoid turning this post into a book. If you find a sub-heading too simplistic, please feel free to skip it. :-)

RTP

Real-time Transport Protocol is a standardized way of taking video or audio data (media) and chopping it up into “packets” (you can literally think of them as packets / parcels) that are sent over the internet using UDP. The purpose is to try and deliver them to the destination as quickly as possible.

UDP (user datagram protocol) is a packet-based alternative to TCP (transmission control protocol), which is connection-based. So when you send something to a destination (IP address + port number), it will be delivered if possible but you have no protocol-level mechanism for finding out if it was received, unlike, say, TCP ACKs.

You can think of this as chucking parcels over a wall towards someone whom you can’t see or hear. A bunch of them will probably be lost, and you have no straightforward way to know how many were actually received.

UDP is used instead of TCP for a number of reasons, but the most important ones are:

1. TCP is designed for perfect delivery of all data, so networks will often try too hard to do that and use ridiculous amounts of buffering (sometimes 30 seconds or more!), which leads to latencies that are too large for two people to be able to talk over a call.

2. UDP doesn’t have that problem, but the trade-off is that it gives no guarantees of delivery at all!

   You’d be right to wonder why nothing new has been created to be a mid-way point between these two extremes. The reason is that new transport protocols don’t get any uptake because existing systems on the Internet (operating systems, routers, switches, etc) don’t (want to) support them. This is called Protocol ossification, and it's a big problem for the Internet.

   Due to this, new protocols are just built on top of UDP and try to add mechanisms to detect packet loss and such. One such mechanism is…

RTCP

RTP Control Protocol refers to standardized messages (closely related to RTP) that are sent by a media sender to all receivers, and also messages that are sent back by the receiver to the sender (feedback). As you might imagine, this message-passing system has been extended to do a lot of things, but the most important are:

1. Receivers use this to send feedback to the sender about how many packets were actually received, what the latency was, etc.
2. Senders send information about the stream to receivers using this, for instance to synchronize audio and video streams (also known as lipsync), to tell receivers that the stream has ended (a BYE message), etc.

Similar to RTP, these messages are also sent over UDP. You might ask “what if these are lost too”? Good question!

RTCP packets are sent at regular intervals, so you’d know if you missed one, and network routers and switches will prioritize RTCP packets over other data, so you’re unlikely to lose too many in a row unless there was a complete loss of connectivity.

Peer

WebRTC is often called a “peer-to-peer” (P2P) protocol. You might’ve heard that phrase in a different context: P2P file transfer, such as Bittorrent.

The word “peer” contrasts with “server-client” architectures, in which “client” computers can only talk to (or via) “server” computers, not directly to each other.

We can contrast server-client architecture with peer-to-peer using a real-world example:

• If you send a letter to your friend using a postal service, that’s a server-client architecture.
• If you leave the letter in your friend’s mailbox yourself, that’s peer-to-peer.

But what if you don’t know what kind of messages the recipient can receive or understand? For that we have…

SDP

Stands for Session Description Protocol which is a standardized message format to tell the other side the following:

• Whether you want to send and/or receive, audio and/or video
• How many streams of audio and/or video you want to send / receive
• What formats you can send or receive, for audio and/or video

This is called an “offer”. Then the other peer uses the same message format to reply with the same information, which is called an “answer”.

This constitutes media “negotiation”, also called “SDP exchange”. One side sends an “offer” SDP, the other side replies with an “answer” SDP, and now both sides know what to do.

As you might expect, there’s a bunch of other technical details here, and you can know all about them at this excellent page that explains every little detail. It even explains the format for ICE messages! Which is…

ICE

Interactive Connectivity Establishment is a standardized mechanism for peers to tell each other how to transmit and receive UDP packets. The simplest way to think of it is that it’s just a list of IP address and port pairs.

Once both sides have successfully sent each other (“exchanged”) ICE messages, both sides know how to send RTP and RTCP packets to each other.

Why do we need IP address + port pairs to know how to send and receive packets? For that you need to understand…

How The Internet Works

If you’re connected to the internet, you always have an IP address. That’s usually something like 192.168.1.150 – a private address that is specific to your local (home) network and has no meaning outside of that. Having someone’s private IP address is basically like having just their house number but no other parts of their address, like the street or the city. Useful if you're living in the same building, but not otherwise.

Most personal devices (computer or phone or whatever) with access to the Internet don’t actually have a public IP address. Picking up the analogy from earlier, a public IP address is the internet equivalent of a full address with a house number, street address, pin code, country.

When you want to connect to (visit) a website, your device actually talks to an ISP (internet service provider) router, which will then talk to the web server on your behalf and ask it for the data (website in this case) that you requested. This process of packet-hopping is called “routing” of network packets.

This ISP router with a public address is called a NAT (Network Address Translator). Like the name suggests, its job is to translate the addresses embedded in packets sent to it from public to private and vice-versa.

Let’s say you want to send a UDP packet to www.google.com. Your browser will resolve that domain to an IP address, say 142.250.205.228. Next, it needs a port to send that packet to, and both sides have to pre-agree on that port. Let’s pick 16789 for now.

Your device will then allocate a port on your device from which to send this packet, let’s say 11111. So the packet header looks a bit like this:

From	To
192.168.1.150:11111	142.250.205.228:16789

Your ISP’s NAT will intercept this packet, and it will replace your private address and port in the From field in the packet header to its own public address, say 169.13.42.111, and it will allocate a new sender port, say 22222:

From	To
169.13.42.111:22222	142.250.205.228:16789

Due to this, the web server never sees your private address, and all it can see is the public address of the NAT.

When the server wants to reply, it can send data back to the From address, and it can use the same port that it received the packet on:

From	To
142.250.205.228:16789	169.13.42.111:22222

The NAT remembers that this port 22222 was recently used for your From address, and it will do the reverse of what it did before:

From	To
142.250.205.228:16789	192.168.1.150:11111

And that’s how packets are send and received by your phone, computer, tablet, whatever when talking to a server.

Since at least one side needs to have a public IP address for this to work, how can your phone send messages to your friend’s phone? Both only have private addresses.

Solution 1: Just Use A Server As A Relay

The simplest solution is to have a server in the middle that relays your messages. This is how all text messaging apps such as iMessage, WhatsApp, Instagram, Telegram, etc work.

You will need to buy a server with a public address, but that’s relatively cheap if you want to send small messages.

For sending RTP (video and audio) this is accomplished with a TURN (Traversal Using Relays around NAT) server.

Bandwidth can get expensive very quickly, so you don’t want to always use a TURN server. But this is a fool-proof method to transmit data, so it’s used a backup.

Solution 2: STUN The NAT Into Doing What You Want

STUN stands for “Simple Traversal of UDP through NATs”, and it works due to a fun trick we can do with most NATs.

Previously we saw how the NAT will remember the mapping between a “port on its public address” and “your device’s private address and port”. With many NATs, this actually works for any packet sent on that public port by anyone.

This means if a public server can be used to create such mappings on the NATs of both peers, then the two can send messages to each other from NAT-to-NAT without a relay server!

Let’s dig into this, and let’s substitute hard-to-follow IP addresses with simple names: AlicePhone, AliceNAT, BobPhone, BobNAT, and finally STUNServer:19302.

First, AlicePhone follows this sequence:

1. AlicePhone sends a STUN packet intended for STUNServer:19302 using UDP

   From	To
   AlicePhone:11111	STUNServer:19302

2. AliceNAT will intercept this and convert it to:

   From	To
   AliceNAT:22222	STUNServer:19302

3. When STUNServer receives this packet, it will know that if someone wants to send a packet to AlicePhone:11111, they could use AliceNAT:22222 as the To address. This is an example of an ICE candidate.

4. STUNServer will then send a packet back to AlicePhone with this information.

Next, BobPhone does the same sequence and discovers that if someone wants to send a packet to BobPhone:33333 they can use BobNAT:44444 as the To address. This is BobPhone’s ICE candidate.

Now, AlicePhone and BobPhone must exchange these ICE candidates.

How do they do this? They have no idea how to talk to each other yet.

The answer is… they Just Use A Server As A Relay! The server used for this purpose is called a Signalling Server.

Note that these called “candidates” because this mechanism won’t work if one of the two NATs changes the public port also based on the public To address, not just the private From address. This is called a Symmetric NAT, and in these (and other) cases, you have to fallback to TURN.

Signalling Server

Signalling is a technical term that simply means: “a way to pass small messages between peers”. In this case, it’s a way for peers to exchange SDP and ICE candidates.

Once these small messages have been exchanged, the peers know how to send data to each other over the internet without needing a relay.

Now open your mind: you could use literally any out of band-mechanism for this. You can use Amazon Kinesis Video Signalling Channels. You can use a custom websocket server or a ProtoBuf server.

Heck, Alice and Bob can copy/paste these messages into iMessage on both ends. In theory, you can even use carrier pigeons — it’ll just take a very long time to exchange messages 😉

That’s it, this is what Signalling means in a WebRTC context, and why it’s necessary for a successful connection!

What a Signalling Server gives you on top of this is state management: checking whether a peer is allowed to send messages to another peer, whether a peer is allowed to join a call, can be invited to a call, which peers are in a call right now, etc.

Based on your use-case, this part can be really easy to implement or really difficult and heavy in corner-cases. Most people can get away with a really simple protocol, just by adding authorization to this multi-party protocol I wrote for the GStreamer WebRTC multiparty send-receive examples. More complex setups require a more bespoke solution, where all peers aren’t equal.

GStreamer has grown a WebRTC implementation

In other news, GStreamer is now almost buzzword-compliant! The next blog post on our list: blockchains and smart contracts in GStreamer.

Late last year, we at Centricular announced a new implementation of WebRTC in GStreamer.  Today we're happy to announce that after community review, that work has been merged into GStreamer itself! The plugin is called webrtcbin, and the library is, naturally, called gstwebrtc.

The implementation has all the basic features, is transparently compatible with other WebRTC stacks (particularly in browsers), and has been well-tested with both Firefox and Chrome.

Some of the more advanced features such as FEC are already a work in progress, and others will be too—if you want them to be! Hop onto IRC on #gstreamer @ Freenode.net or join the mailing list.

How do I use it?

Currently, the easiest way to use webrtcbin is to build GStreamer using either gst-uninstalled (Linux and macOS) or Cerbero (Windows, iOS, Android). If you're a patient person, you can follow @gstreamer and wait for GStreamer 1.14 to be released which will include Windows, macOS, iOS, and Android binaries.

The API currently lacks documentation, so the best way to learn it is to dive into the source-tree examples. Help on this will be most appreciated! To see how to use GStreamer to do WebRTC with a browser, checkout the bidirectional audio-video demos.

Show me the code! [skip]

Here's a quick highlight of the important bits that should get you started if you already know how GStreamer works. This example is in C, but GStreamer also has bindings for Rust, Python, Java, C#, Vala, and so on.

Let's say you want to capture video from V4L2, stream it to a webrtc peer, and receive video back from it. The first step is the streaming pipeline, which will look something like this:

v4l2src ! queue ! vp8enc ! rtpvp8pay !
    application/x-rtp,media=video,encoding-name=VP8,payload=96 ! 
    webrtcbin name=sendrecv

As a short-cut, let's parse the string description to create the pipeline.

1 2 3 4 5

GstElement *pipe; pipe = gst_parse_launch ("v4l2src ! queue ! vp8enc ! rtpvp8pay ! " "application/x-rtp,media=video,encoding-name=VP8,payload=96 !" " webrtcbin name=sendrecv", NULL);

Next, we get a reference to the webrtcbin element and attach some callbacks to it.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

GstElement *webrtc; webrtc = gst_bin_get_by_name (GST_BIN (pipe), "sendrecv"); g_assert (webrtc != NULL); /* This is the gstwebrtc entry point where we create the offer. * It will be called when the pipeline goes to PLAYING. */ g_signal_connect (webrtc, "on-negotiation-needed", G_CALLBACK (on_negotiation_needed), NULL); /* We will transmit this ICE candidate to the remote using some * signalling. Incoming ICE candidates from the remote need to be * added by us too. */ g_signal_connect (webrtc, "on-ice-candidate", G_CALLBACK (send_ice_candidate_message), NULL); /* Incoming streams will be exposed via this signal */ g_signal_connect (webrtc, "pad-added", G_CALLBACK (on_incoming_stream), pipe); /* Lifetime is the same as the pipeline itself */ gst_object_unref (webrtc);

When the pipeline goes to PLAYING, the on_negotiation_needed() callback will be called, and we will ask webrtcbin to create an offer which will match the pipeline above.

1 2 3 4 5 6 7 8 9 10

static void on_negotiation_needed (GstElement * webrtc, gpointer user_data) { GstPromise *promise; promise = gst_promise_new_with_change_func (on_offer_created, user_data, NULL); g_signal_emit_by_name (webrtc, "create-offer", NULL, promise); }

When webrtcbin has created the offer, it will call on_offer_created()

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

static void on_offer_created (GstPromise * promise, GstElement * webrtc) { GstWebRTCSessionDescription *offer = NULL; const GstStructure *reply; gchar *desc; reply = gst_promise_get_reply (promise); gst_structure_get (reply, "offer", GST_TYPE_WEBRTC_SESSION_DESCRIPTION, &offer, NULL); gst_promise_unref (promise); /* We can edit this offer before setting and sending */ g_signal_emit_by_name (webrtc, "set-local-description", offer, NULL); /* Implement this and send offer to peer using signalling */ send_sdp_offer (offer); gst_webrtc_session_description_free (offer); }

Similarly, when we have the SDP answer from the remote, we must call "set-remote-description" on webrtcbin.

1 2 3 4 5 6 7

answer = gst_webrtc_session_description_new ( GST_WEBRTC_SDP_TYPE_ANSWER, sdp); g_assert (answer); /* Set remote description on our pipeline */ g_signal_emit_by_name (webrtc, "set-remote-description", answer, NULL);

ICE handling is very similar; when the "on-ice-candidate" signal is emitted, we get a local ICE candidate which we must send to the remote. When we have an ICE candidate from the remote, we must call "add-ice-candidate" on webrtcbin.

There's just one piece left now; handling incoming streams that are sent by the remote. For that, we have on_incoming_stream() attached to the "pad-added" signal on webrtcbin.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

static void on_incoming_stream (GstElement * webrtc, GstPad * pad, GstElement * pipe) { GstElement *play; play = gst_parse_bin_from_description ( "queue ! vp8dec ! videoconvert ! autovideosink", TRUE, NULL); gst_bin_add (GST_BIN (pipe), play); /* Start displaying video */ gst_element_sync_state_with_parent (play); gst_element_link (webrtc, play); }

That's it! This is what a basic webrtc workflow looks like. Those of you that have used the PeerConnection API before will be happy to see that this maps to that quite closely.

The aforementioned demos also include a Websocket signalling server and JS browser components, and I will be doing an in-depth application newbie developer's guide at a later time, so you can follow me @nirbheek to hear when it comes out!

Tell me more!

The code is already being used in production in a number of places, such as EasyMile's autonomous vehicles, and we're excited to see where else the community can take it.

If you're wondering why we decided a new implementation was needed, read on! For a more detailed discussion into that, you should watch Matthew Waters' talk from the GStreamer conference last year. It's a great companion for this article!

But before we can dig into details, we need to lay some foundations first.

What is GStreamer, and what is WebRTC? [skip]

GStreamer is a cross-platform open-source multimedia framework that is, in my opinion, the easiest and most flexible way to implement any application that needs to play, record, or transform media-like data across an extremely versatile scale of devices and products. Embedded (IoT, IVI, phones, TVs, …), desktop (video/music players, video recording, non-linear editing, videoconferencing and VoIP clients, browsers …), to servers (encode/transcode farms, video/voice conferencing servers, …) and more.

But what I like the most about GStreamer is the pipeline-based model which solves one of the hardest problems in API design: catering to applications of varying complexity; from the simplest one-liners and quick solutions to those that need several hundreds of thousands of lines of code to implement their full featureset. 

If you want to learn more about GStreamer, Jan Schmidt's tutorial from Linux.conf.au is a good start.

WebRTC is a set of draft specifications that build upon existing RTP, RTCP, SDP, DTLS, ICE (and many other) real-time communication specifications and defines an API for making RTC accessible using browser JS APIs.

People have been doing real-time communication over IP for decades with the previously-listed protocols that WebRTC builds upon. The real innovation of WebRTC was creating a bridge between native applications and webapps by defining a standard, yet flexible, API that browsers can expose to untrusted JavaScript code.

These specifications are constantly being improved upon, which combined with the ubiquitous nature of browsers means WebRTC is fast becoming the standard choice for videoconferencing on all platforms and for most applications.

Everything is great, let's build amazing apps! [skip]

Not so fast, there's more to the story! For WebApps, the PeerConnection API is everywhere. There are some browser-specific quirks as usual, and the API itself keeps changing, but the WebRTC JS adapter handles most of that. Overall the WebApp experience is mostly 👍.

Sadly, for native code or applications that need more flexibility than a sandboxed JS app can achieve, there haven't been a lot of great options.

libwebrtc (Chrome's implementation), Janus, Kurento, and OpenWebRTC have traditionally been the main contenders, but after having worked with all of these, we found that each implementation has its own inflexibilities, shortcomings, and constraints.

libwebrtc is still the most mature implementation, but it is also the most difficult to work with. Since it's embedded inside Chrome, it's a moving target, the API can be hard to work with, and the project is quite difficult to build and integrate, all of which are obstacles in the way of native or server app developers trying to quickly prototype and try out things.

It was also not built for multimedia use-cases, so while the webrtc bits are great, the lower layers get in the way of non-browser use-cases and applications. It is quite painful to do anything other than the default "set raw media, transmit" and "receive from remote, get raw media". This means that if you want to use your own filters, or hardware-specific codecs or sinks/sources, you end up having to fork libwebrtc.

In contrast, as shown above, our implementation gives you full control over this as with any other GStreamer pipeline.

OpenWebRTC by Ericsson was the first attempt to rectify this situation, and it was built on top of GStreamer. The target audience was app developers, and it fit the bill quite well as a proof-of-concept—even though it used a custom API and some of the architectural decisions made it quite inflexible for most other use-cases.

However, after an initial flurry of activity around the project, momentum petered out, the project failed to gather a community around itself, and is now effectively dead.

Full disclosure: we worked with Ericsson to polish some of the rough edges around the project immediately prior to its public release.

WebRTC in GStreamer — webrtcbin and gstwebrtc

Remember how I said the WebRTC standards build upon existing standards and protocols? As it so happens, GStreamer has supported almost all of them for a while now because they were being used for real-time communication, live streaming, and in many other IP-based applications. Indeed, that's partly why Ericsson chose it as the base for OWRTC.

This combined with the SRTP and DTLS plugins that were written during OWRTC's development meant that our implementation is built upon a solid and well-tested base, and that implementing WebRTC features is not as difficult as one might presume. However, WebRTC is a large collection of standards, and reaching feature-parity with libwebrtc is an ongoing task.

Lucky for us, Matthew made some excellent decisions while architecting the internals of webrtcbin, and we follow the PeerConnection specification quite closely, so almost all the missing features involve writing code that would plug into clearly-defined sockets.

We believe what we've been building here is the most flexible, versatile, and easy to use WebRTC implementation out there, and it can only get better as time goes by. Bringing the power of pipeline-based multimedia manipulation to WebRTC opens new doors for interesting, unique, and highly efficient applications.

To demonstrate this, in the near future we will be publishing articles that dive into how to use the PeerConnection-inspired API exposed by webrtcbin to build various kinds of applications—starting with a CPU-efficient multi-party bidirectional conferencing solution with a mesh topology that can work with any webrtc stack.

Until next time!