OpenAI’s Realtime WebRTC API: A Guide to the Beta Implementation

This summary is based on my March 2025 webrtcHacks investigation, The Unofficial Guide to OpenAI’s (Beta) Realtime WebRTC API. I started the project because I wanted to repurpose an old Google AIY Voice Kit for ChatGPT voice mode. Connecting the microphone to OpenAI’s sample was easy. Understanding the full WebRTC and Realtime API flow beyond “say hello to the bot” was not.

The resulting article became a long, step-by-step guide with code, logs, diagrams, and a live example. This is the shorter version of what that investigation found. It describes the Beta implementation available when the original article was published.

The architecture was mostly standard WebRTC

The browser first used getUserMedia to capture a microphone track. That track was added to a new RTCPeerConnection, while the incoming track handler connected OpenAI’s generated audio to an HTML audio element. The browser also created a DataChannel named oai-events for Realtime API control messages.

This separation is important. WebRTC carried the actual audio, while the DataChannel carried JSON events that configured the model, started responses, reported transcripts, and invoked functions. From an application perspective, the DataChannel behaved a lot like a WebSocket. It simply traveled with the WebRTC session.

The connection used the usual offer/answer process. The browser created an SDP offer and sent it to OpenAI’s Beta Realtime endpoint. OpenAI returned an SDP answer, which the browser applied as the remote description. WebRTC then handled the audio transport, encryption, congestion control, and network details behind the RTCPeerConnection APIs.

Session behavior came from DataChannel events

Establishing the media connection was only the first part. The application still needed to tell the model how to behave.

When the DataChannel opened, the example sent a session.update message with the voice, instructions, input transcription settings, turn-detection behavior, and available functions. It could then send response.create to ask the model to begin speaking.

Incoming events described the state of the conversation. Some contained user or assistant transcripts. Others indicated that speech had started or stopped, an audio response had finished, a response was canceled, or a function call was ready. The guide walked through many of these events because the ordering and naming were a major part of building anything beyond the smallest demo.

Interruptions were split between WebRTC playback and Realtime API state. When the user spoke over the assistant, server-side voice activity detection could stop the audio response. The client still needed to pay attention to the corresponding events so its interface and conversation state stayed synchronized.

Function calling also traveled over the DataChannel

I added a simple end_session function so the user could verbally ask the application to stop. The function definition was included in the initial session configuration. When the model selected it, the client received a completed function-call event over the DataChannel.

The example then generated a final goodbye, waited for the output audio to finish, stopped the microphone track, and closed the peer connection. This exposed a practical detail that short samples often skip: a voice application needs an orderly shutdown path, especially if it wants the user to hear a final response before media capture ends.

The AIY Voice Kit became a browser device

My original plan was to implement everything in Python on the Raspberry Pi inside the old AIY Voice Kit. The archived hardware libraries and older system image made that awkward. Running the same browser application on the Pi was much simpler and kept the example portable.

The browser handled the WebRTC session, while a variation of the page connected the Voice Kit’s button and light to the conversation. The experiment also pointed toward smaller embedded implementations. A full browser is not required in principle—libraries such as Pion, aiortc, and embedded WebRTC stacks can establish the same kind of connection—but it was the fastest way to bring the old hardware back to life.

The full webrtcHacks guide includes the live example, complete event walkthrough, source code, security caveats, SDP details, and AIY Voice Kit demonstration.