How OpenAI Built Its Realtime WebRTC Gateway

This highlights post is based on my April 2025 webrtcHacks article and interview, OpenAI & WebRTC Q&A with Sean DuBois.

OpenAI’s Realtime API was one of the largest new WebRTC use cases to appear in a long time. Sean DuBois—the founder of the Pion WebRTC project and an OpenAI engineer who worked on the implementation—agreed to explain how it was built. The original includes the replay, an annotated transcript, and my own notes.

The full conversation covered plenty of WebRTC and Pion history. These are the major OpenAI architecture points.

WebRTC and WebSockets served different inputs

At the time of the interview, OpenAI had several real-time media paths. ChatGPT Advanced Voice used LiveKit. The Realtime API began as a WebSocket API, leaving audio capture and playback to the developer. The newer WebRTC interface added familiar browser media handling while keeping the same Realtime API event model.

Sean initially found the WebSocket approach cumbersome for a live conversation because the developer had to build more of the audio path. He also explained a case it handled well: processing recorded audio faster than real time. A service validating a drive-through order, for example, might want to upload and analyze audio as quickly as possible instead of playing it through a real-time media clock.

For a live browser conversation, WebRTC removed much of that work. Media flowed over the peer connection and API messages used the DataChannel.

The WebRTC service was a gateway

The existing Realtime WebSocket service already contained the application logic for API keys, billing, safety checks, session limits, recording, and access to the inference systems. Rebuilding all of that for a new media server would have delayed the WebRTC launch.

Instead, Sean built a Pion-based gateway. It accepted a WebRTC connection, translated the media into the form expected by the WebSocket service, and forwarded DataChannel messages into the existing event path. To the developer, the WebRTC and WebSocket versions remained conceptually parallel.

Running WebRTC inside infrastructure designed around HTTP and WebSockets was another complication. The gateway ran on virtual machines outside the main Kubernetes environment, while still connecting to the Realtime service in the same data center.

ICE was deliberately simple

The service offered UDP and TCP connectivity directly rather than adding a separate TURN service. For a client-server application, Sean viewed TURN as more infrastructure and another potential relay hop. Direct ICE TCP provided a fallback when UDP was blocked.

OpenAI compared the number of clients that posted an offer with the number that reached the connected state. Sean said those measurements had not produced alarming connection-failure levels. This was a pragmatic design choice based on observed results, not a claim that TURN is unnecessary for every WebRTC application.

Audio transport was not the main bottleneck

The gateway decoded and encoded Opus using libopus. It did not initially emphasize optional mechanisms such as discontinuous transmission or redundant audio. Sean’s explanation was straightforward: WebRTC and libwebrtc already handled live audio well enough that customers were more concerned with model behavior, steering, and instruction following.

Even clean audio could still produce problems with numbers or instructions. That pushed engineering attention toward the model and application layer rather than increasingly elaborate media optimization. He still cared about audio quality; it simply was not generating the loudest complaints.

Scaling meant regional gateways in front of one API

GeoIP DNS directed a new offer toward an Azure regional load balancer. Each region distributed sessions across multiple Pion gateway virtual machines. Those gateways converted WebRTC traffic to the same internal Realtime WebSocket API used elsewhere.

That extra translation did not necessarily create a meaningful latency penalty because the gateway and WebSocket service operated within the same data center. Moving business logic out of the WebSocket service might eventually enable a more direct media path, but Sean did not see a reason to do that without a measurable customer benefit.

SIP and embedded WebRTC were natural extensions

Sean wanted the Realtime API to support SIP eventually, although he did not promise a roadmap. His concern was that protocol bridges can discard or reinterpret information, especially when media-control behavior differs between SIP and WebRTC implementations.

Embedded devices were another promising area. WebRTC offered encrypted, low-latency media and widely available implementations, while a WHIP-style exchange could reduce the client integration to an offer and answer. The ecosystem ranged from browser-based Raspberry Pi experiments to small native stacks designed for constrained hardware.

Read the full annotated interview on webrtcHacks for the replay, Sean’s complete answers, Pion updates, and the audience discussion.