Inside OpenAI’s gpt-realtime WebRTC Implementation

This summary is based on my September 2025 webrtcHacks investigation, How OpenAI does WebRTC in the new gpt-realtime. OpenAI had released the generally available gpt-realtime API, and I originally expected to update my long guide to the Beta API. The WebRTC changes were substantial enough to examine separately, so I updated my single-file demo, inspected the negotiation, and compared the API with ChatGPT’s browser-based voice implementation.

The complete webrtcHacks article contains the packet-level investigation. This version covers the main observations from that work.

A compact 1:1 WebRTC stack

The implementation remained deliberately small. One RTCPeerConnection used BUNDLE for audio, video, and a standard SCTP DataChannel. Media was encrypted with DTLS-SRTP. Audio offered Opus with in-band forward error correction plus PCMU and PCMA fallbacks. The video path used H.264 without simulcast.

The server advertised host candidates rather than a conventional mix of host, server-reflexive, and TURN relay candidates. Congestion feedback used transport-wide congestion control. The HTTP exchange looked similar to WebRTC HTTP Ingest Protocol (WHIP), although it was not the WHIP specification.

That is a relatively clean set of components for a 1:1 browser-to-service session. There was no multiparty Selective Forwarding Unit (SFU) architecture to expose and no complicated signaling service for the developer to manage.

Offer/answer and session creation were combined

The Beta flow sent raw SDP to the Realtime endpoint and then used a separate session.update message over the DataChannel to configure the model. The generally available interface introduced a calls endpoint that accepted both the SDP offer and a JSON session object in one multipart request.

OpenAI returned the SDP answer, the browser applied it to the peer connection, and later updates continued over the DataChannel. Combining WebRTC negotiation and model-session creation reduced the number of initialization steps without changing the basic separation between media and API events.

ICE advertised more paths

The Beta SDP I had inspected advertised one server address with UDP and TCP candidates. The September implementation advertised 3 Azure addresses, each with UDP on port 3478 and TCP on port 443.

More addresses provided additional regional paths and resilience. Moving TCP to port 443 also made that fallback more likely to pass through restrictive firewalls. Separate RTCP candidates disappeared because rtcp-mux already carried RTP and RTCP on the same transport. The SDP also stopped declaring an immediate end to candidate gathering, leaving room for later candidates.

These were small protocol details, but together they showed a more deliberate connection-establishment setup than the earlier implementation.

Video was a media stream feeding images

Adding a camera track to the demo was mechanically simple: request audio and video from getUserMedia and add both to the peer connection. The negotiated video codec was H.264, with several profiles offered. H.264 was a practical choice for hardware acceleration across many devices.

The less obvious question was what happened after the video reached OpenAI. API events and usage observations did not look like the model continuously consumed a normal video stream. Sean DuBois confirmed that the gateway converted the WebRTC video into images passed toward the model over the internal WebSocket path.

That suggested a different capture strategy from a conventional video call. Sending 30 frames per second was unnecessary when the system selected occasional images for analysis. My demo used high-resolution video at 1 frame per second to reduce the transmitted media while retaining image detail.

I also experimented with the associated image-input charges, but the small number of tests produced inconsistent totals. I did not think those observations were enough to establish a reliable pricing formula.

ChatGPT and the API had converged

I opened a ChatGPT browser voice session and compared its WebRTC negotiation with the API. The offers were nearly identical. ChatGPT did not capture a camera in that web session, but it negotiated the same video capabilities. The direction attributes differed because the API demo offered to send and receive while ChatGPT only offered to send media in the direction its application required.

The larger observation was alignment. Earlier OpenAI real-time products had visibly different media paths. By September 2025, the browser experience and the developer API appeared to share the same fundamental WebRTC implementation.

Read the complete webrtcHacks analysis for the full analysis, live demo, SDP inspection, ICE comparison, video experiments, and supporting links.