--- title: "Inside OpenAI’s gpt-realtime WebRTC Implementation" canonical: "https://cogint.ai/openai-gpt-realtime-webrtc-summary/" description: "A condensed packet- and SDP-oriented analysis of OpenAI’s generally available gpt-realtime WebRTC path. It documents combined session creation and offer/answer, expanded ICE candidates, an H.264 video path converted to images internally, and convergence between the API and ChatGPT browser negotiation." author: "Chad Hart" published: "2025-09-23T00:00:00.000-04:00" modified: "2026-07-14T16:36:48.000-04:00" site_tags: - "voice-ai" - "openai" - "webrtc" machine_topics: - "Voice AI" - "OpenAI" - "WebRTC" is_based_on: - "https://webrtchacks.com/how-openai-does-webrtc-in-the-new-gpt-realtime/" content_fidelity: "faithful_html_conversion" annotations_not_part_of_article: true word_count: 728 approx_tokens: 969 content_hash: "3502b31adbd2c3d6fa7a923dcbe30818c2cd0f2ef2908694ab45a5c02d00031e" markdown_url: "https://cogint.ai/openai-gpt-realtime-webrtc-summary/index.md" --- # Machine-oriented context This section is generated metadata and is not part of the original article. ## Summary A condensed packet- and SDP-oriented analysis of OpenAI’s generally available gpt-realtime WebRTC path. It documents combined session creation and offer/answer, expanded ICE candidates, an H.264 video path converted to images internally, and convergence between the API and ChatGPT browser negotiation. ## Key findings - One bundled peer connection carried audio, video, and an SCTP DataChannel. - The GA calls endpoint combined SDP negotiation and initial model-session configuration in one multipart request. - The observed service advertised three Azure addresses with UDP 3478 and TCP 443 candidates. - WebRTC video was converted into images for the model rather than consumed as a conventional continuous video stream. - ChatGPT browser voice and the developer API showed closely aligned WebRTC offers in the author’s comparison. ## Provenance and sources - Adaptation of [How OpenAI does WebRTC in the new gpt-realtime](https://webrtchacks.com/how-openai-does-webrtc-in-the-new-gpt-realtime/), originally published 2025-09-23; relationship: Condensed technical summary. - Tested or observed at: 2025-09-23 - [How OpenAI does WebRTC in the new gpt-realtime](https://webrtchacks.com/how-openai-does-webrtc-in-the-new-gpt-realtime/) — webrtcHacks; original hands-on analysis - [Realtime API with WebRTC](https://platform.openai.com/docs/guides/realtime-webrtc) — OpenAI; official developer documentation ## Limitations - Observed SDP, infrastructure, pricing behavior, and ChatGPT implementation details are time-bounded. - The limited image-usage experiments did not establish a reliable pricing formula. # Published article Faithful Markdown conversion of the human-readable article follows. This summary is based on my September 2025 webrtcHacks investigation, [How OpenAI does WebRTC in the new gpt-realtime](https://webrtchacks.com/how-openai-does-webrtc-in-the-new-gpt-realtime/?ref=cogint.ai). OpenAI had released the generally available `gpt-realtime` API, and I originally expected to update my long guide to the Beta API. The WebRTC changes were substantial enough to examine separately, so I updated my single-file demo, inspected the negotiation, and compared the API with ChatGPT’s browser-based voice implementation. The complete webrtcHacks article contains the packet-level investigation. This version covers the main observations from that work. ## A compact 1:1 WebRTC stack The implementation remained deliberately small. One `RTCPeerConnection` used BUNDLE for audio, video, and a standard SCTP DataChannel. Media was encrypted with DTLS-SRTP. Audio offered Opus with in-band forward error correction plus PCMU and PCMA fallbacks. The video path used H.264 without simulcast. The server advertised host candidates rather than a conventional mix of host, server-reflexive, and TURN relay candidates. Congestion feedback used transport-wide congestion control. The HTTP exchange looked similar to WebRTC HTTP Ingest Protocol (WHIP), although it was not the WHIP specification. That is a relatively clean set of components for a 1:1 browser-to-service session. There was no multiparty Selective Forwarding Unit (SFU) architecture to expose and no complicated signaling service for the developer to manage. ## Offer/answer and session creation were combined The Beta flow sent raw SDP to the Realtime endpoint and then used a separate `session.update` message over the DataChannel to configure the model. The generally available interface introduced a calls endpoint that accepted both the SDP offer and a JSON session object in one multipart request. OpenAI returned the SDP answer, the browser applied it to the peer connection, and later updates continued over the DataChannel. Combining WebRTC negotiation and model-session creation reduced the number of initialization steps without changing the basic separation between media and API events. ## ICE advertised more paths The Beta SDP I had inspected advertised one server address with UDP and TCP candidates. The September implementation advertised 3 Azure addresses, each with UDP on port 3478 and TCP on port 443. More addresses provided additional regional paths and resilience. Moving TCP to port 443 also made that fallback more likely to pass through restrictive firewalls. Separate RTCP candidates disappeared because `rtcp-mux` already carried RTP and RTCP on the same transport. The SDP also stopped declaring an immediate end to candidate gathering, leaving room for later candidates. These were small protocol details, but together they showed a more deliberate connection-establishment setup than the earlier implementation. ## Video was a media stream feeding images Adding a camera track to the demo was mechanically simple: request audio and video from `getUserMedia` and add both to the peer connection. The negotiated video codec was H.264, with several profiles offered. H.264 was a practical choice for hardware acceleration across many devices. The less obvious question was what happened after the video reached OpenAI. API events and usage observations did not look like the model continuously consumed a normal video stream. Sean DuBois confirmed that the gateway converted the WebRTC video into images passed toward the model over the internal WebSocket path. That suggested a different capture strategy from a conventional video call. Sending 30 frames per second was unnecessary when the system selected occasional images for analysis. My demo used high-resolution video at 1 frame per second to reduce the transmitted media while retaining image detail. I also experimented with the associated image-input charges, but the small number of tests produced inconsistent totals. I did not think those observations were enough to establish a reliable pricing formula. ## ChatGPT and the API had converged I opened a ChatGPT browser voice session and compared its WebRTC negotiation with the API. The offers were nearly identical. ChatGPT did not capture a camera in that web session, but it negotiated the same video capabilities. The direction attributes differed because the API demo offered to send and receive while ChatGPT only offered to send media in the direction its application required. The larger observation was alignment. Earlier OpenAI real-time products had visibly different media paths. By September 2025, the browser experience and the developer API appeared to share the same fundamental WebRTC implementation. Read [the complete webrtcHacks analysis](https://webrtchacks.com/how-openai-does-webrtc-in-the-new-gpt-realtime/?ref=cogint.ai) for the full analysis, live demo, SDP inspection, ICE comparison, video experiments, and supporting links.