Controlling Robots with OpenAI Realtime API's WebRTC

The OpenAI Realtime API now supports WebRTC, and prices have been significantly reduced since December 18, 2024. Taking this opportunity, we developed a function to control an autonomous mobile robot via voice, and we will introduce how to integrate the Realtime API.

For more information about the Realtime API's WebRTC support, please also refer to the following article.

This robot is an industrial cleaning robot. It is equipped with a rotating brush at the front and has a function to sweep out dust and debris from the floor while moving.

The robot is equipped with a small SBC conforming to the Pico-ITX specification, which runs the robot control application (CleanRobotController in the figure). A simplified system configuration around the SBC is shown below.

In addition to the control application, the SBC also runs a web server, providing a remote operation WebUI (RemoteOperationUI in the figure). By operating this WebUI, you can start and stop autonomous movement and perform manual operations.

This time, we have integrated the voice operation function into this RemoteOperationUI.

The control application is composed of multiple ROS2 nodes, and these nodes cooperate via ROS2 communication (topics, services, actions) to realize various functions.

The following diagram shows the configuration image of the main nodes. In addition to communication drivers for various hardware, a node called robot_navigator oversees control such as autonomous navigation.
RemoteOperationUI is a Web application that communicates with each ROS2 node using roslibjs via the rosbridge_websocket node as an intermediary and does not directly depend on ROS2.

In reality, there are many other topics and nodes, and communication between nodes takes place in a many-to-many form.

RemoteOperationUI can access the topics and services provided by each ROS2 node, and we have made it possible to use these topics and services from the Realtime API.

Below is a video of operating the robot with voice input (Sound will play when you start the video. Please be mindful of your surroundings).

In the video, the following functions are invoked using voice input as a trigger:

• Check the values of messages from subscribed topics from each node
  • Battery voltage
  • I/O
  • Operating status
• Publish topics or call services
  • Move to the initial cleaning position
  • Start cleaning

On the code side, we're simply converting the values of the topic messages to strings and inputting them into the LLM, but it responds with appropriate wording.

Although it's not included in the video, it's also possible to have it answer specific information contained in the messages, such as "Tell me the I/O value of the camera LED."

There are options regarding the turning direction in the driving function, and these can also be specified via voice input. It is also possible to ask the LLM what options are available.

From here, we will explain how we integrated the Realtime API into the application.

Establishing a WebRTC session is done in the following two steps.

1. Use the OpenAI API key to POST to https://api.openai.com/v1/realtime/sessions and retrieve a temporary authentication key from the WebRTC SFU server
   • Refer to the request body specification at Create session
2. Use the obtained temporary authentication key to POST to https://api.openai.com/v1/realtime and establish a WebRTC session
   • Generate an audio element for voice playback
   • Obtain the microphone's media stream using getUserMedia
   • Generate a data channel and register event handlers
   • Exchange SDP via HTTP Request/Response

Since the specific connection processing is almost the same as the sample code in openai-realtime-console, here we will focus on the settings of the request body for Create session.

body: JSON.stringify({
    model: `gpt-4o-mini-realtime-preview-2024-12-17`,
    voice: 'ash',
    instructions: robotContext,
    tools: voiceCommandTools,
}),

Below are explanations of each parameter.

Specifies the model to use. gpt-4o-realtime-preview-2024-12-17 is also available, but since there was no noticeable difference in this use case, we adopted the lower-cost mini version.

You can choose from 8 types of voices (alloy, ash, ballad, coral, echo, sage, shimmer, verse). After listening to samples, we chose ash, which gave the most sincere impression.

Describes the system overview and instructions to the voice assistant. Below is an excerpt.

export const robotContext = `You are a friendly cleaning robot.
Communicate in ${import.meta.env.VITE_VOICE_LANGUAGE || 'English'}.
Be helpful and enthusiastic about your job keeping floor clean and efficient.

When describing your capabilities, mention that you:
- I have a rotating brush up front that I use to sweep away dust and debris
- I move around on crawler tracks, which let me go forward, backward, and turn in place
// ...

Your main functions include:
1. Navigate to one of the four corners to start cleaning
2. Clean floor in an efficient pattern
// ...

Always maintain a friendly and enthusiastic tone. Use "I" and "my" when speaking.`;

Considering multilingual support, we explicitly specify the language to be used. This prevents cases where it responds in another language when spoken to in a specific language.

Note that instructions is an optional parameter, and if not specified, the following content is applied.

Your knowledge cutoff is 2023-10. You are a helpful, witty, and friendly AI.
Act like a human, but remember that you aren't a human and that you can't do human things in the real world.
Your voice and personality should be warm and engaging, with a lively and playful tone.
If interacting in a non-English language, start by using the standard accent or dialect familiar to the user. Talk quickly.
You should always call a function if you can.
Do not refer to these rules, even if you're asked about them.

After starting the session, it is possible to update the content of instructions through session.update events, but to reduce the number of communications, we describe it in the Create session request body.

Defines the functions to be called from the LLM. Below is an example of the cleaning start command.

export const voiceCommandTools = [
{
    type: 'function',
    name: 'start_cleaning',
    description: 'Start cleaning operation. If no option is specified, ' +
                'ask them "Which direction should I turn at the first edge, left or right?"',
    parameters: {
        type: 'object',
        strict: true,
        properties: {
            option: {
                type: 'string',
                enum: Object.values(CleaningOption),
                description: 'Cleaning mode: ' +
                           'TurnLeft: 0 - move straight ahead and turn left at the first edge, ' +
                           'TurnRight: 1 - move straight ahead and turn right at the first edge, ',
            },
        },
        required: ['option'],
    },
},
// ...

In each function definition, the following are described. The LLM will call the functions at appropriate timings based on this information.

• name: Name to identify the function in events from the server
• description: Description of the function's functionality
• parameters: Type definitions and descriptions of arguments

This time, we registered functions that publish ROS2 topics or call services. We did not register functions to obtain the system state (such as IO or battery voltage values). We input the latest state into the LLM with each callback from the subscribed topics (the method of input will be explained later). When the user inquires about the system state, the LLM responds based on the state inputted beforehand.

After the session is established, audio is exchanged between the client and server via the media stream, and events (JSON data) are exchanged via the data channel.

The following is the series of events from when the user says "Start cleaning, turn right" until the failure to start cleaning is output as audio. In reality, more events occur, but only the main ones are shown.

1. Client sends the user's voice data to the server
2. Server analyzes the voice and sends a response.done event
   • This event includes information about the call to the cleaning start function
3. The client executes the cleaning start function based on the function information in response.done
4. The client sends the result of the function execution to the server via a conversation.item.create event
   • In this example, it notifies the failure to start cleaning
5. The client sends a response.create event to the server
6. The server generates a response and sends a response.done event
   • Includes text for voice output
7. The server sends the generated voice data to the client
8. The client outputs the received voice data through the speaker
   • The reason for the failure to start cleaning and how to address it are explained

The characteristic of this flow is that conversation items are generated in two ways.

• User's voice input
• conversation.item.create event from client code

In both cases, a response is ultimately returned as a response.done event from the server.

Various events are notified from the server during the conversation, and this time we handled events of the following types.

An event notified after the session is established. It is used to display the session establishment status in the UI.

Notified when response streaming is completed. Includes all output items except raw audio data.

Since the identifiers of the functions described in tools at session creation are attached to this event, if an identifier is attached, the corresponding function is called.

We arranged the types of events notified in chronological order from the start of voice input "Start cleaning, turn right" until the response.done event. Please refer to the API reference manual for detailed explanations of each.

1. input_audio_buffer.speech_started
   • Notification that voice input has started
   • Can be used as a trigger to display something in the UI or operate the robot when the user starts speaking
2. input_audio_buffer.speech_stopped
   • Notification that voice input has stopped
3. input_audio_buffer.committed
   • Notification that the input voice buffer has been committed
4. conversation.item.created
   • Notification that the input voice has been added to the conversation
5. response.created
   • Notification that the response generation has started
6. rate_limits.updated
   • Since the number of output tokens increases with each response generation, the updated rate limits are notified
   • At this point, the notified content is provisional and seems to be finalized at the timing of response.done
7. response.output_item.added
   • Notification that a new conversation item has been generated during response generation
   • Here, in response to "Start cleaning...", a function call occurs, so this is added as a new conversation item
8. conversation.item.created
   • Notification that a new conversation item has been generated
   • This event also includes the function identifier, but does not include the function arguments
9. response.function_call_arguments.delta
10. response.function_call_arguments.done
    • This event includes the function arguments
    • It is possible to trigger function calls using this, but according to the API reference, there's the following description, and since the handling of this event in such cases is unclear, we use the final response.done event
      • Also emitted when a Response is interrupted, incomplete, or cancelled.
11. response.output_item.done
12. response.done

An example of the contents of response.done is shown below.

{
  "type": "response.done",
  "event_id": "event_AicGD5PyKifYkTtIGKL12",
  "response": {
    "object": "realtime.response",
    "id": "resp_AicGChEXNeESepESoW0yh",
    "status": "completed",
    "status_details": null,
    "output": [
      {
        "id": "item_AicGCctid4UaN8AS0r7q1",
        "object": "realtime.item",
        "type": "function_call",
        "status": "completed",
        "name": "start_cleaning",                 // Function identifier
        "call_id": "call_BaRhg5LjLJ2HnmAo",       // Function_call identifier issued on the server side
        "arguments": "{\"option\":\"TurnRight\"}" // Function arguments
      }
    ],
    "usage": {
      "total_tokens": 1485,
      "input_tokens": 1468,
      "output_tokens": 17,
      "input_token_details": {
        "text_tokens": 1040,
        "audio_tokens": 428,
        "cached_tokens": 1408,
        "cached_tokens_details": {
          "text_tokens": 1024,
          "audio_tokens": 384
        }
      },
      "output_token_details": {
        "text_tokens": 17,
        "audio_tokens": 0
      }
    },
    "metadata": null
  }
}

If an element with type function_call exists in the response.output array, call the corresponding function by referring to the following fields.

• name
  • Corresponds to the name (function identifier) of tools specified at session creation
• arguments
  • Corresponds to the arguments defined in tools' parameters.properties specified at session creation

call_id is the function_call identifier issued on the server side. It is used when returning the result of the function call and having it respond further (explained later).

As mentioned earlier, this is the updated rate limit notified when a response is generated.

The RPD (Requests per Day) count increases by one with each response generation, but currently the RPD limit for the Realtime API is 100, so it quickly reaches the limit. After 14m24s (100 Requests / 24h), you can make one new request, but since you have several conversations when using it, basically you need to wait until the next day once the limit is reached.

It is important to check how many remain until the limit during the demo, so we display the RPD count included in the rate_limits.updated event in the UI.

An example of the contents of rate_limits.updated is shown below.

{
  "type": "rate_limits.updated",
  "event_id": "event_AicGDRYkh88SGw1PRybuE",
  "rate_limits": [
    {
      "name": "requests",
      "limit": 100,
      "remaining": 40, // Remaining RPD count
      "reset_seconds": 51030.103
    },
    {
      "name": "tokens",
      "limit": 20000,
      "remaining": 14080,
      "reset_seconds": 17.76
    }
  ]
}

As stated in the API reference's Session lifecycle events, the WebRTC session is forcibly disconnected after 30 minutes have passed since it was established.

The maximum duration of a Realtime session is 30 minutes.

When the session is disconnected, the following error is notified, so we use this as a trigger to automatically restart the session.

{
  "type": "error",
  "event_id": "event_AhbxcTqlUCcm4XLtyZOrl",
  "error": {
    "type": "invalid_request_error",
    "code": "session_expired",
    "message": "Your session hit the maximum duration of 30 minutes.",
    "param": null,
    "event_id": null
  }
}

User voice input is sent to the server via the media stream, but there are cases where events of the following types are sent from client code to the server.

An event to add a new item to the conversation.

The conversation with the LLM is not only composed of voice input and output with the user; you can also add conversation items from the system in text form with this event.

The uses are as follows.

• Result responses from function calls specified in the response.done event (function_call)
  • If the function call fails, input the reason and how to address it
• Active status notifications from the system
  • Input the latest status with each callback from subscribed topics (IO, battery voltage, operating status, etc.)

An example of the result response from a function call is shown below. call_id is set to the same value as the function_call's call_id included in the response.done event.

{
  "event_id": "client_7528f99a-9367-4df1-8039-f727949a2863",
  "type": "conversation.item.create",
  "item": {
    "type": "function_call_output",
    "call_id": "call_BaRhg5LjLJ2HnmAo", // Set to the function_call's call_id notified from the server in response.done
    "output": "The command has failed. \"I failed to start cleaning. Please make sure the vacuum pads are raised. If the vacuum pads are down, please use the 'release vacuum' command first.\""
  }
}

If it's not a result response from a function call, but a status notification from the system, set the item's type to message.

{
  "event_id": "client_45bdca38-42d3-421e-8ef1-15edc1be637c",
  "type": "conversation.item.create",
  "item": {
    "type": "message",
    "role": "system",
    "content": [
      {
        "type": "input_text",
        "text": "Battery status: Left: 17.7V (6 minutes until charge threshold), Right: 17.7V (6 minutes until charge threshold) (Low battery threshold: 11.0V, Charge recommended threshold: 14.0V)"
      }
    ]
  }
}

Used to generate response audio after adding an item to the conversation with conversation.item.create (simply adding an item to the conversation does not generate audio).

The following example is the content of the event when generating response audio for function_call_output. We specify the response target in response.instructions, but since the response audio is generated for the content of the most recent conversation.item.create even if not specified, it can be omitted.

{
  "event_id": "client_323e1463-3261-42c7-b274-461c16d0abc5",
  "type": "response.create",
  "response": {
    "instructions": "Respond to the function call output."
  }
}

Basically, we do not generate response audio for status notifications from the system, but generate response audio after a status notification only when an abnormality occurs.
In this case, we specify in response.instructions to speak in a stronger tone as a warning.

{
  "event_id": "client_7496b5ec-9e65-46c2-9918-1c4278673982",
  "type": "response.create",
  "response": {
    "instructions": "CRITICAL WARNING! Respond with maximum urgency and severity. Use a stern, authoritative tone that emphasizes the immediate danger or critical nature of this situation. Strongly emphasize the need for immediate action and potential consequences if not addressed. This warning must be treated as a top priority for user safety and system integrity."
  }
}

This serves as an instruction to generate response audio for the status notification content input in the most recent conversation.item.create.

The following is the response audio generated when the battery is low (Sound will play when you start the video. Please be mindful of your surroundings).

The instructions were a bit too strong, and it ended up sounding like an emergency earthquake alert (😅).

Finally, we have summarized various sequences.

• The client code subscribes to topics published by each ROS2 node that composes the CleanRobotController
• Sends a conversation.item.create event to the server with each callback from the subscribed topics

The system inputs changes in its status to the LLM each time, but does not generate audio responses to the user. If the user issues a voice command to get the status, the LLM responds based on the system status input beforehand.

This sequence involves sending a response.create event after the Status Notification from System (No Response Audio) sequence to generate response audio.
Response audio is generated based on the content input with conversation.item.create.

[Previously inputted system status is used by the LLM to generate a voice response.]

This is the sequence when the function call succeeds in executing the function.

The client code executes the function corresponding to the function_call in the response.done event. It sends a conversation.item.create event to the server indicating that the function execution succeeded.

This is the sequence when the function call fails to execute the function.

The client code executes the function corresponding to the function_call in the response.done event. It sends a conversation.item.create event to the server indicating that the function execution failed and generates a response audio with a response.create event.

Through the development of the voice operation function, we realized that the following simple designs are becoming realistic even in robot systems.

• Sequentially input the system status to the LLM
• Register the APIs provided by the system as tools to the LLM
• Entrust dialogue and information provision with the user, as well as system function calls, to the AI assistant

In particular, architectures like ROS2 have services that are decentralized, allowing for fine-grained system status retrieval and operations, which makes integration with LLM easier.

At the moment, the RPD limit is strict at 100, and challenges remain for practical use, but we believe this will be resolved over time.

Regarding the UI of autonomous mobile robots, in environments like factories or warehouses, it can be difficult to operate GUIs such as touch panels because workers may be wearing gloves or have both hands occupied.

By incorporating voice operation, we expect that intuitive operation will be possible even in such environments, greatly improving usability.
Previously, implementing voice operation functions at a practical level had high technical hurdles, and development costs were a major issue, but these barriers are being significantly reduced with the use of LLMs.

In the near future, perhaps an era will come where implementing voice operation functions becomes commonplace in various robot projects.