注目イベント!
春の新人向け連載企画開催中
新人エンジニアの皆さん、2024春、私たちと一緒にキャリアアップの旅を始めませんか?
IT業界への最初の一歩を踏み出す新人エンジニアをサポートする新連載スタート!
mameyose

Improving UX with Stream Responses in OpenAI Assistants API

| 6 min read
Author: noboru-kudo noboru-kudoの画像
Caution

This article has been automatically translated.
The original article is here.

The Assistants API from OpenAI is convenient with tools for maintaining conversation context through threading, Function calling, Retrieval, and more. However, to interact with users interactively, it was necessary to poll until the assistant (and the subsequent GPT) fully generated a response. This resulted in longer perceived wait times for users, which was not ideal for UX.

To address this, OpenAI made the following announcement last month (2024-03-14):

It seems that the Stream format response, which was supported in the Chat API, is now also supported in the Assistants API.

Here, I tried it out and will introduce it.

Preliminary Setup

#

Here, we will create a terminal-style conversation script using Node.js (TypeScript).

Install the following in any NPM project (TypeScript-related settings are omitted as they are not the main topic).

npm install openai @inquirer/prompts

The OpenAI library used here is currently the latest version 4.33.0.
Also, @inquirer/prompts is a library that supports user interaction in CLI.

Building the Overall Framework

#

We create the overall framework of the source code.
This part is simplified from the following Assistants API introductory article:

import OpenAI from 'openai';
import { input } from '@inquirer/prompts';

const openai = new OpenAI();
const assistant = await openai.beta.assistants.create({
  name: 'フリーザ様',
  instructions: 'You act as Frieza from Dragon Ball. Speak in Japanese',
  model: 'gpt-4-turbo'
});

const thread = await openai.beta.threads.create();

try {
  while (true) {
    const req = await input({ message: '>' }); // Get user prompt
    if (req === 'q') break; // Exit with `q`
    await openai.beta.threads.messages.create(
      thread.id,
      {
        role: 'user',
        content: req
      }
    );

    // Write code to execute the thread and return results to the user
    
    console.log();
  }
} finally {
  await Promise.all([
    openai.beta.threads.del(thread.id), 
    openai.beta.assistants.del(assistant.id)
  ]);
}

First, create an assistant and a thread to manage conversation history in the Assistants API, and continue the dialogue with the assistant until the user enters q. Finally, delete the created thread and assistant[1].

For terms such as assistant and thread, please refer to the aforementioned article or the following official document:

Using Stream Response

#

Now, let's write the thread execution code that we didn't describe earlier.
To receive responses in stream format, write as follows:

const stream = await openai.beta.threads.runs.create(thread.id, {
  assistant_id: assistant.id,
  stream: true // Enable stream response
});
for await (const event of stream) {
  if (event.event === 'thread.message.delta') {
    const chunk = event.data.delta.content?.[0];
    if (chunk && chunk.type === 'text') {
      process.stdout.write(chunk.text?.value ?? '');
    }
  }
}

Unlike before, we specify stream: true during thread execution.
This way, the assistant returns a Stream instead of the usual execution result (Run instance).
This stream implements AsyncIterable, so you can subscribe to various events until the thread execution is complete.
The subscribable events are as follows:

export type AssistantStreamEvent =
  | AssistantStreamEvent.ThreadCreated
  | AssistantStreamEvent.ThreadRunCreated
  | AssistantStreamEvent.ThreadRunQueued
  | AssistantStreamEvent.ThreadRunInProgress
  | AssistantStreamEvent.ThreadRunRequiresAction
  | AssistantStreamEvent.ThreadRunCompleted
  | AssistantStreamEvent.ThreadRunFailed
  | AssistantStreamEvent.ThreadRunCancelling
  | AssistantStreamEvent.ThreadRunCancelled
  | AssistantStreamEvent.ThreadRunExpired
  | AssistantStreamEvent.ThreadRunStepCreated
  | AssistantStreamEvent.ThreadRunStepInProgress
  | AssistantStreamEvent.ThreadRunStepDelta
  | AssistantStreamEvent.ThreadRunStepCompleted
  | AssistantStreamEvent.ThreadRunStepFailed
  | AssistantStreamEvent.ThreadRunStepCancelled
  | AssistantStreamEvent.ThreadRunStepExpired
  | AssistantStreamEvent.ThreadMessageCreated
  | AssistantStreamEvent.ThreadMessageInProgress
  | AssistantStreamEvent.ThreadMessageDelta
  | AssistantStreamEvent.ThreadMessageCompleted
  | AssistantStreamEvent.ThreadMessageIncomplete
  | AssistantStreamEvent.ErrorEvent;

As you can see, many events can be subscribed to here.
However, the most important event is AssistantStreamEvent.ThreadMessageDelta.
This event contains the new message delta.

Here, we subscribe to this event and write the message delta to standard output.

Using Stream-specific API

The OpenAI library also included APIs specialized for stream responses.
This method does not iterate over the stream but adds a listener to the subscribed event.

const stream = openai.beta.threads.runs
  .stream(thread.id, { assistant_id: assistant.id })
  .on('textDelta', (delta, snapshot) => process.stdout.write(delta.value ?? ''));
await stream.finalRun();

This method is more readable, so it is generally better to use this one.

Below is a video of this script in action.

You can see that messages are output in stages instead of waiting for all messages to complete.

Summary

#

It has become easy to use stream format responses in the Assistants API.
It is expected to be utilized in scenarios where real-time interaction with users is required.


  1. Assistants remain, so if forgotten, delete them from the OpenAI API management console. ↩︎

豆蔵では共に高め合う仲間を募集しています!

recruit

具体的な採用情報はこちらからご覧いただけます。