Create a streaming chatbot

Check your docs version

This version of the Anyscale docs is deprecated. Go to the latest version for up to date information.

This example shows how to use Anyscale Private Endpoints to set up a simple interactive chatbot capable of holding a conversation with response streaming.

Step 0: Install dependencies

pip install openai==1.3.2

Step 1: Define a Chatbot class

Create a Python file called chatbot.py for this example. Copy the following code into chatbot.py. The Chatbot class manages the conversation state and handles the communication with the API.

import os
from openai import OpenAI

class Chatbot:
    def __init__(self, model: str):
        self.message_history = []  # Stores the conversation history.
        self.model = model  # The OpenAI model to be used.

    def process_input(self, user_input: str):
        # Add user's message to the conversation history.
        self._add_to_message_history('user', user_input)

        # Create a chat completion with the OpenAI API.
        response = client.chat.completions.create(
            model=self.model,
            messages=self.message_history,
            stream=True
        )

        # Process and stream the response.
        response_content = ''
        first_token = True

        for token in response:
            delta = token.choices[0].delta.content
            if first_token:
                # Skip first token to unblock response.
                first_token = False
                continue
            elif not delta:
                # End token indicating the end of the response.
                self._add_to_message_history('assistant', response_content)
                break
            else:
                # Append content to message and stream it.
                response_content += delta
                yield delta

    def _add_to_message_history(self, role: str, content: str):
        self.message_history.append({'role': role, 'content': content})

Step 2: Set up the interaction Loop

Create an instance of the Chatbot class and set up a simple loop to interact with the user. For a quick demo, you can paste in your API key, but for development, follow best practices for setting your API base and key.

# Initialize the OpenAI client.
client = OpenAI(
    api_key=os.environ['OPENAI_API_KEY'],
    base_url=os.environ['OPENAI_BASE_URL']
)

# Create an instance of Chatbot with a specified model.
model = "meta-llama/Llama-2-70b-chat-hf"
agent = Chatbot(model)

print("Let's have a chat. Enter `quit` to exit.")
while True:
    user_input = input('> ')
    if user_input.lower() == 'quit':
        break
    for response_part in agent.process_input(user_input):
        print(response_part, end='')

Step 3: Run the chatbot

Once you've set up the code with your token and model, run the Python script using python chatbot.py in a terminal.

The chatbot prompts you to start the conversation. Enter your message after the > prompt, and the chatbot responds. Type quit when you want to end the chat.