Skip to main content

Create a streaming chatbot

This example shows how to use Anyscale Private Endpoints to set up a simple interactive chatbot capable of holding a conversation with response streaming.

Step 0: Install dependencies

pip install openai==1.3.2

Step 1: Define a Chatbot class

Create a Python file called for this example. Copy the following code into The Chatbot class manages the conversation state and handles the communication with the API.

import os
from openai import OpenAI

class Chatbot:
def __init__(self, model: str):
self.message_history = [] # Stores the conversation history.
self.model = model # The OpenAI model to be used.

def process_input(self, user_input: str):
# Add user's message to the conversation history.
self._add_to_message_history('user', user_input)

# Create a chat completion with the OpenAI API.
response =

# Process and stream the response.
response_content = ''
first_token = True

for token in response:
delta = token.choices[0].delta.content
if first_token:
# Skip first token to unblock response.
first_token = False
elif not delta:
# End token indicating the end of the response.
self._add_to_message_history('assistant', response_content)
# Append content to message and stream it.
response_content += delta
yield delta

def _add_to_message_history(self, role: str, content: str):
self.message_history.append({'role': role, 'content': content})

Step 2: Set up the interaction Loop

Create an instance of the Chatbot class and set up a simple loop to interact with the user. For a quick demo, you can paste in your API key, but for development, follow best practices for setting your API base and key.

# Initialize the OpenAI client.
client = OpenAI(

# Create an instance of Chatbot with a specified model.
model = "meta-llama/Llama-2-70b-chat-hf"
agent = Chatbot(model)
print("Let's have a chat. Enter `quit` to exit.")
while True:
user_input = input('> ')
if user_input.lower() == 'quit':
for response_part in agent.process_input(user_input):
print(response_part, end='')

Step 3: Run the chatbot

Once you've set up the code with your token and model, run the Python script using python in a terminal.

The chatbot prompts you to start the conversation. Enter your message after the > prompt, and the chatbot responds. Type quit when you want to end the chat.