Create a streaming chatbot
This example shows how to use Anyscale Private Endpoints to set up a simple interactive chatbot capable of holding a conversation with response streaming.
Step 0: Install dependencies
pip install openai==1.3.2
Step 1: Define a Chatbot
class
Create a Python file called chatbot.py
for this example. Copy the following code into chatbot.py
. The Chatbot
class manages the conversation state and handles the communication with the API.
import os
from openai import OpenAI
class Chatbot:
def __init__(self, model: str):
self.message_history = [] # Stores the conversation history.
self.model = model # The OpenAI model to be used.
def process_input(self, user_input: str):
# Add user's message to the conversation history.
self._add_to_message_history('user', user_input)
# Create a chat completion with the OpenAI API.
response = client.chat.completions.create(
model=self.model,
messages=self.message_history,
stream=True
)
# Process and stream the response.
response_content = ''
first_token = True
for token in response:
delta = token.choices[0].delta.content
if first_token:
# Skip first token to unblock response.
first_token = False
continue
elif not delta:
# End token indicating the end of the response.
self._add_to_message_history('assistant', response_content)
break
else:
# Append content to message and stream it.
response_content += delta
yield delta
def _add_to_message_history(self, role: str, content: str):
self.message_history.append({'role': role, 'content': content})
Step 2: Set up the interaction Loop
Create an instance of the Chatbot
class and set up a simple loop to interact with the user. For a quick demo, you can paste in your API key, but for development, follow best practices for setting your API base and key.
# Initialize the OpenAI client.
client = OpenAI(
api_key=os.environ['OPENAI_API_KEY'],
base_url=os.environ['OPENAI_BASE_URL']
)
# Create an instance of Chatbot with a specified model.
model = "meta-llama/Llama-2-70b-chat-hf"
agent = Chatbot(model)
print("Let's have a chat. Enter `quit` to exit.")
while True:
user_input = input('> ')
if user_input.lower() == 'quit':
break
for response_part in agent.process_input(user_input):
print(response_part, end='')
Step 3: Run the chatbot
Once you've set up the code with your token and model, run the Python script using python chatbot.py
in a terminal.
The chatbot prompts you to start the conversation. Enter your message after the >
prompt, and the chatbot responds. Type quit
when you want to end the chat.