Skip to main content
Version: Canary 🐤

Combining Anyscale and OpenAI endpoints

Changes to Anyscale Endpoints API

Effective August 1, 2024 Anyscale Endpoints API will be available exclusively through the fully Hosted Anyscale Platform. Multi-tenant access to LLM models will be removed.

With the Hosted Anyscale Platform, you can access the latest GPUs billed by the second, and deploy models on your own dedicated instances. Enjoy full customization to build your end-to-end applications with Anyscale. Get started today.

Anyscale Endpoints works together with the OpenAI endpoint. Have both an Anyscale Endpoints token and an OpenAI API key ready to call the OpenAI endpoint and Anyscale endpoint with this example code.

Call the OpenAI endpoint with an OpenAI API key​

Execute unset OPENAI_BASE_URL and unset OPENAI_API_KEY before running the code below to avoid calling the wrong APIs. Install openai>=1.0.0 for this example.

import openai

system_content = "You will be provided with a product description and seed words. Your task is to generate potential product names."
user_content = "Product description: A home milkshake maker. Seed words: fast, healthy, compact."
client = openai.OpenAI(api_key="YOUR_OPENAI_API_KEY")
chat_completion = client.chat.completions.create(
    model="gpt-3.5-turbo-0301",
    messages=[{"role": "system", "content": system_content},
              {"role": "user", "content": user_content}],
    temperature=0.7
)
product_names = chat_completion.choices[0].message.content
print("Results from the OpenAI endpoint:\n", product_names)

Call Anyscale Endpoints​

You can also call Anyscale Endpoints with base_url and api_key parameters set for the OpenAI client.

client = openai.OpenAI(
           base_url = "https://api.endpoints.anyscale.com/v1",
           api_key="YOUR_ANYSCALE_ENDPOINT_TOKEN"
        )
chat_completion = client.chat.completions.create(
    model="meta-llama/Llama-2-70b-chat-hf",
    messages=[{"role": "system", "content": system_content},
              {"role": "user", "content": user_content}],
    temperature=0.7
)

product_names = chat_completion.choices[0].message.content
print("Results from Anyscale Endpoint:\n", product_names)