Introduction to Endpoints

Check your docs version

These docs are for the new Anyscale design. If you started using Anyscale before April 2024, use Version 1.0.0 of the docs. If you're transitioning to Anyscale Preview, see the guide for how to migrate.

Anyscale Endpoints offers the best unmodified open-source large language models (LLMs) as fully managed API endpoints. The Anyscale platform provides simple APIs to:

• Query text generation models
• Fine-tune LLMs
• Generate embeddings

Get started

1. Register an account.
2. Generate an API key.
3. Run your first query:

Python SDK streaming

import openai

query = "Write a program to load data from S3 with Ray and train using PyTorch."

client = openai.OpenAI(
    base_url = "https://api.endpoints.anyscale.com/v1",
    api_key = "esecret_ANYSCALE_API_KEY"
)
# Note: Anyscale doesn't support all arguments and ignores some of them in the backend.
chat_completion = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.1",
    messages=[{"role": "system", "content": "You are a helpful assistant."}, 
              {"role": "user", "content": query}],
    temperature=0.1,
    stream=True
)
for message in chat_completion:
    print(message.choices[0].delta.content, end="", flush=True)

Python SDK

import openai

query = "Write a program to load data from S3 with Ray and train using PyTorch."

client = openai.OpenAI(
    base_url = "https://api.endpoints.anyscale.com/v1",
    api_key = "esecret_ANYSCALE_API_KEY"
)
# Note: Anyscale doesn't support all arguments and ignores some of them in the backend.
chat_completion = client.chat.completions.create(
    model="mistralai/Mistral-7B-Instruct-v0.1",
    messages=[{"role": "system", "content": "You are a helpful assistant."}, 
              {"role": "user", "content": query}],
    temperature=0.1
)
print(chat_completion.choices[0].message.content)

Node Streaming

import OpenAI from "openai";
const anyscale = new OpenAI({
    baseURL: "https://api.endpoints.anyscale.com/v1",
    apiKey: "esecret_ANYSCALE_API_KEY"
});

async function chat_complete(prompt) {
    const completion = await anyscale.chat.completions.create({
        model: "mistralai/Mistral-7B-Instruct-v0.1",
        messages: [{"role": "system", "content": "You are a helpful assistant."},
                   {"role": "user", "content": prompt}],
        temperature: 0.1,
        stream: true
    });
    for await (const chunk of completion) {
      process.stdout.write(chunk.choices[0]?.delta?.content || '');
    }
}

const query = "Write a program to load data from S3 with Ray and train using PyTorch."
chat_complete(query);

Node

import OpenAI from "openai";
const anyscale = new OpenAI({
    baseURL: "https://api.endpoints.anyscale.com/v1",
    apiKey: "esecret_ANYSCALE_API_KEY"
});

async function chat_complete(prompt) {
    const completion = await anyscale.chat.completions.create({
        model: "mistralai/Mistral-7B-Instruct-v0.1",
        messages: [{"role": "system", "content": "You are a helpful assistant."},
                   {"role": "user", "content": prompt}],
        temperature: 0.1
    });
    process.stdout.write(completion.choices[0]?.message?.content);
}

const query = "Write a program to load data from S3 with Ray and train using PyTorch."
chat_complete(query);

cURL

export ANYSCALE_BASE_URL="https://api.endpoints.anyscale.com/v1"
export ANYSCALE_API_KEY="YOUR_ANYSCALE_ENDPOINT_API_KEY"

curl "$ANYSCALE_BASE_URL/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ANYSCALE_API_KEY" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.1",
    "messages": [{"role": "system", "content": "You are a helpful assistant."}, 
    {"role": "user", "content": "Write a program to load data from S3 with Ray and train using PyTorch."}],
    "temperature": 0.7
  }'

See Query a model for more model query details.

Data retention

Anyscale may securely retain your queries or results for up to 30 days to help prevent security or technical problems.

Anyscale treats your queries and results as “Customer Data” under our terms of service (see anyscale.com/terms). Anyscale doesn't use your queries or results to train or fine-tune models – unless you use the fine-tuning service, in which case Anyscale tunes the model only for you. If you are interested in fine-tuning a model, reach out to preview-help@anyscale.com.

If you use the fine-tuning feature, Anyscale stores both the fine-tuning data as well as the fine-tuned model (including parameters) within the Anyscale account to provide the fine-tuning feature. You may delete the fine tuning data with the API. See the Endpoints docs for more information. Don't rely on this storage as backup for the fine-tuning data. Keep a backup of any fine-tuning data you wish to keep. If you wish to delete the fine-tuned model, reach out to preview-help@anyscale.com.

Compute resources

Endpoints may utilize shared resources like GPUs and computing nodes. If you require dedicated GPUs, reach out to preview-help@anyscale.com.

See anyscale.com/services-csp-info for the latest available Cloud Service Providers, which Anyscale updates periodically. If you require the ability to choose which Cloud Service Provider Anyscale uses to host the endpoints, reach out to preview-help@anyscale.com.

Account deletion

To close your account, reach out to us at preview-help@anyscale.com. Anyscale can help you delete your account if you no longer wish to use it.