Skip to main content

Fine-tuning API

Changes to Anyscale Endpoints API

Effective August 1, 2024 Anyscale Endpoints API will be available exclusively through the fully Hosted Anyscale Platform. Multi-tenant access to LLM models will be removed.

With the Hosted Anyscale Platform, you can access the latest GPUs billed by the second, and deploy models on your own dedicated instances. Enjoy full customization to build your end-to-end applications with Anyscale. Get started today.

Fine-tuning allows you to take a pre-trained model and adapt it to your specific use case. You provide a dataset, and then train the model on that dataset. The fine-tuned model generates text that follows the same form as your training dataset.

Prerequisites​

  • Upload your files with the Files API - Upload file
  • Use either (1) the OpenAI SDK or (2) cURL as a REST client
info

The fine-tuning API is fully compatible with OpenAI's fine-tuning API, aside from listing events. You only need to setup your API base URL and API key to point to Anyscale Endpoints and you can use OpenAI's SDK to get started.

Use the OpenAI SDK​

import openai

client = openai.OpenAI(
    base_url = "https://api.endpoints.anyscale.com/v1",
    api_key = "esecret_yourAuthTokenHere"  # from https://app.endpoints.anyscale.com/credentials
)

Use cURL​

export ANYSCALE_BASE_URL="https://api.endpoints.anyscale.com/v1"
export ANYSCALE_API_KEY="esecret_yourAuthTokenHere"

APIs​

Create fine-tuning job​

POST https://api.endpoints.anyscale.com/v1/fine_tuning/jobs

Submits a fine-tuning job.

Request body​

  • model string required

  • training_file string required

    • ID of the .jsonl file corresponding to the training dataset uploaded using the Files API. See Upload file.

  • validation_file string (See Data Format - Validation Data)

    • ID of the .jsonl file corresponding to the validation dataset uploaded using the Files API. See Upload file.

  • hyperparameters object

    • n_epochs integer (See Data Format - Number of epochs)
      • Number of epochs to train for. If not specified, it's auto-calculated based on the training dataset size.
    • context_length integer (See Data Format - Context Length)
      • Context window size during training. If not specified, it's auto-calculated based on sequence length of the examples in the dataset.

  • suffix string  Defaults to user's first name

    • Appended to model along with a hash to generate fine_tuned_model
      (For example: mistralai/Mixtral-8x7B-Instruct-v0.1:my_suffix:aBc1234)

Returns​

A fine-tuning.job object

Examples

Use the OpenAI SDK​

client.fine_tuning.jobs.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    training_file="file_123",
)

Use cURL​

curl "$ANYSCALE_BASE_URL/fine_tuning/jobs" \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $ANYSCALE_API_KEY" \
    -d '{
        "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
        "training_file": "file_123"
    }'

Get fine-tuning job​

GET https://api.endpoints.anyscale.com/v1/fine_tuning/jobs/{fine_tuning_job_id}

Retrieves a fine-tuning job by ID.

Path parameters​

  • fine_tuning_job_id
    • ID of the fine-tuning job

Returns​

A fine-tuning.job object

Examples

Use the OpenAI SDK​

client.fine_tuning.jobs.retrieve("eftjob_123")

Use cURL​

curl "$ANYSCALE_BASE_URL/fine_tuning/jobs/eftjob_123" \
    -H "Authorization: Bearer $ANYSCALE_API_KEY"

List fine-tuning jobs​

GET https://api.endpoints.anyscale.com/v1/fine_tuning/jobs

Lists all fine-tuning jobs that you have access to, from newest to oldest.

Query parameters optional​

  • after string
    • ID of the fine-tuning job from the last page of results
  • limit integer  Defaults to 20
    • Maximum number of fine-tuning jobs to return

Returns​

A list of fine-tuning.job objects

{
    "data": [fine-tuning.job, ...]
}
Examples

Use the OpenAI SDK​

client.fine_tuning.jobs.list()

Use cURL​

curl "$ANYSCALE_BASE_URL/fine_tuning/jobs" \
    -H "Authorization: Bearer $ANYSCALE_API_KEY"

Cancel fine-tuning job​

POST https://api.endpoints.anyscale.com/v1/fine_tuning/jobs/{fine_tuning_job_id}/cancel

Cancels a fine-tuning job by ID.

Path parameters​

  • fine_tuning_job_id
    • ID of the fine-tuning job

Returns​

The fine-tuning.job object that terminated.

Examples

Use the OpenAI SDK​

client.fine_tuning.jobs.cancel("eftjob_123")

Use cURL​

curl "$ANYSCALE_BASE_URL/fine_tuning/jobs/eftjob_123/cancel" \
    -H "Authorization: Bearer $ANYSCALE_API_KEY" \
    -X 'POST'

Additional information​

After your job has entered succeeded status, Anyscale sends you an email notifying you that your fine-tuned model is ready for use. See Query a model for more details on how to query your fine-tuned model.

info

See Supported models for a list of open source models that can be fine-tuned.

Job scheduling​

Fine-tuning jobs process using a queue. Your job queue with pending status and scheduled to run.

Models​

fine-tuning.job​

Attributes​

  • id string required
    • ID of the fine-tuning job
  • created_at string required
    • UTC creation data and time of the fine-tuning job.
  • creator_id string required
    • ID of the user who created the fine-tuning job
  • status string required
    • Status of the fine-tuning job
    • pending | running | succeeded | failed | cancelled
  • model string required
  • fine_tuned_model string required
    • Name of the fine-tuned model for use on succeeded status
      For example, mistralai/Mixtral-8x7B-Instruct-v0.1:my_suffix:aBc1234.
  • training_file string required
    • ID of the .jsonl file used for training
  • validation_file string (See Data Format - Validation Data)
    • ID of the .jsonl file used for validation
  • hyperparameters object
    • n_epochs integer (See Data Format - Number Of Epochs)
      • Number of epochs to train for. If not specified, it's auto-calculated based on the training dataset size.
    • context_length integer (See Data Format - Context Length)
      • Number of tokens to use as context for each training example
  • result_files array[string]
    • Contains ID of the .jsonl file that contains fine-tuning results. See Result file.
  • trained_tokens integer (see Data Format - Token Counting)
    • Number of tokens trained on
  • finished_at string
    • UTC date and time of when the fine-tuning job entered succeeded, failed, or cancelled from a status of running
  • error string
    • Error message if the job failed
info

hyperparameters (if not specified during Create fine-tuning job) and trained_tokens are available only on succeeded status. To view these fields while the job is running, access the result file.

Example​

{
  "id": "eftjob_123",
  "created_at": "2023-01-01T00:00:00.123456+00:00",
  "creator_id": "euser_123",
  "status": "succeeded",
  "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
  "fine_tuned_model": "mistralai/Mixtral-8x7B-Instruct-v0.1:my_suffix:123aBcD",
  "training_file": "file_123",
  "validation_file": "file_456",
  "hyperparameters": {
    "n_epochs": 1,
    "context_length": 512
  },
  "result_files": ["file_789"],
  "trained_tokens": 123456,
  "finished_at": "2023-01-01T00:30:00.123456+00:00",
  "error": null
}

Result file​

The result file contains the fine-tuning results in .jsonl format.

Read the file with the Files API - Get file contents:

Use the OpenAI SDK​

result_file_id = client.fine_tuning.jobs.retrieve("eftjob_123").result_files[0]
client.files.retrieve_content(result_file_id)

Example​

{"epoch": 0, "iteration": 1, "train_loss": 2.942214250564575, "trained_tokens": 36192, "valid_loss": null, "perplexity": null, "hyperparameters": {"n_epochs": 5}, "time_since_job_start": 173.76054668426514}
...
{"epoch": 0, "iteration": 13, "train_loss": 0.5098391771316528, "trained_tokens": 8325090, "valid_loss": 0.44729286432266235, "perplexity": 1.5640722931908475, "hyperparameters": {"n_epochs": 1, "context_length": 512}, "time_since_job_start": 241.6144163608551}
info

If you provide a validation dataset, the job reports perplexity and validation loss at the end of each training epoch. For lines that don't correspond to the end of each epoch, the value for these metrics are null.