Fine tuning: end-to-end example
Effective August 1, 2024 Anyscale Endpoints API will be available exclusively through the fully Hosted Anyscale Platform. Multi-tenant access to LLM models will be removed.
With the Hosted Anyscale Platform, you can access the latest GPUs billed by the second, and deploy models on your own dedicated instances. Enjoy full customization to build your end-to-end applications with Anyscale. Get started today.
Goals
By the end of this example you should be able to:
- Convert your domain specific data into the appropriate JSON format.
- Fine tune either Llama 7B or 13B or 70B on Anyscale Endpoints.
- Use a fine tuned model using the standard Anyscale Endpoint APIs.
Prerequisites
Before starting this example, you should have:
- An Anyscale Endpoints API Key
- A set of examples you want to improve performance on in your preferred format
Previous experience with Anyscale Endpoints is helpful.
This guide uses the OpenAI SDK in Python to do this. If you’ve done fine tuning on OpenAI before, the following should look familiar to you.
Anyscale Endpoints fine-tuning design principles
One design principle of Anyscale Endpoints is to be as compatible as possible with the OpenAI approach, to make it easier for users transitioning from OpenAI. This approach includes:
- API calls that are similar. You can use the OpenAI Python SDK and only change a few environment variables.
- The file format for fine tuning is the same.
Outline of stages
The general process for fine tuning is as follows:
- Collect a set of system prompt, user prompt, and assistant responses that represent your desired patterns.
- Convert these responses into training data in the OpenAI prompt format. Optionally, also convert validation data.
- Upload the training data to Anyscale Endpoints. Uploading the validation data is also an option.
- Start a fine tuning job that refers to the base model and the training data.
- Receive notification of a completed fine tuning job.
- Use the fine tuned model.
0. Download training and validation files
Ready-to-use training and validation files are available for you. Directly download them and skip step 1 and 2.
curl -o train.jsonl https://gist.githubusercontent.com/robertnishihara/ac613ff0404487fbf115cf9c0224080e/raw/80a1af0b1c2940a03830c477f7f722323b77472e/train.jsonl
curl -o valid.jsonl https://gist.githubusercontent.com/robertnishihara/ac613ff0404487fbf115cf9c0224080e/raw/80a1af0b1c2940a03830c477f7f722323b77472e/valid.jsonl
1. Collecting the desired patterns
This example tests your fine tuning on the SPIDER dataset for translating natural language to SQL. You can download the SPIDER from: https://yale-lily.github.io/spider. See Spider Dataset
in “Getting Started”.
Examine the input data from SPIDER:
{
"db_id": "department_management",
"question": "How many heads of the departments are older than 56 ?",
"query": "SELECT count(*) FROM head WHERE age > 56",
...
}, ...
You want to convert into the form used for fine tuning. The data you want to produce should generally consist of a triple that looks like the following:
[{"role": "system",
"content": "You are a helpful assistant that helps people convert natural language to SQL. You are given the database format using a CREATE command, and your goal is to convert this to a SQL query that is a SELECT command."
},
{"role": "user",
"content": "The database is department_management. Convert the following to a SQL command: How many heads of the departments are older than 56?"
},
{ "role": "assistant",
"content": SELECT count(*) FROM head WHERE age >