Quickstart with default mode
LLMForge supports a default mode for users to start fine-tuning quickly.
Similar to OpenAI's fine-tuning experience, the default mode provides a minimal and efficient setup. It allows you to quickly start a fine-tuning job by setting just a few parameters, model_id and train_path. All other settings are optional and Anyscale automatically selects them based on dataset statistics and predefined configurations.
For example, given a file config.yaml containing the following:
model_id: meta-llama/Meta-Llama-3-8B-Instruct
train_path: s3://...
you can launch a fine-tuning job simply by running:
llmforge anyscale finetune config.yaml --default
Default mode vs custom mode
While default mode sets reasonable parameters for getting started quickly, custom mode offers more flexibility and control over the fine-tuning process, allowing for advanced optimizations and customizations. You need to provide more configurations to set up this mode, for example prompt format, hardware, batch size, etc.
The following is a full comparison of the two modes:
| Feature | Default Mode | Custom Mode |
|---|---|---|
| Ideal For | Prototyping what's possible, focusing on dataset cleaning, fine-tuning, and evaluation pipeline | Optimizing model quality by controlling more parameters, hardware control |
| Command | llmforge anyscale finetune config.yaml --default | llmforge anyscale finetune config.yaml |
| Model Support | Popular models with their prompt format, for example meta-llama/Meta-Llama-3-8B-Instruct* | Any Hugging Face model, any prompt format, for example meta-llama/Meta-Llama-Guard-2-8B |
| Task Support | Instruction tuning for multi-turn chat | Causal language modeling, Instruction tuning, Classification |
| Data Format | Supports chat-style datasets, with fixed prompt formats per model | Supports chat-style datasets, with flexible prompt format |
| Hardware | Automatically selected (limited by availability) | User-configurable |
| Fine-tuning type | Only supports LoRA (Rank-8, all linear layers) | User-defined LoRA and Full-parameter |
| Context Length | Limited to 4096 | Supports long context lengths with RoPE scaling |
*NOTE: old models will get deprecated.
Choose the mode that best fits your project requirements and level of customization needed.
Note for default mode:
- Cluster type for all models: 8xA100-80G
- Supported context length for models: 512 up to max. context length of each model in powers of 2.
Models supported in default mode
Default mode supports a select list of models, with a fixed cluster type of 8xA100-80G. For each model, LLMForge only supports context lengths of 512 up to the maximum context length in increments of 2x, that is, 512, 1024, and so on. The following are the supported models and their configurations:
| Model family | model_id | Max. context length in default mode |
|---|---|---|
| Llama-3.2 | meta-llama/Meta-Llama-3.2-1B-Instruct | 4096 |
| Llama-3.2 | meta-llama/Meta-Llama-3.2-3B-Instruct | 4096 |
| Llama-3.1 | meta-llama/Meta-Llama-3.1-8B-Instruct | 4096 |
| Llama-3.1 | meta-llama/Meta-Llama-3.1-70B-Instruct | 4096 |
| Llama-3 | meta-llama/Meta-Llama-3-8B-Instruct | 4096 |
| Llama-3 | meta-llama/Meta-Llama-3-70B-Instruct | 4096 |
| Mistral | mistralai/Mistral-Nemo-Instruct-2407 | 4096 |
| Mistral | mistralai/Mistral-7B-Instruct-v0.3 | 4096 |
| Mixtral | mistralai/Mixtral-8x7B-Instruct-v0.1 | 4096 |
Example configs
The following are some examples for getting started in default mode. To run these examples, open the fine-tuning template as a workspace on Anyscale and run the commands in the terminal. Find the example configs under ./training_configs. Outside of a workspace you can also find them on GitHub.
Fine-tune Llama-3-8B-Instruct in default mode, LoRA rank 8.
Command:
llmforge anyscale finetune training_configs/default/meta-llama/Meta-Llama-3-8B-Instruct-simple.yaml --default
Config:
model_id: meta-llama/Meta-Llama-3-8B-Instruct
train_path: s3://...
Fine-tune Llama-3-8B-Instruct in default mode with additional hyperparameters.
Command:
llmforge anyscale finetune training_configs/custom/meta-llama--Meta-Llama-3-8B-Instruct/lora/16xA10-512.yaml
Config:
model_id: meta-llama/Meta-Llama-3-8B-Instruct
train_path: s3://...
valid_path: s3://...
context_length: 512
deepspeed:
config_path: deepspeed_configs/zero_3_offload_optim+param.json
worker_resources:
accelerator_type:A10G: 0.001