LLM Model API Reference

Customer-hosted cloud features

note

Some features are only available on customer-hosted clouds. Reach out to support@anyscale.com for info.

LLM Model CLI

`anyscale llm model get` Alpha

warning

This command is in early development and may change. Users must be tolerant of change.

Usage

anyscale llm model get [OPTIONS]

Gets the model card for the given model ID or corresponding job ID.

Example usage:

anyscale llm model get --model-id my-model-id

anyscale llm model get --job-id job_123

Options

--model-id: ID for the model of interest
--job-id: ID for the Anyscale job corresponding to the fine-tuning run

Examples

$ anyscale llm model get --model-id my-model-id
Output
{
    'id': 'my-model-id',
    'base_model_id': 'meta-llama/Meta-Llama-3-8B',
    'storage_uri': 'gs://my_bucket/my_folder',
    'ft_type': 'LORA',
    'cloud_id': 'cld_tffbxe9ia5phqr1unxhz4f7e1e',
    'project_id': 'prj_dqb6ha67zubz3gdlvn2tmmglb8',
    'created_at': 1725563985,
    'creator': 'test@anyscale.com',
    'job_id': 'N/A',
    'workspace_id': 'expwrk_yje3t8twim18iuta9r45gwcgcn',
    'generation_config': {
        'prompt_format': {
            'system': '<|start_header_id|>system<|end_header_id|>\n\n{instruction}<|eot_id|>',
            'assistant': '<|start_header_id|>assistant<|end_header_id|>\n\n{instruction}<|eot_id|>',
            'trailing_assistant': '<|start_header_id|>assistant<|end_header_id|>\n\n',
            'user': '<|start_header_id|>user<|end_header_id|>\n\n{instruction}<|eot_id|>',
            'bos': '<|begin_of_text|>',
            'default_system_message': '',
            'add_system_tags_even_if_message_is_empty': False,
            'system_in_user': False,
            'system_in_last_user': False,
            'strip_whitespace': True
        },
        'stopping_sequences': None
    }
}

`anyscale llm model list` Alpha

warning

This command is in early development and may change. Users must be tolerant of change.

Usage

anyscale llm model list [OPTIONS]

Lists fine-tuned models available to the user.

By default, all models in all visible clouds under all visible projects to the user are listed. This is optionally filtered by project_id and/or cloud_id.

Example usage:

anyscale llm model list

anyscale llm model list --max-items 50

anyscale llm model list --cloud-id cld_123

anyscale llm model list --project-id prj_123

anyscale llm model list --cloud-id cld_123 --project-id prj_123

NOTE: If you are running this from within an Anyscale workspace, and neither cloud_id nor project_id are provided, the cloud and project of the workspace will be used.

Options

--cloud-id: Cloud ID to filter by. If not specified, all models from all visible clouds (filtered optionally by project_id) are listed.
--max-items: Maximum number of items to show in the list. By default, the 20 most recently created models are fetched.

Examples

$ anyscale llm model list --cloud-id cld_1j41ls4gwkga4pwp8nbql6f239 --project_id prj_i4wy1t442cbe2sthxp61dmtkbh --max-items 2
Output
[
    {
        'id': 'meta-llama/Meta-Llama-3-8B-Instruct:test:bnkve',
        'base_model_id': 'meta-llama/Meta-Llama-3-8B-Instruct',
        'storage_uri': 's3://anyscale-production-data-cld-1j41ls4gwkga4pwp...',
        'ft_type': 'LORA',
        'cloud_id': 'cld_1j41ls4gwkga4pwp8nbql6f239',
        'project_id': 'prj_i4wy1t442cbe2sthxp61dmtkbh',
        'created_at': 1725572462,
        'creator': 'test@anyscale.com',
        'job_id': 'N/A',
        'workspace_id': 'expwrk_bqld1y579g3clukr49rsnd7i5m',
        'generation_config': '{"prompt_format": {"system": "<|start_header_id|>s...'
    },
    {
        'id': 'neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8:test:czcal',
        'base_model_id': 'neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8',
        'storage_uri': 'gs://storage-bucket-cld-tffbxe9ia5phqr1unxhz4f7e1e...',
        'ft_type': 'LORA',
        'cloud_id': 'cld_1j41ls4gwkga4pwp8nbql6f239',
        'project_id': 'prj_i4wy1t442cbe2sthxp61dmtkbh',
        'created_at': 1725563985,
        'creator': 'test@anyscale.com',
        'job_id': 'N/A',
        'workspace_id': 'expwrk_yje3t8twim18iuta9r45gwcgcn',
        'generation_config': '{"prompt_format": {"system": "<|start_header_id|>s...'
    }
]

`anyscale llm model delete` Alpha

warning

This command is in early development and may change. Users must be tolerant of change.

Usage

anyscale llm model delete [OPTIONS] MODEL_ID

Deletes the model for the given model ID. Requires owner permission for the corresponding Anyscale project.

MODEL_ID = ID for the model of interest

Example usage:

anyscale llm model delete my-model-id

Options

Examples

$ anyscale llm model delete --model-id my-model-id
Output
{'id': 'my-model-id', 'deleted_at': 1725572462}

LLM Model SDK

`anyscale.llm.model.get`

Retrieves model card for a finetuned model.

Arguments

model_id (str | None) = None: ID of the finetuned model that is being retrieved.
job_id (str | None) = None: ID of the Anyscale job corresponding to the fine-tuning run.

Returns: FineTunedModel

Examples

Python

import anyscale

anyscale.llm.model.get(model_id="my-model-id")
anyscale.llm.model.get(job_ib="prodjob_123")

`anyscale.llm.model.list`

Lists fine-tuned models available to the user.

By default, all models in all visible clouds under all visible projects to the user are listed. This is optionally filtered by project_id and/or cloud_id.

Arguments

cloud_id (str | None) = None: Cloud ID to filter by. If not specified, all models from all the clouds visible to the user (filtered optionally by project_id) are listed.
project_id (str | None) = None: Project ID to filter by. If not specified, all the models from all visible projects to the user (filtered optionally by cloud_id) are listed.
max_items (int) = 20: Maximum number of items to show in the list. By default, the 20 most recently created models are fetched.

Returns: List[FineTunedModel]

Examples

Python

import anyscale

anyscale.llm.model.list()
anyscale.llm.model.list(cloud_id="cld_123")
anyscale.llm.model.list(project_id="prj_123")
anyscale.llm.model.list(cloud_id="cld_123", project_id="prj_123")
anyscale.llm.model.list(project_id="prj_123", max_items=10)

`anyscale.llm.model.delete`

Deletes a finetuned model. Requires owner permission for the corresponding Anyscale project.

Arguments

model_id (str): ID of the finetuned model to delete.

Returns: DeletedFineTunedModel

Examples

Python

import anyscale

anyscale.llm.model.delete(model_id="my-model-id")

LLM Model Models

`FineTunedModel`

Represents a fine-tuned model with its associated metadata.

Fields

id (str): Unique ID/tag for the fine-tuned model.
base_model_id (str): Base model ID used for fine-tuning.
cloud_id (str): ID for the Anyscale Cloud corresponding to the fine-tuning run.
created_at (int): Time at which the fine-tuned model was created.
creator (str | None): Email address for the user who created the model.
ft_type (FineTuningType): Fine-tuning type.
generation_config (Dict[str, Any] | None): Inference generation config with chat-templating parameters and stopping sequences.
job_id (str | None): ID for the Anyscale job corresponding to the fine-tuning run, if applicable.
project_id (str | None): ID for the Anyscale Project corresponding to the fine-tuning run.
storage_uri (str): URI at which the fine-tuned model checkpoint is stored.
workspace_id (str | None): ID for the Anyscale Workspace in which the model was fine-tuned, if any.

Python Methods

def to_dict(self) -> Dict[str, Any]
    """Return a dictionary representation of the model."""

Examples

Python
CLI

import anyscale
from anyscale.llm.model.models import FineTunedModel
model: FineTunedModel = anyscale.llm.model.get(model_id="my-model-id")

$ anyscale llm model get --model-id my-model-id
id: my-model-id
base_model_id: meta-llama/Llama-3-8B-Instruct
cloud_id: cloud_abc123
created_at: 1725473924
job_id: prodjob_xyz789
ft_type: LORA
...

`FineTuningType`

An enumeration.

Values

LORA: Low-Rank Adaptation (LoRA) fine-tuning method.
FULL_PARAM: Full parameter fine-tuning method.

`DeletedFineTunedModel`

Represents a deleted fine-tuned model with its deletion metadata.

Fields

id (str): Unique ID/tag for the fine-tuned model.
deleted_at (int): Unix timestamp (in seconds) at which the fine-tuned model was deleted.

Python Methods

def to_dict(self) -> Dict[str, Any]
    """Return a dictionary representation of the model."""

Examples

Python
CLI

import anyscale
from anyscale.llm.model.models import DeletedFineTunedModel
deleted_model: DeletedFineTunedModel = anyscale.llm.model.delete("my-model-id")

$ anyscale models get-deleted my-model-id
id: my-model-id
deleted_at: 1725473924

Customer-hosted cloud features​

LLM Model CLI​

anyscale llm model get Alpha​

Examples​

anyscale llm model list Alpha​

Examples​

anyscale llm model delete Alpha​

Examples​

LLM Model SDK​

anyscale.llm.model.get​

Examples​

anyscale.llm.model.list​

Examples​

anyscale.llm.model.delete​

Examples​

LLM Model Models​

FineTunedModel​

Fields​

Python Methods​

Examples​

FineTuningType​

Values​

DeletedFineTunedModel​

Fields​

Python Methods​

Examples​

Customer-hosted cloud features

LLM Model CLI

`anyscale llm model get` Alpha

Examples

`anyscale llm model list` Alpha

Examples

`anyscale llm model delete` Alpha

Examples

LLM Model SDK

`anyscale.llm.model.get`

Examples

`anyscale.llm.model.list`

Examples

`anyscale.llm.model.delete`

Examples

LLM Model Models

`FineTunedModel`

Fields

Python Methods

Examples

`FineTuningType`

Values

`DeletedFineTunedModel`

Fields

Python Methods

Examples