Skip to main content

LLM Model API Reference (0.26.46)

View Markdown

LLM Model API Reference (0.26.46)

note

This is archived documentation for version 0.26.46. For the current documentation, see current documentation.

Customer-hosted cloud features​

note

Some features are only available on customer-hosted clouds. Reach out to support@anyscale.com for info.

LLM Model CLI​

anyscale llm model get Alpha​

warning

This command is in early development and may change. Users must be tolerant of change.

Usage

anyscale llm model get [OPTIONS]

Gets the model card for the given model ID or corresponding job ID.

Example usage:

anyscale llm model get --model-id my-model-id

anyscale llm model get --job-id job_123

Options

  • --model-id: ID for the model of interest
  • --job-id: ID for the Anyscale job corresponding to the fine-tuning run

Examples​

$ anyscale llm model get --model-id my-model-id
Output
{
    'id': 'my-model-id',
    'base_model_id': 'meta-llama/Meta-Llama-3-8B',
    'storage_uri': 'gs://my_bucket/my_folder',
    'ft_type': 'LORA',
    'cloud_id': 'cld_tffbxe9ia5phqr1unxhz4f7e1e',
    'project_id': 'prj_dqb6ha67zubz3gdlvn2tmmglb8',
    'created_at': 1725563985,
    'creator': 'test@anyscale.com',
    'job_id': 'N/A',
    'workspace_id': 'expwrk_yje3t8twim18iuta9r45gwcgcn',
    'generation_config': {
        'prompt_format': {
            'system': '<|start_header_id|>system<|end_header_id|>\n\n{instruction}<|eot_id|>',
            'assistant': '<|start_header_id|>assistant<|end_header_id|>\n\n{instruction}<|eot_id|>',
            'trailing_assistant': '<|start_header_id|>assistant<|end_header_id|>\n\n',
            'user': '<|start_header_id|>user<|end_header_id|>\n\n{instruction}<|eot_id|>',
            'bos': '<|begin_of_text|>',
            'default_system_message': '',
            'add_system_tags_even_if_message_is_empty': False,
            'system_in_user': False,
            'system_in_last_user': False,
            'strip_whitespace': True
        },
        'stopping_sequences': None
    }
}

anyscale llm model list Alpha​

warning

This command is in early development and may change. Users must be tolerant of change.

Usage

anyscale llm model list [OPTIONS]

Lists fine-tuned models available to the user.

By default, all models in all visible clouds under all visible projects to the user are listed. This is optionally filtered by project_id and/or cloud_id.

Example usage:

anyscale llm model list

anyscale llm model list --max-items 50

anyscale llm model list --cloud-id cld_123

anyscale llm model list --project-id prj_123

anyscale llm model list --cloud-id cld_123 --project-id prj_123

NOTE: If you are running this from within an Anyscale workspace, and neither cloud_id nor project_id are provided, the cloud and project of the workspace will be used.

Options

  • --cloud-id: Cloud ID to filter by. If not specified, all models from all visible clouds (filtered optionally by project_id) are listed.
  • --project-id: Project ID to filter by. If not specified, all the models from all visible projects (filtered optionally by cloud_id) are listed.
  • --max-items: Maximum number of items to show in the list. By default, the 20 most recently created models are fetched.

Examples​

$ anyscale llm model list --cloud-id cld_1j41ls4gwkga4pwp8nbql6f239 --project_id prj_i4wy1t442cbe2sthxp61dmtkbh --max-items 2
Output
[
    {
        'id': 'meta-llama/Meta-Llama-3-8B-Instruct:test:bnkve',
        'base_model_id': 'meta-llama/Meta-Llama-3-8B-Instruct',
        'storage_uri': 's3://anyscale-production-data-cld-1j41ls4gwkga4pwp...',
        'ft_type': 'LORA',
        'cloud_id': 'cld_1j41ls4gwkga4pwp8nbql6f239',
        'project_id': 'prj_i4wy1t442cbe2sthxp61dmtkbh',
        'created_at': 1725572462,
        'creator': 'test@anyscale.com',
        'job_id': 'N/A',
        'workspace_id': 'expwrk_bqld1y579g3clukr49rsnd7i5m',
        'generation_config': '{"prompt_format": {"system": "<|start_header_id|>s...'
    },
    {
        'id': 'neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8:test:czcal',
        'base_model_id': 'neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8',
        'storage_uri': 'gs://storage-bucket-cld-tffbxe9ia5phqr1unxhz4f7e1e...',
        'ft_type': 'LORA',
        'cloud_id': 'cld_1j41ls4gwkga4pwp8nbql6f239',
        'project_id': 'prj_i4wy1t442cbe2sthxp61dmtkbh',
        'created_at': 1725563985,
        'creator': 'test@anyscale.com',
        'job_id': 'N/A',
        'workspace_id': 'expwrk_yje3t8twim18iuta9r45gwcgcn',
        'generation_config': '{"prompt_format": {"system": "<|start_header_id|>s...'
    }
]

anyscale llm model delete Alpha​

warning

This command is in early development and may change. Users must be tolerant of change.

Usage

anyscale llm model delete [OPTIONS] MODEL_ID

Deletes the model for the given model ID. Requires owner permission for the corresponding Anyscale project.

MODEL_ID = ID for the model of interest

Example usage:

anyscale llm model delete my-model-id

Options

Examples​

$ anyscale llm model delete --model-id my-model-id
Output
{'id': 'my-model-id', 'deleted_at': 1725572462}

LLM Model SDK​

anyscale.llm.model.get​

Retrieves model card for a finetuned model.

Arguments

  • model_id (str | None) = None: ID of the finetuned model that is being retrieved.
  • job_id (str | None) = None: ID of the Anyscale job corresponding to the fine-tuning run.

Returns: FineTunedModel

Examples​

import anyscale

anyscale.llm.model.get(model_id="my-model-id")
anyscale.llm.model.get(job_ib="prodjob_123")

anyscale.llm.model.list​

Lists fine-tuned models available to the user.

By default, all models in all visible clouds under all visible projects to the user are listed. This is optionally filtered by project_id and/or cloud_id.

Arguments

  • cloud_id (str | None) = None: Cloud ID to filter by. If not specified, all models from all the clouds visible to the user (filtered optionally by project_id) are listed.
  • project_id (str | None) = None: Project ID to filter by. If not specified, all the models from all visible projects to the user (filtered optionally by cloud_id) are listed.
  • max_items (int) = 20: Maximum number of items to show in the list. By default, the 20 most recently created models are fetched.

Returns: List[FineTunedModel]

Examples​

import anyscale

anyscale.llm.model.list()
anyscale.llm.model.list(cloud_id="cld_123")
anyscale.llm.model.list(project_id="prj_123")
anyscale.llm.model.list(cloud_id="cld_123", project_id="prj_123")
anyscale.llm.model.list(project_id="prj_123", max_items=10)

anyscale.llm.model.delete​

Deletes a finetuned model. Requires owner permission for the corresponding Anyscale project.

Arguments

  • model_id (str): ID of the finetuned model to delete.

Returns: DeletedFineTunedModel

Examples​

import anyscale

anyscale.llm.model.delete(model_id="my-model-id")

LLM Model Models​

FineTunedModel​

Represents a fine-tuned model with its associated metadata.

Fields​

  • id (str): Unique ID/tag for the fine-tuned model.
  • base_model_id (str): Base model ID used for fine-tuning.
  • cloud_id (str): ID for the Anyscale Cloud corresponding to the fine-tuning run.
  • created_at (int): Time at which the fine-tuned model was created.
  • creator (str | None): Email address for the user who created the model.
  • ft_type (FineTuningType): Fine-tuning type.
  • generation_config (Dict[str, Any] | None): Inference generation config with chat-templating parameters and stopping sequences.
  • job_id (str | None): ID for the Anyscale job corresponding to the fine-tuning run, if applicable.
  • project_id (str | None): ID for the Anyscale Project corresponding to the fine-tuning run.
  • storage_uri (str): URI at which the fine-tuned model checkpoint is stored.
  • workspace_id (str | None): ID for the Anyscale Workspace in which the model was fine-tuned, if any.

Python Methods​

def to_dict(self) -> Dict[str, Any]
    """Return a dictionary representation of the model."""

Examples​

import anyscale
from anyscale.llm.model.models import FineTunedModel
model: FineTunedModel = anyscale.llm.model.get(model_id="my-model-id")

FineTuningType​

An enumeration.

Values​

  • LORA: Low-Rank Adaptation (LoRA) fine-tuning method.
  • FULL_PARAM: Full parameter fine-tuning method.

DeletedFineTunedModel​

Represents a deleted fine-tuned model with its deletion metadata.

Fields​

  • id (str): Unique ID/tag for the fine-tuned model.
  • deleted_at (int): Unix timestamp (in seconds) at which the fine-tuned model was deleted.

Python Methods​

def to_dict(self) -> Dict[str, Any]
    """Return a dictionary representation of the model."""

Examples​

import anyscale
from anyscale.llm.model.models import DeletedFineTunedModel
deleted_model: DeletedFineTunedModel = anyscale.llm.model.delete("my-model-id")