Service API Reference
These docs are for the new Anyscale design. If you started using Anyscale before April 2024, use Version 1.0.0 of the docs. If you're transitioning to Anyscale Preview, see the guide for how to migrate.
Customer-hosted cloud features
Some features are only available on customer-hosted clouds. Reach out to preview-help@anyscale.com for info.
Service Models
ServiceConfig
Configuration options for a service.
Fields
name
(str | None): Unique name of the service. When running inside a workspace, this defaults to the name of the workspace, else it is required.image_uri
(str | None): URI of an existing image. Exclusive withcontainerfile
.containerfile
(str | None): The file path to a containerfile that will be built into an image before running the workload. Exclusive withimage_uri
.compute_config
(ComputeConfig | Dict | str | None): The name of an existing registered compute config or an inlined ComputeConfig object.working_dir
(str | None): Directory that will be used as the working directory for the application. If a local directory is provided, it will be uploaded to cloud storage automatically. When running inside a workspace, this defaults to the current working directory ('.').excludes
(List[str] | None): A list of file path globs that will be excluded when uploading local files forworking_dir
.requirements
(str | List[str] | None): A list of pip requirements or a path to arequirements.txt
file for the workload. When running inside a workspace, this defaults to the workspace-tracked requirements.env_vars
(Dict[str, str] | None): A dictionary of environment variables that will be set for the workload.py_modules
(List[str] | None): A list of local directories or remote URIs that will be uploaded and added to the Python path.cloud
(str | None): The Anyscale Cloud to run this workload on. If not provided, the organization default will be used (or, if running in a workspace, the cloud of the workspace).project
(str | None): The project for the workload. If not provided, the default project for the cloud will be used (or, if running in a workspace, the project of the workspace).registry_login_secret
(str | None): A name or identifier of the secret containing credentials to authenticate to the docker registry hosting the image. This can only be used when 'image_uri' is specified and the image is not hosted on Anyscale.ray_version
(str | None): The Ray version (X.Y.Z) specified for this image specified by either an image URI or a containerfile. If not provided, the latest Ray version will be used.applications
(List[Dict[str, Any]]): List of Ray Serve applications to run. At least one application must be specified. For details, see the Ray Serve config file format documentation: https://docs.ray.io/en/latest/serve/production-guide/config.htmlquery_auth_token_enabled
(bool): Whether or not queries to this service is gated behind an authentication token. IfTrue
, an auth token is generated the first time the service is deployed. You can find the token in the UI or by fetching the status of the service.http_options
(Dict[str, Any] | None): HTTP options that will be passed to Ray Serve. See https://docs.ray.io/en/latest/serve/production-guide/config.html for supported options.grpc_options
(Dict[str, Any] | None): gRPC options that will be passed to Ray Serve. See https://docs.ray.io/en/latest/serve/production-guide/config.html for supported options.logging_config
(Dict[str, Any] | None): Logging options that will be passed to Ray Serve. See https://docs.ray.io/en/latest/serve/production-guide/config.html for supported options.ray_gcs_external_storage_config
(RayGCSExternalStorageConfig | Dict | None): Configuration options for external storage for the Ray Global Control Store (GCS).- Only available on customer-hosted clouds.
tracing_config
(TracingConfig | Dict | None): Configuration options for tracing.- Only available on customer-hosted clouds.
Python Methods
def __init__(self, **fields) -> ServiceConfig
"""Construct a model with the provided field values set."""
def options(self, **fields) -> ServiceConfig
"""Return a copy of the model with the provided field values overwritten."""
def to_dict(self) -> Dict[str, Any]
"""Return a dictionary representation of the model."""
Examples
- YAML
- Python
name: my-service
working_dir: .
applications:
- import_path: main:app
# An inline dictionary can also be provided.
compute_config: my-compute-config:1
# A containerfile path can also be provided.
image_uri: anyscale/image/my-image:1
from anyscale.service.models import ServiceConfig
config = ServiceConfig(
name="my-service",
working_dir=".",
applications=[{"import_path": "main:app"}],
# An inline `ComputeConfig` can also be provided.
compute_config="my-compute-config:1",
# A containerfile path can also be provided.
image_uri="anyscale/image/my-image:1",
)
RayGCSExternalStorageConfig
Configuration options for external storage for the Ray Global Control Store (GCS).
When external storage is enabled for a service, it will be able to continue serving traffic if the head node goes down. Without external storage, the entire cluster will be restarted.
Fields
enabled
(bool): Enable or disable external storage. WhenFalse
, the other fields are ignored.address
(str | None): Address to connect to the redis instance at. Defaults to the cloud-wide configuration.certificate_path
(str | None): Path to the TLS certificate file to use for authentication when using secure connections. Defaults to the cloud-wide configuration.
Python Methods
def __init__(self, **fields) -> RayGCSExternalStorageConfig
"""Construct a model with the provided field values set."""
def options(self, **fields) -> RayGCSExternalStorageConfig
"""Return a copy of the model with the provided field values overwritten."""
def to_dict(self) -> Dict[str, Any]
"""Return a dictionary representation of the model."""
Examples
- YAML
- Python
name: my-service
applications:
- import_path: main:app
ray_gcs_external_storage_config:
enabled: true
address: http://redis-address:8888
certificate_path: /etc/ssl/certs/ca-certificates.crt
from anyscale.service.models import ServiceConfig, RayGCSExternalStorageConfig
config = ServiceConfig(
name="my-service",
applications=[{"import_path": "main:app"}],
ray_gcs_external_storage_config=RayGCSExternalStorageConfig(
enabled=True,
address="http://redis-address:8888",
certificate_path="/etc/ssl/certs/ca-certificates.crt",
),
)
TracingConfig
Configuration options for tracing.
Tracing provides a way to collect telemetry on requests that are
handled by your Serve application. If tracing is enabled, you can instrument
your Serve application and export traces to a specific backend by configuring the
exporter_import_path
option. This feature is currently experimental.
Fields
enabled
(bool): Flag to enable tracing. WhenTrue
and noexporter_import_path
is defined, then the default tracing exporter will be used to export traces to serve logs. WhenFalse
, the other fields are ignored.exporter_import_path
(str | None): Path to exporter function. Should be of the form "module.submodule_1...submodule_n.export_tracing". This is equivalent to "from module.submodule_1...submodule_n import export_tracing. Exporter function takes no arguments and returns List[SpanProcessor].sampling_ratio
(float): Ratio of traces to record and export. Reducing the ratio can help with minimizing serving latency and storage overhead.
Python Methods
def __init__(self, **fields) -> TracingConfig
"""Construct a model with the provided field values set."""
def options(self, **fields) -> TracingConfig
"""Return a copy of the model with the provided field values overwritten."""
def to_dict(self) -> Dict[str, Any]
"""Return a dictionary representation of the model."""
Examples
- YAML
- Python
name: my-service
applications:
- import_path: main:app
tracing_config:
enabled: true
exporter_import_path: "my_module:custom_tracing_exporter"
sampling_ratio: 1.0
from anyscale.service.models import ServiceConfig, TracingConfig
config = ServiceConfig(
name="my-service",
applications=[{"import_path": "main:app"}],
tracing_config=TracingConfig(
enabled=True,
exporter_import_path="my_module:custom_tracing_exporter",
sampling_ratio=1.0,
),
)
ServiceStatus
Current status of a service.
Fields
name
(str): Unique name of the service.id
(str): Unique ID of the service (generated when the service is first deployed).state
(ServiceState | str): Current state of the service.query_url
(str): URL used to query the service.query_auth_token
(str | None): Query auth token required to query the service (if any). Must be passed in as a header like:Authorization: Bearer <token>
.primary_version
(ServiceVersionStatus | Dict | None): Primary version of the service. During a rollout, this is the service version that was running prior to the deploy.canary_version
(ServiceVersionStatus | Dict | None): Canary version of the service (the one that is being deployed). When there is no rollout ongoing, this will beNone
.
Python Methods
def to_dict(self) -> Dict[str, Any]
"""Return a dictionary representation of the model."""
Examples
- Python
- CLI
import anyscale
from anyscale.service.models import ServiceStatus
status: ServiceStatus = anyscale.service.status("my-service")
$ anyscale service status -n my-service
name: my-service
id: service2_uz6l8yhy2as5wrer3shzj6kh67
state: RUNNING
primary_version:
id: 601bd56c4b
state: RUNNING
weight: 100
ServiceState
Current state of a service.
Values
STARTING
: The service is starting up.RUNNING
: The service is running and healthy.UPDATING
: One of the service versions or the networking stack is updating.ROLLING_OUT
: The service is rolling out to a new canary version.ROLLING_BACK
: The service is rolling back to the primary version.UNHEALTHY
: The service has become unhealthy due to an unexpected issue such as a node failure. In most cases these issues should automatically recover.TERMINATING
: The service is terminating.TERMINATED
: The service is terminated.UNKNOWN
: The CLI/SDK received an unexpected state from the API server. In most cases, this means you need to update the CLI.SYSTEM_FAILURE
: An unexpected error occurred in the Anyscale stack. Please reach out to customer support immediately.
ServiceVersionStatus
Current status of a single version of a running service.
Fields
id
(str): Unique ID of the service version (not the service).state
(ServiceVersionState | str): Current state of the service version.weight
(int): Current canary weight of the service version. This is100
if there is no ongoing rollout.config
(ServiceConfig | Dict): Configuration of this service version.
Python Methods
def to_dict(self) -> Dict[str, Any]
"""Return a dictionary representation of the model."""
Examples
- Python
- CLI
import anyscale
from anyscale.service.models import ServiceStatus, ServiceVersionStatus
status: ServiceStatus = anyscale.service.status("my-service")
primary_version_status: ServiceVersionStatus = status.primary_version
$ anyscale service status -n my-service
name: my-service
id: service2_uz6l8yhy2as5wrer3shzj6kh67
state: RUNNING
primary_version:
id: 601bd56c4b
state: RUNNING
weight: 100
ServiceVersionState
Current state of a service version.
Values
STARTING
: The service version is starting up.UPDATING
: The application(s) on the service version are updating due to a config deploy or update.RUNNING
: The service version is running and healthy.UNHEALTHY
: The service version has become unhealthy due to an unexpected issue such as a node failure. In most cases these issues should automatically recover.TERMINATING
: The service version is terminating.TERMINATED
: The service version is terminated.UNKNOWN
: The CLI/SDK received an unexpected state from the API server. In most cases, this means you need to update the CLI.SYSTEM_FAILURE
: An unexpected error occurred in the Anyscale stack. Please reach out to customer support immediately.
Service CLI
anyscale service deploy
Usage
anyscale service deploy [OPTIONS] [IMPORT_PATH] [ARGUMENTS]...
Deploy or update a service.
Options
-f/--config-file
: Path to a YAML config file to deploy. When deploying from a file, import path and arguments cannot be provided. Command-line flags will overwrite values read from the file.-n/--name
: Unique name for the service. When running in a workspace, this defaults to the workspace name.--image-uri
: Container image to use for the service. This cannot be changed when using --in-place and is exclusive with --containerfile. When running in a workspace, this defaults to the image of the workspace.--registry-login-secret
: Name or identifier of the secret containing credentials to authenticate to the docker registry hosting the image. This can only be used when 'image_uri' is specified and the image is not hosted on Anyscale.--containerfile
: Path to a containerfile to build the image to use for the service. This cannot be changed when using --in-place and is exclusive with --image-uri.--ray-version
: The Ray version (X.Y.Z) to the image specified by --image-uri. This is only used when --image-uri is provided. If not provided, the latest Ray version will be used.--compute-config
: Named compute configuration to use for the service. This cannot be changed when using --in-place. When running in a workspace, this defaults to the compute configuration of the workspace.-w/--working-dir
: Path to a local directory that will be the working directory for the service. The files in the directory will be automatically uploaded to cloud storage. When running in a workspace, this defaults to the current working directory.-e/--exclude
: File pattern to exclude when uploading local directories. This argument can be specified multiple times and the patterns will be appended to the 'excludes' list in the config file (if any).-r/--requirements
: Path to a requirements.txt file containing dependencies for the service. These will be installed on top of the image. When running in a workspace, this defaults to the workspace dependencies.-i/--in-place
: Perform an in-place upgrade without starting a new cluster. This can be used for faster iteration during development but is not currently recommended for production deploys. This cannot be used to change cluster-level options such as image and compute config (they will be ignored).--canary-percent
: The percentage of traffic to send to the canary version of the service (0-100). This can be used to manually shift traffic toward (or away from) the canary version. If not provided, traffic will be shifted incrementally toward the canary version until it reaches 100. Not supported when using --in-place.--max-surge-percent
: Amount of excess capacity allowed to be used while updating the service (0-100). Defaults to 100. Not supported when using --in-place.--env
: Environment variables to set for the service. The format is 'key=value'. This argument can be specified multiple times. When the same key is also specified in the config file, the value from the command-line flag will overwrite the value from the config file.--py-module
: Python modules to be available for import in the Ray workers. Each entry must be a path to a local directory.--cloud
: The Anyscale Cloud to run this workload on. If not provided, the organization default will be used (or, if running in a workspace, the cloud of the workspace).--project
: Named project to use for the service. If not provided, the default project for the cloud will be used (or, if running in a workspace, the project of the workspace).
anyscale service status
Usage
anyscale service status [OPTIONS]
Get the status of a service.
Options
-n/--name
: Name of the service.-f/--config-file
: Path to a YAML config file to read the name from.--cloud
: The Anyscale Cloud to run this workload on. If not provided, the organization default will be used (or, if running in a workspace, the cloud of the workspace).--project
: Named project to use for the service. If not provided, the default project for the cloud will be used (or, if running in a workspace, the project of the workspace).-j/--json
: Output the status in a structured JSON format.-v/--verbose
: Include verbose details in the status.
anyscale service wait
Usage
anyscale service wait [OPTIONS]
Wait for a service to enter a target state.
Options
-n/--name
: Name of the service.-f/--config-file
: Path to a YAML config file to read the name from.--cloud
: The Anyscale Cloud to run this workload on. If not provided, the organization default will be used (or, if running in a workspace, the cloud of the workspace).--project
: Named project to use for the service. If not provided, the default project for the cloud will be used (or, if running in a workspace, the project of the workspace).-s/--state
: The ServiceState to wait for the service to reach. Defaults to RUNNING.-t/--timeout-s
: Timeout to wait for the service to reach the target state. Defaults to 600s (10min).
anyscale service rollback
Usage
anyscale service rollback [OPTIONS]
Roll back a service.
Options
-n/--name
: Name of service.-f/--config-file/--service-config-file
: Path to a YAML config file to read the name from.--service-config-file
is deprecated, use-f
or--config-file
.--max-surge-percent
: Max amount of excess capacity allocated during the rollback (0-100).
anyscale service terminate
Usage
anyscale service terminate [OPTIONS]
Terminate a service.
Options
-n/--name
: Name of service.-f/--config-file/--service-config-file
: Path to a YAML config file to read the name from.--service-config-file
is deprecated, use-f
or--config-file
.
Service SDK
anyscale.service.deploy
Deploy a service.
If no service with the provided name is running, one will be created, else the existing service will be updated.
This command is asynchronous, so it always returns immediately.
Returns the id of the deployed service.
Arguments
config
(ServiceConfig): The config options defining the service.in_place
(bool) = False: Perform an in-place upgrade without starting a new cluster. This can be used for faster iteration during development but is not currently recommended for production deploys. This cannot be used to change cluster-level options such as image and compute config (they will be ignored).canary_percent
(int | None) = None: The percentage of traffic to send to the canary version of the service (0-100). This can be used to manually shift traffic toward (or away from) the canary version. If not provided, traffic will be shifted incrementally toward the canary version until it reaches 100. Not supported when using --in-place.max_surge_percent
(int | None) = None: Amount of excess capacity allowed to be used while updating the service (0-100). Defaults to 100. Not supported when using --in-place.
Returns: str
Examples
- Python
import anyscale
from anyscale.service.models import ServiceConfig
anyscale.service.deploy(
ServiceConfig(
name="my-service",
applications=[
{"import_path": "main:app"},
],
working_dir=".",
),
canary_percent=50,
)
anyscale.service.status
Get the status of a service.
Arguments
name
(str): Name of the service.cloud
(str | None) = None: The Anyscale Cloud to run this workload on. If not provided, the organization default will be used (or, if running in a workspace, the cloud of the workspace).project
(str | None) = None: Named project to use for the service. If not provided, the default project for the cloud will be used (or, if running in a workspace, the project of the workspace).
Returns: ServiceStatus
Examples
- Python
import anyscale
from anyscale.service.models import ServiceStatus
status: ServiceStatus = anyscale.service.status(name="my-service")
anyscale.service.wait
Wait for a service to reach a target state.
Arguments
name
(str): Name of the service.cloud
(str | None) = None: The Anyscale Cloud to run this workload on. If not provided, the organization default will be used (or, if running in a workspace, the cloud of the workspace).project
(str | None) = None: Named project to use for the service. If not provided, the default project for the cloud will be used (or, if running in a workspace, the project of the workspace).state
(str | ServiceState) = RUNNING: The state to wait for the service to reach.timeout_s
(float) = 600: Timeout to wait for the service to reach the target state.
Examples
- Python
import anyscale
from anyscale.service.models import ServiceState
anyscale.service.wait(name="my-service", state=ServiceState.RUNNING)
anyscale.service.rollback
Rollback to the primary version of the service.
This command can only be used when there is an active rollout in progress. The rollout will be cancelled and the service will revert to the primary version.
This command is asynchronous, so it always returns immediately.
Returns the id of the rolled back service.
Arguments
name
(str | None): Name of the service. When running in a workspace, this defaults to the workspace name.cloud
(str | None) = None: The Anyscale Cloud to run this workload on. If not provided, the organization default will be used (or, if running in a workspace, the cloud of the workspace).project
(str | None) = None: Named project to use for the service. If not provided, the default project for the cloud will be used (or, if running in a workspace, the project of the workspace).max_surge_percent
(int | None) = None: Amount of excess capacity allowed to be used while rolling back to the primary version of the service (0-100). Defaults to 100.
Returns: str
Examples
- Python
import anyscale
anyscale.service.rollback(name="my-service")
anyscale.service.terminate
Terminate a service.
This command is asynchronous, so it always returns immediately.
Returns the id of the terminated service.
Arguments
name
(str | None): Name of the service. When running in a workspace, this defaults to the workspace name.cloud
(str | None) = None: The Anyscale Cloud to run this workload on. If not provided, the organization default will be used (or, if running in a workspace, the cloud of the workspace).project
(str | None) = None: Named project to use for the service. If not provided, the default project for the cloud will be used (or, if running in a workspace, the project of the workspace).
Returns: str
Examples
- Python
import anyscale
anyscale.service.terminate(name="my-service")