Anyscale Services

note

Use of Anyscale Services requires Ray 2.3+.

Anyscale Services help you deploy your ML applications built with Ray Serve in production. Anyscale Services is designed to help you build scalable, fault tolerant applications that serve mission critical traffic 24/7 without downtime.

Key features of Anyscale Services:

1. Serve high volume HTTP traffic
2. High availability and 0 downtime upgrades
3. Autoscaling
4. Built in observability with Grafana

This walk-through will help you run your Ray Serve application as a production service on Anyscale.

Setup your environment

You can follow this guide from an Anyscale Workspace, or from your laptop.

First, make sure you have setup a cloud.

We will start with a simple "HelloWorld" app.

import os

# When the env var is updated, users see new return value.
msg = os.getenv("SERVE_RESPONSE_MESSAGE", "Hello world!")


class HelloWorld:
    def hello(self):
        return msg

The first step is to convert it into a Ray Serve application with FastAPI. Create a file named serve_hello.py and add the following content.

from fastapi import FastAPI
from ray import serve
import os

# When the env var is updated, users see new return value.
msg = os.getenv("SERVE_RESPONSE_MESSAGE", "Hello world!")

app = FastAPI()

@serve.deployment(route_prefix="/")
@serve.ingress(app)
class HelloWorld:
    @app.get("/")
    def hello(self):
        return msg

entrypoint = HelloWorld.bind()

Deploy your first Service

important

If you are running from your laptop, you will need cloud provider credentials in this section to upload local files to the storage bucket for your Anyscale Cloud.

See the documentation for configuring credentials on AWS or GCP.

To start a service, we need to first create a Service configuration file. The Service configuration file is a YAML file that is used to specify important parameters such as dependencies, compute requirements and deployment settings for your Anyscale Service.

An example service configuration file is shown below. Create a file named service-v2.yaml.

name: service-hello
cluster_env: default_cluster_env_2.9.0_py310:1
ray_serve_config:
  applications:
    - name: default
      import_path: serve_hello:entrypoint
      runtime_env:
        working_dir: .
        env_vars:
          SERVE_RESPONSE_MESSAGE: service says hello

Walking through this example, we see:

• name: This is a unique identifier for the service.
• cluster_env: This specifies the docker image and dependencies that are deployed on the Service cluster. See guide for more details on cluster environment.
• runtime_env/working_dir: This guide has details on the options for configuring the runtime environment. The working directory specifies where the code is found. By putting ., the service will be created with the current working directory by uploading it to the default artifact storage bucket configured in your cloud. Alternatively, you can specify a URI for the working directory or use a local working directory and upload it to a custom storage bucket.
• SERVE_RESPONSE_MESSAGE: This will show up as the query output in this example.

Please refer to Configure Anyscale Service for a detailed explanation of the configurations in the YAML file. Let's run the example using the anyscale service rollout command. This command is both used to start and update services.

anyscale service rollout -f service-v2.yaml
(anyscale +5.6s) View the service in the UI at https://console.anyscale.com/services/service2_7icqbda33pjtqluy3snzkzaxxv

Query your Service

When you deploy your service, the CLI should print a URL to the page for your service. The Service page will show the Service status. Immediately after you start your service, its state will be STARTING, and once it has started, it will be RUNNING. Once the service is running, it is ready to query.

You can find the endpoint for the Service in the Anyscale UI by clicking the "Query" button in the top right hand corner.

You should find a CURL command that looks something like the below. This command will hit the root / endpoint on your service and you can also query the /-/routes endpoint to see the list of endpoints available under the Service.

Query With Curl

curl -H "Authorization: Bearer SECRET_TOKEN" https://service-hello-jrvwy.cld-kvedzwag2qa8i5bj.s.anyscaleuserdata-staging.com/
"service says hello"

# Query the `/-/routes` endpoint
curl -H "Authorization: Bearer SECRET_TOKEN" https://service-hello-jrvwy.cld-kvedzwag2qa8i5bj.s.anyscaleuserdata-staging.com/-/routes
{"/":"HelloWorld"}

note

If you are using custom networking behind a VPN, you will need to be within your VPN to access your service. If this request is failing, please make sure that you are in your VPN, and can access the Web Terminal on clusters in your cloud.

Upgrade your Service

Let's upgrade the service to show a different message when we query it. First, update the SERVE_RESPONSE_MESSAGE environment variable in service-v2.yaml. This will allow us to see a different output when the service is updated.

caution

You cannot change the Anyscale Cloud your service runs in once it has been deployed.

name: service-hello
cluster_env: default_cluster_env_2.9.0_py310:1
ray_serve_config:
  applications:
    - name: default
      import_path: serve_hello:entrypoint
      runtime_env:
        working_dir: .
        env_vars:
-         SERVE_RESPONSE_MESSAGE: service says hello
+         SERVE_RESPONSE_MESSAGE: service was rolled out

Then, perform a new rollout using the anyscale service rollout CLI command. Be sure that the name parameter does not change. Anyscale Services will use the name attribute to determine which service to upgrade.

anyscale service rollout -f service-v2.yaml


(anyscale +3.3s) View the service in the UI at https://console.anyscale.com/services/service_sc2JFXVupTCLy6CKqcYq5rmN.

Your service will start rolling out to a new version. Once the new version is healthy, the service will slowly shift traffic to the new version. Once the rollout is complete, the service will automatically terminate the previous version.

Once your rollout is complete, you should be able to query your service to see the new message.

Query With Curl

curl -H "Authorization: Bearer SECRET_TOKEN" https://service-hello-jrvwy.cld-kvedzwag2qa8i5bj.s.anyscaleuserdata-staging.com/

"service was rolled out"

Roll back your Service

note

Rolling back is only supported during a rollout. After a rollout is complete, you can roll back by initiating a new rollout to the previous version of the Service.

At any point during the rollout, you can click the rollback button in the UI to stop the rollout and rollback to the previous version of the Service.

You can also initiate a rollback from the CLI using the anyscale service rollback command

# roll back the specified service
# Note that the configuration in service-v2.yaml will be ignored here
# Anyscale will only use the yaml to detect which service to rollback
anyscale service rollback -f service-v2.yaml

Terminate your Service

When you are done, terminate the Service using the CLI.

anyscale service terminate -n service-hello

You can also terminate the Service on the Service page in the UI.