gRPC services

Check your docs version

Anyscale is rolling out a new design. If you have preview access to the enhanced experience, use the latest version of the docs and see the migration guide for transitioning.

This tutorial walks you through deploying Ray Serve applications using gRPC in an Anyscale Service. All code in this tutorial can be found in the GitHub repo.

For more information on integrating gRPC services into Ray Serve apps, see the Ray Serve gRPC service docs.

Build an image with gRPC Protobufs and Servicer functions

To run gRPC services, Ray Serve needs to access gRPC servicer functions in the cluster environment. To make them available in an Anyscale Service, you must build a container image that includes both your custom Protocol Buffers and Ray Serve application code.

First, define a user_defined_protos.proto file like the following:

syntax = "proto3";

option java_multiple_files = true;
option java_package = "io.ray.examples.user_defined_protos";
option java_outer_classname = "UserDefinedProtos";

package userdefinedprotos;

message UserDefinedMessage {
  string name = 1;
}

message UserDefinedResponse {
  string greeting = 1;
}

service UserDefinedService {
  rpc __call__(UserDefinedMessage) returns (UserDefinedResponse);
}

Next, define a Ray Serve application in deployment.py:

from ray import serve
from user_defined_protos_pb2 import UserDefinedMessage, UserDefinedResponse


@serve.deployment
class GrpcDeployment:
    def __call__(self, user_message: UserDefinedMessage) -> UserDefinedResponse:
        greeting = f"Hello {user_message.name}!"
        user_response = UserDefinedResponse(greeting=greeting)
        return user_response


grpc_app = GrpcDeployment.options(name="grpc-deployment").bind()

Define the Dockerfile with the Protobuf and deployment definitions:

# Use Anyscale base image
FROM anyscale/ray:2.30.0-py310

WORKDIR /home/ray

# Copy Protobuf and deployment definitions into the Docker image
COPY user_defined_protos.proto /home/ray/user_defined_protos.proto
COPY deployment.py /home/ray/deployment.py

# Add working directory into Python path so any nodes can import them
ENV PYTHONPATH=/home/ray

# Build Python code from .proto file
RUN python -m grpc_tools.protoc -I=. --python_out=. --grpc_python_out=. ./user_defined_protos.proto

After creating these three files locally, build and push the Docker image with the following command:

# Build the Docker image
docker build . -t my-registry/my-image:tag

# Push the Docker image to your registry
docker push my-registry/my-image:tag

Deploy a service using the custom Docker image

Create a service definition file called service.yaml:

name: grpc-service
image_uri: REGISTRY/IMAGE:TAG  # Replace with your image
cloud: CLOUD_NAME  # Replace with your cloud name

grpc_options:
  service_names:
    - userdefinedprotos.  # The name of the gRPC service in your .proto file.
  grpc_servicer_functions:
    - user_defined_protos_pb2_grpc.add_UserDefinedServiceServicer_to_server

applications:
  - name: grpc_app
    route_prefix: /grpc_app
    import_path: deployment:grpc_app
    runtime_env: { }

Ensure grpc_options.service_names matches the service name in the .proto file.

Deploy the Anyscale Service:

anyscale service deploy -f service.yaml

You should see the following output with a URL for the services UI.

% anyscale service deploy -f service.yaml
(anyscale +1.8s) Starting new service 'grpc-service'.
(anyscale +4.1s) Service 'grpc-service' deployed.
(anyscale +4.1s) View the service in the UI: 'EXAMPLE_UI_URL'
(anyscale +4.1s) Query the service once it's running using the following curl command:
(anyscale +4.1s) curl -H 'Authorization: Bearer EXAMPLE_API_TOKEN' EXAMPLE_BASE_URL

Send test requests to the service

Once the service is Running, query it with your API token and base URL. You can find it by clicking Query in the upper right corner of the services page or by copying it from the previous output of anyscale service deploy.

Create a test_client.py script:

import grpc
from user_defined_protos_pb2_grpc import UserDefinedServiceStub
from user_defined_protos_pb2 import UserDefinedMessage


# Replace URL and token with your own.
url = "grpc-service-XXXXX.cld-XXXXXXXXXXXXXXXX.s.anyscaleuserdata.com"
token = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

credentials = grpc.ssl_channel_credentials()
channel = grpc.secure_channel(url, credentials)
stub = UserDefinedServiceStub(channel)
request = UserDefinedMessage(name="Ray")
auth_token_metadata = ("authorization", f"bearer {token}")
metadata = (
    ("application", "grpc_app"),
    auth_token_metadata,
)
response, call = stub.__call__.with_call(request=request, metadata=metadata)
print(call.trailing_metadata())  # Request ID is returned in the trailing metadata
print("Output type:", type(response))  # Response is a type of UserDefinedMessage
print("Full output:", response)

You should see the following output from running the script.

% python test_client.py
(_Metadatum(key='request_id', value='ff6e0810-dd22-488c-ad0e-9f9949bed5be'),)
Output type: <class 'user_defined_protos_pb2.UserDefinedResponse'>
Full output: greeting: "Hello Ray!"