gRPC services
Anyscale is rolling out a new design. If you have preview access to the enhanced experience, use the latest version of the docs and see the migration guide for transitioning.
gRPC services are only available in AWS. Support for other cloud providers is in development.
This tutorial walks you through deploying Ray Serve applications using gRPC in an Anyscale Service. All code in this tutorial can be found in the GitHub repo.
For more information on integrating gRPC services into Ray Serve apps, see the Ray Serve gRPC service docs.
Build an image with gRPC Protobufs and Servicer functions
To run gRPC services, Ray Serve needs to access gRPC servicer functions in the cluster environment. To make them available in an Anyscale Service, you must build a container image that includes both your custom Protocol Buffers and Ray Serve application code.
First, define a user_defined_protos.proto
file like the following:
syntax = "proto3";
option java_multiple_files = true;
option java_package = "io.ray.examples.user_defined_protos";
option java_outer_classname = "UserDefinedProtos";
package userdefinedprotos;
message UserDefinedMessage {
string name = 1;
}
message UserDefinedResponse {
string greeting = 1;
}
service UserDefinedService {
rpc __call__(UserDefinedMessage) returns (UserDefinedResponse);
}
Next, define a Ray Serve application in deployment.py
:
from ray import serve
from user_defined_protos_pb2 import UserDefinedMessage, UserDefinedResponse
@serve.deployment
class GrpcDeployment:
def __call__(self, user_message: UserDefinedMessage) -> UserDefinedResponse:
greeting = f"Hello {user_message.name}!"
user_response = UserDefinedResponse(greeting=greeting)
return user_response
grpc_app = GrpcDeployment.options(name="grpc-deployment").bind()
Define the Dockerfile
with the Protobuf and deployment definitions:
# Use Anyscale base image
FROM anyscale/ray:2.30.0-py310
WORKDIR /home/ray
# Copy Protobuf and deployment definitions into the Docker image
COPY user_defined_protos.proto /home/ray/user_defined_protos.proto
COPY deployment.py /home/ray/deployment.py
# Add working directory into Python path so any nodes can import them
ENV PYTHONPATH=/home/ray
# Build Python code from .proto file
RUN python -m grpc_tools.protoc -I=. --python_out=. --grpc_python_out=. ./user_defined_protos.proto
After creating these three files locally, build and push the Docker image with the following command:
# Build the Docker image
docker build . -t my-registry/my-image:tag
# Push the Docker image to your registry
docker push my-registry/my-image:tag
Deploy a service using the custom Docker image
Create a service definition file called service.yaml
:
name: grpc-service
image_uri: REGISTRY/IMAGE:TAG # Replace with your image
cloud: CLOUD_NAME # Replace with your cloud name
grpc_options:
service_names:
- userdefinedprotos. # The name of the gRPC service in your .proto file.
grpc_servicer_functions:
- user_defined_protos_pb2_grpc.add_UserDefinedServiceServicer_to_server
applications:
- name: grpc_app
route_prefix: /grpc_app
import_path: deployment:grpc_app
runtime_env: { }
Ensure grpc_options.service_names
matches the service name in the .proto
file.
Deploy the Anyscale Service:
anyscale service deploy -f service.yaml
You should see the following output with a URL for the services UI.
% anyscale service deploy -f service.yaml
(anyscale +1.8s) Starting new service 'grpc-service'.
(anyscale +4.1s) Service 'grpc-service' deployed.
(anyscale +4.1s) View the service in the UI: 'EXAMPLE_UI_URL'
(anyscale +4.1s) Query the service once it's running using the following curl command:
(anyscale +4.1s) curl -H 'Authorization: Bearer EXAMPLE_API_TOKEN' EXAMPLE_BASE_URL
Send test requests to the service
Once the service is Running, query it with your API token and base URL. You can
find it by clicking Query in the upper right corner of the services page or by
copying it from the previous output of anyscale service deploy
.
Create a test_client.py
script:
import grpc
from user_defined_protos_pb2_grpc import UserDefinedServiceStub
from user_defined_protos_pb2 import UserDefinedMessage
# Replace URL and token with your own.
url = "grpc-service-XXXXX.cld-XXXXXXXXXXXXXXXX.s.anyscaleuserdata.com"
token = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
credentials = grpc.ssl_channel_credentials()
channel = grpc.secure_channel(url, credentials)
stub = UserDefinedServiceStub(channel)
request = UserDefinedMessage(name="Ray")
auth_token_metadata = ("authorization", f"bearer {token}")
metadata = (
("application", "grpc_app"),
auth_token_metadata,
)
response, call = stub.__call__.with_call(request=request, metadata=metadata)
print(call.trailing_metadata()) # Request ID is returned in the trailing metadata
print("Output type:", type(response)) # Response is a type of UserDefinedMessage
print("Full output:", response)
You should see the following output from running the script.
% python test_client.py
(_Metadatum(key='request_id', value='ff6e0810-dd22-488c-ad0e-9f9949bed5be'),)
Output type: <class 'user_defined_protos_pb2.UserDefinedResponse'>
Full output: greeting: "Hello Ray!"