Skip to main content


When running Anyscale workloads in production, we recommend shipping logs & metrics to a third-party provider to enable rich querying, filtering, and alerting capabilities. In these docs, we'll walk through how to set up these integrations by installing a third-party monitoring tool (Vector) into the Ray container.


The following guide requires creating a new cluster environment: either with an Anyscale-provided Docker image, or a Bring your own Docker environment.

Step 0: Requirements

  • Vector is a tool for building observability pipelines. It accepts a configuration file that defines a data pipeline consisting of sources, transforms, and sinks. It supports many common third-party monitoring solutions. In this guide, we will use Vector to scrape logs & metrics from Ray, and ship them to a location of your choice.
  • SupervisorD is a process control system. In this guide, we will use SupervisorD to manage the Vector process.

Step 1: Write a Vector Configuration File

To write and test a configuration file, we recommend using an Anyscale workspace. Start a workspace, and open VS Code (as a text editor).

A Vector configuration file is a directed graph, consisting of one or more sources, transforms, and sinks. Below, we walk through how to build a configuration file to ship Ray logs & metrics to a few third-party providers supported by Vector.

Create a file called vector.yaml, and paste the following configuration in.

Source/Transform Configuration

type: file
ignored_header_bytes: 0
strategy: device_and_inode
- /tmp/ray/*/logs/**/job-driver-*.*
- /tmp/ray/*/logs/**/runtime_env_setup-*.*
- /tmp/ray/*/logs/**/worker-*.out
- /tmp/ray/*/logs/**/worker-*.err
- /tmp/ray/*/logs/**/serve/*.*
# The session_latest directory is a symlink to an actual session directory,
# so we intentionally exclude it here so Vector doesn't ingest duplicates.
- /tmp/ray/session_latest/logs/**/*.*
type: prometheus_scrape
instance_tag: ScrapeTarget
scrape_interval_secs: 15

# These transforms add useful attributes to your log files. To use other environment variables,
# see for all available options.
type: remap
inputs: ["raw_ray_logs"]
source: |-
.cluster_id = "${ANYSCALE_CLUSTER_ID}"
.instance_id = "${ANYSCALE_INSTANCE_ID}"
.node_ip = "${ANYSCALE_NODE_IP}"
type: remap
inputs: ["raw_ray_metrics"]
source: |-
.tags.cluster_id = "${ANYSCALE_CLUSTER_ID}"
.tags.instance_id = "${ANYSCALE_INSTANCE_ID}"
.tags.node_ip = "${ANYSCALE_NODE_IP}"
.tags = compact(.tags, recursive: true)

Sink Configuration

Then, choose one of the sinks below, and add it to vector.yaml.

AWS CloudWatch requires additional access for the Cluster IAM role. This can be modified in the AWS IAM Console. Make sure to replace YOUR_ACCOUNT_ID with your AWS Account ID.

IAM Cloudwatch Policy
"Statement": [
"Action": "cloudwatch:PutMetricData",
"Effect": "Allow",
"Resource": "*",
"Sid": "CloudwatchMetricsWrite"
"Action": [
"Effect": "Allow",
"Resource": "*",
"Sid": "CloudwatchLogsRead"
"Action": "logs:PutLogEvents",
"Effect": "Allow",
"Resource": "arn:aws:logs:*:YOUR_ACCOUNT_ID:log-group:/anyscale*:*",
"Sid": "CloudwatchLogsEventsWrite"
"Action": [
"Effect": "Allow",
"Resource": "arn:aws:logs:*:YOUR_ACCOUNT_ID:log-group:/anyscale*",
"Sid": "CloudwatchLogsWrite"
"Version": "2012-10-17"

Once the IAM Role has been updated, update vector.yaml to include a sink section as follows:

region: us-west-2
codec: json
group_name: "/anyscale/"
inputs: ["ray_logs"]
# One of ANYSCALE_PRODJOB_ID / ANYSCALE_SERVICE_ID will be set for jobs / services.
type: aws_cloudwatch_logs
region: us-west-2
default_namespace: anyscale
inputs: ["ray_metrics"]
type: aws_cloudwatch_metrics

Step 2: Test the Configuration File

Save the Vector configuration above in a file in your local directory (for example, vector.yaml). Then, run the following commands:

# Install Vector.
sudo apt-get install curl -y
curl --proto '=https' --tlsv1.2 -sSfL | bash
source /home/ray/.profile

# Create a state directory for Vector & make it accessible.
sudo mkdir -p /var/lib/vector/
sudo chmod 777 /var/lib/vector/

# Run Vector
vector --config vector.yaml

# In a new tab, generate fake log content.
mkdir -p /tmp/ray/session_fake/logs/
for i in {1..5000}; do echo "Log Line $i" >> /tmp/ray/session_fake/logs/job-driver-fake.log && echo "Wrote line $i" && sleep 1; done

# Look for warnings / errors in Vector - if you don't see any, check upstreams to see if logs & metrics are being received.

Step 3: Move to Production

To move to production, we will first need to build a SupervisorD file, so that we can configure the Vector process to run automatically on cluster startup & in a process manager (so it will be restarted on failure). Let's create a file like the one below at supervisord.conf in the same workspace as earlier.

command=bash --login -c -i "sudo -E /home/ray/.vector/bin/vector --config=/etc/vector/vector.yaml"

Then, follow the instructions below to package both of these configuration files into a Ray container image.

  1. On your laptop (or wherever you build your Dockerfile), change directory into the directory with your Dockerfile in it.
  2. Copy vector.yaml from your Workspace into this directory.
  3. Copy supervisord.conf from your Workspace into this directory.
  4. Add the following lines to your Dockerfile.
# Install Vector.
RUN curl --proto '=https' --tlsv1.2 -sSfL | bash -s -- -y

# Write the Vector config.
RUN sudo mkdir -p /etc/vector/
RUN chmod 777 /etc/vector/
COPY vector.yaml /etc/vector/vector.yaml

# Write the SupervisorD config.
RUN sudo mkdir -p /etc/supervisor/customer.conf.d/
RUN chmod 777 /etc/supervisor/customer.conf.d/
COPY supervisord.conf /etc/supervisor/customer.conf.d/vector.conf
  1. Build & push your Docker image, create a cluster environment with this Docker image, and start an Anyscale Job or Service.