Accessing GCP Secrets Manager

This section provides instructions on configuring access to GCP Secret Manager for Anyscale clusters. Accessing stored secrets in GCP Secret Manager is often necessary for various workloads and clusters. The configuration process involves two key steps:

1. Policy Assignment for service account: Ensure that the service account associated with Anyscale Clusters has the appropriate access control for reading secrets.
2. (Optional) Update secret level IAM bindings:: Optionally, you may need to grant access to the service account with a secret level IAM binding.

The following documentation will guide users through these steps to facilitate secure and effective integration with GCP Secret Manager.

Configure the service account

Pre-requisites

• Enable the Secret Manager API in the Google Project you are using.
• Create a secret using Secret Manager.

note

Determining the Cluster's service account

By default, Anyscale clusters run with a cloud-specific service account (instructions are here).

If you followed instructions on how to run with a custom service account, use that service account for the rest of the instructions.

Ensure the Cluster's service account has access to your Secret

1. In the GCP console, go to the IAM page.
2. Search for and select the Cluster service account and then click Edit.

3. In the Assign Roles section that appears, click Add another role and search for Secret Manager Secret Accessor.

4. Click Save.

info

These steps will grant access to all secrets stored in that project. For more granular access, please see the GCP Documentation.

Depending on your security requirements, you may need to add the Cluster service account as a Principal to the Secret in place of the steps above. Please refer to the Principle of lease privilege in the GCP documentation

Using Secrets

There are several ways that Secrets can be utilized with clusters and applications running with Anyscale. Here is an example:

Cluster Initialization Scripts

Cluster Init Scripts are part of an Anyscale Cluster Environment. These shell scripts run on all nodes in a cluster after the node is started, but before Ray starts. This gives you a very flexible way to integrate third-party tools, clone private git repos, or perform other application startup requirements. Init scripts need to be created inside the folder /anyscale/init.

Here is an example of using a init script to retrieve a GitHub Personal Access Token stored in Secrets Manager, and perform a git clone of a private git repo.

1. Create a Personal Access Token in GitHub, granting it permissions to read from your private GitHub repository.
2. Create a Cluster Environment using either a Custom Docker Image hosted in ECR, or a standard Cluster Environment using a YAML file for the definition.
3. Follow one of the following examples to create a Cluster Environment.

Cluster Environment

base_image: anyscale/ray:2.9.0-py310
env_vars:
  PRIVATE_REPO_NAME: <your_private_repo>
  SECRET_NAME: <example_secret-github_token>
debian_packages:
  - jq
post_build_cmds:
  - sudo mkdir -p /anyscale/init
  - sudo touch /anyscale/init/github_clone.sh
  - echo 'mkdir -p $ANYSCALE_WORKING_DIR && cd $ANYSCALE_WORKING_DIR' | sudo tee -a /anyscale/init/github_clone.sh
  - echo 'SECRET_VALUE=$(gcloud secrets versions access latest --secret=$SECRET_NAME)' | sudo tee -a /anyscale/init/github_clone.sh
  - echo 'GITHUB_TOKEN=$(echo ${SECRET_VALUE} | jq -r .github_token)' | sudo tee -a /anyscale/init/github_clone.sh
  - echo 'git clone https://${GITHUB_TOKEN}@github.com/${PRIVATE_REPO_NAME}' | sudo tee -a /anyscale/init/github_clone.sh

Custom Docker Image

You will need to create both the dockerfile and the github_clone.sh files before building the Docker image. Here is an example dockerfile:

# Use Anyscale base image
FROM anyscale/ray:2.9.0-py310

RUN sudo apt-get update && sudo apt-get install -y axel nfs-common zip unzip && sudo apt-get clean

# Required for this example
RUN sudo apt-get install -y jq git curl && sudo apt-get clean

RUN pip install --no-cache-dir -U anyscale

ADD ./github_clone.sh /anyscale/init/github_clone.sh

# (Optional) Verify that dependencies from the base image still work. This
# is useful for catching dependency conflicts at build time.
RUN echo "Testing Ray Import..." && python -c "import ray"
RUN ray --version
RUN jupyter --version
RUN anyscale --version
RUN sudo supervisord --version
RUN aws --version

Here is an example github_clone.sh:

#/bin/bash

# PreReqs: awscli, jq
# Env Variables required: ANYSCALE_WORKING_DIR, SECRET_NAME, PRIVATE_REPO_NAME, AWS_DEFAULT_REGION

# Create and then change directories into the working directory - either default or project_name
mkdir -p $ANYSCALE_WORKING_DIR
cd $ANYSCALE_WORKING_DIR

SECRET_VALUE=$(gcloud secrets versions access latest --secret=$SECRET_NAME)
GITHUB_TOKEN=$(echo ${SECRET_VALUE} | jq -r .github_token)

# Clone the repo
git clone https://${GITHUB_TOKEN}@github.com/${PRIVATE_REPO_NAME}

Once your Docker file is built, you can follow our guide for Bringing your own Docker, making sure to set the environment variables: SECRET_NAME, PRIVATE_REPO_NAME, and AWS_DEFAULT_REGION.