Skip to main content
Version: Latest

Secret management

Store sensitive data such as API keys, tokens, credentials, and passwords in a secret manager rather than using exposed environment variables. On a self-hosted Anyscale Cloud, you can grant workspaces, jobs, and services the ability to retrieve and use data stored in Amazon Secrets Manager or GCP Secret Manager by attaching additional IAM policies to the associated IAM roles.

Self-hosted clouds only

You can only connect with Amazon Secrets Manager or GCP Secret Manager if you have deployed your own cloud, not an Anyscale-hosted cloud. See Cloud deployment for set-up instructions.

Configure IAM role​

For self-hosted Anyscale Clouds on AWS, you should configure access to Amazon Secrets Manager for your Anyscale workloads.

Prerequisites​

  1. Retrieve your secret's Amazon Resource Name (ARN), which looks like arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:SECRET_NAME-6_RANDOM_CHARACTERS.
  2. Determine the IAM role that your cluster uses. By default, Anyscale clusters run with a cloud-specific IAM role. If you set up a custom IAM role, use that role.

Grant the IAM role access to the secret​

Each IAM role associated with Anyscale must have the appropriate IAM policy for reading secrets. To grant access, complete the following:

  1. In the AWS console, go to the IAM > Roles console.
  2. Search for your cluster's role, and select it.
  3. Add an inline policy to the role, which grants read access to the secret.
    1. In the Permissions tab, select Add permission and then Create inline policy.
    2. Select the JSON tab.
    3. Use the following JSON as a starting point, making sure to replace SECRET_ARN with your secret's ARN.
{
  "Statement": [
    {
      "Action": ["secretsmanager:GetSecretValue"],
      "Effect": "Allow",
      "Resource": ["SECRET_ARN"],
      "Sid": "SecretsManagerGetSecretValue"
    }
  ]
}
  1. (Optional) The resource-based policy associated with the secret may require modification to grant permission to the IAM role used by Anyscale. See AWS Secrets Manager permissions policy docs for details.
info

For more information on managing IAM policies associated with IAM Roles, see AWS IAM docs. For other examples of AWS Secrets Manager policies, see the AWS Secrets Manager docs.

Using secrets​

To inject secrets into the container running your workload, you can use init scripts to fetch, export, and use secrets.

Initialization script​

Init scripts are shell scripts that run inside the Ray container on all nodes before Ray starts. They give you a flexible way to integrate third-party tools, clone private repos, or perform commands to fetch resources and runtime dependencies.

To add init scripts to your container image, write them into /anyscale/init.

Example​

To retrieve a GitHub personal access token stored in Amazon Secrets Manager and clone a private GitHub repository, follow these steps:

  1. Create a personal access token in GitHub with permissions to access your private repository.

  2. Create a github_clone.sh script like the one below:

    #/bin/bash

    # Prereqs: awscli, jq
    # Env variables required: ANYSCALE_WORKING_DIR, SECRET_NAME, PRIVATE_REPO_NAME, AWS_DEFAULT_REGION

    # Create and navigate to the working directory.
    mkdir -p $ANYSCALE_WORKING_DIR
    cd $ANYSCALE_WORKING_DIR

    # Retrieve and extract the GitHub token from AWS Secrets Manager.
    SECRET_VALUE=$(aws secretsmanager get-secret-value --secret-id ${SECRET_NAME} | jq -r .SecretString)
    GITHUB_TOKEN=$(echo ${SECRET_VALUE} | jq -r .github_token)

    # Clone the private repository.
    git clone https://${GITHUB_TOKEN}@github.com/${PRIVATE_REPO_NAME}

  3. Build a container image hosted in ECR. Below is an example Dockerfile:

    # Use Anyscale base image
    FROM anyscale/ray:2.30.0-slim-py310

    # Create the init directory
    RUN sudo mkdir -p /anyscale/init

    # Install necessary packages
    RUN sudo apt-get update && sudo apt-get install -y axel nfs-common zip unzip && sudo apt-get clean
    RUN sudo apt-get install -y jq git curl && sudo apt-get clean

    # Install AWS CLI v2
    RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
    RUN unzip awscliv2.zip && sudo ./aws/install

    # Install Python dependencies
    RUN pip install --no-cache-dir -U sympy anyscale

    # Add the GitHub clone init script
    ADD ./github_clone.sh /anyscale/init/github_clone.sh
    RUN sudo chmod +x /anyscale/init/github_clone.sh

    # (Optional) Verify base image dependencies
    RUN echo "Testing Ray Import..." && python -c "import ray"
    RUN ray --version
    RUN jupyter --version
    RUN anyscale --version
    RUN sudo supervisord --version
    RUN aws --version

  4. Once you've built the image, you can follow the guide for using container images.