Third-party Docker registries

Bring your own Docker environments can be configured to pull from private Docker registries that are hosted outside of your cloud provider, such as JFrog Artifactory and Docker Hub. In order to authenticate, Anyscale managed nodes need access to Docker login credentials to authenticate to the registry hosting the image.

The following diagram illustrates the relationship between the Anyscale control plane, the data plane, and the third-party registry:

The credentials for the third-party registry never enter the Anyscale control plane. The Anyscale control plane stores a reference to a secret in your cloud provider's secret manager and nodes in your account fetch the contents of the secret to authenticate at runtime.

---

Step 1. Obtain access credentials for your registry

This step will vary depending on your Docker registry provider. The credentials only need to allow read access to the registry containing your image, write and delete permissions are not needed. If you have Docker setup locally, you can test the credentials with docker login:

docker login example.registry.com # This should prompt you for a username and password
docker pull example.registry.com/myimage:my-tag # This should succeed

Step 2. Store credentials in your cloud provider's secret manager

Once you have obtained credentials to authenticate to your Docker registry provider, they should be stored in your cloud provider's secret manager in the following JSON format:

{
    "username": "myusername",
    "password": "mypassword",
    "server": "example.registry.com"
}

The server field should match the server in the image you want to pull. For example, for the image example.jfrog.io/anyscale:2.4.0 the server field should be set to example.jfrog.io. For images hosted on Docker Hub, use the server docker.io. At runtime, Anyscale managed nodes will check if the server field in the secret matches the server hosting the image. If there is a mismatch, the credentials will not be used to authenticate to avoid leaking credentials to the wrong registry provider.

Step 3. Grant read access to managed nodes

AWS

See this guide for granting a role access to a secret. Make sure to grant access to the role used by Anyscale managed nodes in your account. In most cases, this should be <cloud_id>-cluster_node_role role. If your nodes are being launched with the role ray-autoscaler-v1, or if you are using a custom AWS IAM role then you can apply the same steps to that role instead to grant access.

GCP

See this guide for granting a service account access to a secret. Make sure to grant access to the service account used by Anyscale managed nodes in your account. In most cases, this should be cloud-id@project-id.iam.gserviceaccount.com. If you are using a custom AWS IAM role then you can apply the same steps to that role instead to grant access.

Step 4. Register the secret with Anyscale

When creating a cluster environment for your image, use the registry_login_secret field to configure your cluster to authenticate using your secret.

• On AWS, you can either specify the name of your secret or the full ARN. If only the name is specified, then instances will look for the secret in the same account and region as the instance. To access secrets across accounts or regions, use the full ARN.
• On GCP, you can either specify the name of your secret or the full identifier (/projects/<project-id>/secrets/<secret-name>/versions/<version>). If you just specify the secret name, instances will look for the secret in the same project, and use the latest version. To access secrets across projects or specific versions, use the full identifier.

CLI

Create a YAML configuration file like the following:

docker_image: my-registry/my-image:tag
ray_version: 2.7.0 # Replace this with the version of Ray in your image
env_vars: # Optionally, specify environment variables
  MY_VAR: value
registry_login_secret: mysecretid

Then, run the following:

anyscale cluster-env build -n <cluster-env-name> my_cluster_env.yaml

SDK

from anyscale import AnyscaleSDK
from anyscale.sdk.anyscale_client import CreateBYODClusterEnvironment

sdk = AnyscaleSDK()

create_cluster_environment = CreateBYODClusterEnvironment(
    name="<cluster-env-name>",
    config_json={
        "docker_image": "my-registry/my-image:tag",
        "ray_version": "2.7.0", # Replace this with the version of Ray in your image
        "env_vars": {  # Optionally, specify environment variables
            "MY_VAR": "value"
        },
        "registry_login_secret": "mysecretid"
    }
)

cluster_environment_build = sdk.build_cluster_environment(
    create_cluster_environment=create_cluster_environment
)

UI

In the Anyscale console, navigate to "Configurations -> Cluster Environments -> Create new environment," and under "Base docker image" select "Use my own docker image." Configure the secret with the "Registry login secret" field under the Additional Configurations section.