Object Storage for Anyscale Clouds

An object storage bucket is created or registered for each Anyscale Cloud when it is created. Anyscale stores application data generated in your Anyscale Cloud in this bucket. For AWS clouds, we store data in S3 buckets. For GCP clouds, we store data in GCS buckets.

Each Cloud's bucket is also available for users to store workload-specific data while using Anyscale Workspace, Jobs, and Services. All the Workspaces, Jobs, and Services within an Anyscale Cloud can read and write to its bucket.

Bucket Naming

The bucket is named in this format anyscale-production-data-{cloud_id} but this can be customized by the user if they choose to bring their own bucket.

Within the bucket, Anyscale managed data will be stored in the {organization_id}/ folder. For cloud-specific managed data, we further group together the data into a {organization_id}/{cloud_id} folder. There are still some legacy folders where Anyscale managed data is stored detailed below.

Managed Data

Avoid modifying or deleting the data that Anyscale manages and stores on behalf of the customer. If the data is deleted, the Anyscale platform will have a degraded experience for features such as log viewing, log downloading, and others.

Logs

Logs are stored in {organization_id}/{cloud_id}/logs and /logs folders. The /logs folder is a legacy location and we plan to migrate all logs to the {organization_id}/{cloud_id}/logs folder.

Here, we store all job logs, Web Terminal command logs, and ray logs. For performance reasons, we store logs in various formats for different use cases. For example, when streaming logs, we may produce many small files to allow for fresher data to be downloaded by the user.