Accessing logs
Accessing logs
Anyscale offers multiple ways to access logs for your Ray clusters.
Log viewer
The log viewer allows you to view and download log files generated from your Ray apps. To access the log viewer UI, select the Logs tab in the workspace, job, or service detail page.
The log viewer only displays logs from individual log files. When you download logs, you receive all the log files on the cluster.
Log ingestion and query
Log ingestion and query is an optional feature that allows you to search, filter, and query logs. Any cluster created after enabling this feature sends logs to the Anyscale control plane, where they're retained for 30 days. Enabling this feature might incur extra data transfer costs from your cloud provider.
Anyscale-hosted clouds have log ingestion and query enabled by default. For self-hosted clouds, you must opt-in with the following command:
# Change `<your-cloud-id>` to your actual cloud ID
anyscale cloud config update --cloud-id <your-cloud-id> --enable-log-ingestion
See anyscale cloud update for more information.
Search
Use the search bar to find specific keywords, request IDs, and regular expression patterns.
Filter
The filter field displays buttons for available filter categories. If a category has filters applied, its button has a black background. A button with no background indicates there is no filter set for the category. Click a button to open its individual filters and select or deselect the appropriate checkboxes to filter displayed logs.
By default, the logs shown are from the last hour of real time that the workload was running. You can select a preset time range or use the available date pickers to set an exact one.
Live mode
Turn Live mode on to automatically refresh the display with new logs as they become available. Live mode is on when the toggle is to the right and has a black background.
To turn off log ingestion and query, run anyscale cloud config update --cloud-id <your-cloud-id> --disable-log-ingestion. Anyscale retains previously sent logs for 30 days. Ray clusters you create after running this command don't send logs to the Anyscale control plane.
Ray Dashboard
Ray Dashboard is a consolidated tool for monitoring and debugging your applications available when the Ray cluster is running. To access the logs, select the Ray Dashboard tab and then the Logs tab.
For information on the log files, see Configuring logging on Ray docs.
Logs CLI
Use the CLI to download logs, even after a cluster has terminated.
Download workload logs
The Anyscale CLI provides log commands for jobs, workplace, and service to download relevant logs.
- jobs
- workplace
- service
# Change <target-job-id> to the desired job ID
anyscale logs job --id <target-job-id> -d --download-dir /tmp
# Change <target-workplace-id> to the desired workplace ID
anyscale logs job --id <target-workplace-id> -d --download-dir /tmp
# Change <target-service-id> to the desired service ID
anyscale logs job --id <target-service-id> -d --download-dir /tmp
For large clusters, downloading all logs can take a long time. Use --head-only to download only head node logs, which typically completes in about 10 seconds:
# Change <target-job-id> to the desired job ID
anyscale logs job --id <target-job-id> --head-only -d
For Ray Data debugging, head-only logs are usually sufficient because ray-data.log lives on the head node.
Filter and speed up log downloads
Use the flags in the following table with any anyscale logs subcommand to target specific nodes or control download behavior.
| Flag | Description |
|---|---|
--node-ip | Target a specific node. |
--instance-id | Target a specific node. |
--worker-only | Download only worker node logs. |
--parallelism | Control concurrent file downloads for faster transfers. |
--tail N | Download only the last N lines. |
See anyscale logs cluster for the full list of available options.
Download cluster logs
Use the logs cluster command to download cluster logs.
# Change <target-cluster-id> to your cluster ID
anyscale logs cluster --id <target-cluster-id> -d --download-dir /tmp
Find logs
Anyscale stores logs in {organization_id}/{cloud_id}/logs and /logs folders. The /logs folder is a legacy location. Anyscale sends most logs to the {organization_id}/{cloud_id}/logs folder.
Anyscale stores all job logs, web terminal command logs, and Ray logs. For performance reasons, Anyscale stores logs in various formats for different use cases. For example, when streaming logs, Anyscale might produce many small files to allow for fresher data for the user to download.