Skip to main content

Troubleshooting

Network connectivity​

Anyscale monitors connectivity with running clusters. All traffic flows directly from your browser to the cluster, bypassing the Anyscale control plane. When health checks fail, a banner appears, indicating potential problems.

Example Banner

Impacted features

  • Ray Dashboard
  • Workspace Jupyter notebook
  • Workspace terminal
  • Workspace duplication

Possible causes

Network connectivity issues can arise within any components between the browser on your machine and the web servers on the cluster's head node. Common sources of issues are:

  • Browser restrictions on cross-origin requests.
  • Residential ISP blocking.
  • Restrictive networking configuration in the Anyscale cloud.
  • High resource usage on the cluster's head node.
note

For clouds deployed using customer defined networking, because access to Anyscale isn't routed over the internet, ensure that your VPN or local access solution is functioning correctly.

Potential workarounds

  1. Verify head node health - Using the Grafana dashboard, check that the head node isn't over-utilizing resources like CPU, memory, or disk.
  2. ISP restrictions - If on residential WiFi, check your router's settings. ISPs such as Comcast or AT&T may block Anyscale connections. Access your router's advanced protection settings and allow the Anyscale cluster's domain. Alternatively, you could try using a VPN or connecting through a different ISP, such as a mobile hot spot or different location.
  3. Browser compatibility - Test with a different browser to rule out cross-origin blocking caused by privacy settings. This problem may occur because the clusters and console are on a different domains.
  4. Cloud platform settings - Ensure that outbound traffic, or egress, in your cloud environment allows access from your location. Restrictive security groups on AWS or firewall rules on GCP can limit which IP addresses can access Anyscale.

Grafana dashboards not loading​

In Safari and Firefox, Grafana dashboards may fail to load with a 401 Unauthorized error. This issue also affects the Ray Dashboard. A permanent fix is in progress, but temporary workarounds are available.

Workaround

  • Safari: Disable Website tracking: Prevent cross-site tracking.

Safari Fix

  • Firefox: Disable Enhanced Tracking Protection.

Firefox Fix

Hibernation of Anyscale cloud

An Anyscale cloud hibernates after one week of without activity, defined as:

  • Running workloads with workspaces, jobs, or services.
  • Creating compute configurations.
  • Accessing the Grafana dashboard.

With a hibernated cloud, the first activity triggers a process to bring the resources back up. After ten minutes, retry with the activated cloud.