Skip to main content

Known Issues

pip packages fail to install within a cluster environment​

Problem​

If you see this message in Cluster Environment build logs:

[INFO] 6/23/2021, 4:20:16 PM:     Running setup.py install for <SOME PIP PACKAGE>: finished with status 'error'
[ERROR] 6/23/2021, 4:20:16 PM:     ERROR: Command errored out with exit status 1:
...
ImportError: Something is wrong with the numpy installation. While importing we detected an older version of numpy in ['/home/ray/anaconda3/lib/python3.8/site-packages/numpy']. One method of fixing this is to repeatedly uninstall numpy until none is found, then reinstall this version.

Workaround​

  • Remove from pip section
  • Add the following to post build commands:
/home/ray/anaconda3/bin/python -m pip uninstall -y numpy
rm -rf /home/ray/anaconda3/lib/python3.<7 OR 8>/site-packages/numpy
/home/ray/anaconda3/bin/pip install numpy
/home/ray/anaconda3/bin/pip install --upgrade --no-cache-dir <SOME PIP PACKAGE>

Installing Horovod requires extra steps​

Problem​

Horovod cannot be installed simply by adding it to the pip packages.

Workaround​

Set the appropriate environment variables in the cluster environment:

HOROVOD_WITH_TENSORFLOW=1
HOROVOD_WITH_GLOO=1

Add the following post-build commands:

pip install horovod[tensorflow]
pip install horovod[ray]

Anyscale VSCode Desktop Plugin on Windows​

Problem​

The Anyscale Plugin for VSCode Desktop is not supported on Windows. Anyscale is working on a resolution for this. There is no workaround at this time.

Workaround​

This will be resolved with a future Anyscale Plugin for VSCode Desktop.

Cluster Runtime Environment Log is empty​

Problem​

This is a known issue with the UI and will be resolved with a future Anyscale release.

Workaround​

Logs for a cluster can be downloaded via

anyscale logs cluster --id <cluster_id> --download

Anyscale Services fail to start with Ray 2.3.0​

Problem​

Anyscale Services with Ray 2.3.0 will fail to start if the entrypoint uses serve. ex:

entrypoint: serve run --non-blocking serve_hello:entrypoint

Workaround​

Verify that you're python code has:

if __name__ == __main__:
    serve.run()

And change the entrypoint to just call your python file:

entrypoint: python <python_entry_point.py>

This will be fixed in Ray 2.3.1

Anyscale Workspaces fails to persist Git repositories​

Problem​

After using an Anyscale Workspace and cloning a git repo, the files are not persisted after the Workspace is terminated and restarted.

Workaround​

Verify that you are cloning the git repo into the working directory - the trailing '.' is required:

git clone <repo> .

Workaround​

Verify that you are cloning the git repo into the working directory - the trailing '.' is required:

git clone <repo> .

Grafana Dashboards fail to load​

Problem​

Cluster Grafana dashboards fail to load with a 401 Unauthorized error in Safari and Firefox browsers. This alert can also appear on the Ray dashboard. While this will be addressed in a future release of Anyscale, there are currently workarounds for both browsers.

Workaround​

In Safari, you need to disable Prevent cross-site tracking.

Safari Fix

In Firefox, you need to disable Enhanced Tracking Protection

Firefox Fix