Known Issues
pip packages fail to install within a cluster environment
Problem
If you see this message in Cluster Environment build logs:
[INFO] 6/23/2021, 4:20:16 PM: Running setup.py install for <SOME PIP PACKAGE>: finished with status 'error'
[ERROR] 6/23/2021, 4:20:16 PM: ERROR: Command errored out with exit status 1:
...
ImportError: Something is wrong with the numpy installation. While importing we detected an older version of numpy in ['/home/ray/anaconda3/lib/python3.8/site-packages/numpy']. One method of fixing this is to repeatedly uninstall numpy until none is found, then reinstall this version.
Workaround
- Remove from pip section
- Add the following to post build commands:
/home/ray/anaconda3/bin/python -m pip uninstall -y numpy
rm -rf /home/ray/anaconda3/lib/python3.<7 OR 8>/site-packages/numpy
/home/ray/anaconda3/bin/pip install numpy
/home/ray/anaconda3/bin/pip install --upgrade --no-cache-dir <SOME PIP PACKAGE>
Installing Horovod requires extra steps
Problem
Horovod cannot be installed simply by adding it to the pip packages.
Workaround
Set the appropriate environment variables in the cluster environment:
HOROVOD_WITH_TENSORFLOW=1
HOROVOD_WITH_GLOO=1
Add the following post-build commands:
pip install horovod[tensorflow]
pip install horovod[ray]
Anyscale VSCode Desktop Plugin on Windows
Problem
The Anyscale Plugin for VSCode Desktop is not supported on Windows. Anyscale is working on a resolution for this. There is no workaround at this time.
Workaround
This will be resolved with a future Anyscale Plugin for VSCode Desktop.
Cluster Runtime Environment Log is empty
Problem
This is a known issue with the UI and will be resolved with a future Anyscale release.
Workaround
Logs for a cluster can be downloaded via
anyscale logs cluster --id <cluster_id> --download
Anyscale Services fail to start with Ray 2.3.0
Problem
Anyscale Services with Ray 2.3.0 will fail to start if the entrypoint uses serve. ex:
entrypoint: serve run --non-blocking serve_hello:entrypoint
Workaround
Verify that you're python code has:
if __name__ == __main__:
serve.run()
And change the entrypoint to just call your python file:
entrypoint: python <python_entry_point.py>
This will be fixed in Ray 2.3.1
Anyscale Workspaces fails to persist Git repositories
Problem
After using an Anyscale Workspace and cloning a git repo, the files are not persisted after the Workspace is terminated and restarted.
Workaround
Verify that you are cloning the git repo into the working directory - the trailing '.' is required:
git clone <repo> .
Workaround
Verify that you are cloning the git repo into the working directory - the trailing '.' is required:
git clone <repo> .
Grafana Dashboards fail to load
Problem
Cluster Grafana dashboards fail to load with a 401 Unauthorized error
in Safari and Firefox browsers. This alert can also appear on the Ray dashboard. While this will be addressed in a future release of Anyscale, there are currently workarounds for both browsers.
Workaround
In Safari, you need to disable Prevent cross-site tracking
.
In Firefox, you need to disable Enhanced Tracking Protection