Skip to main content

Workspaces

What are workspaces?​

Workspace is a fully managed development environment focused on the developer productivity. With workspaces, ML practitioners and ML platform developers can quickly build distributed Ray applications and advance from research to development to production easily, all within single environment.

Workspaces provide remote experience, so that you can program the cluster while working with familiar tools like JupyterLab notebooks or Visual Studio Code.

Main features​

Main workspaces features include:

  • Hosted JupyterLab notebooks: Start working with notebooks in a single click.

  • Visual Studio Code remote experience: Work on the hosted VS code or choose local one. Remote experience let you program the cluster while having laptop experience.

  • Persistence: Similarly to local development, your JupyterLab notebooks and code on the workspace are auto-saved. Pick up from where you left off the previous day by just starting your workspace. You can terminate the workspace at any moment and start it later without loosing work.

  • Storage: Network File System (NFS) is mounted by default across all nodes (header + workers) of the cluster. Store your model training datasets or any files you need to access.

  • Configurable environment: You can rapidly modify your environment by installing PyPI packages - they will be available on every node of the cluster. Moreover you can share files, for example datasets for machine learning model training, across all workers.

  • Git integration: All workspaces are git repos, and we provide credentials integration so GitHub access from Anyscale workspace is seamless. Everything tracked by git, including uncommitted changes (that's not included in the .gitignore), will be persisted.

  • Workspace sharing: Let others in your organization preview your work, check your repro steps or do pair-programming with you.

  • Workspace cloning: Do you want to run an experiment in a separate cluster? No problem, just hit the "Clone" button on your workspace, or submit a background job from the workspace.

  • Zero Setup: Managed Ray cluster allows you to start building distributed applications without solving setup issues.

Conceptual overview​

Check the diagram below and associated description to gain understanding of the workspaces.

workspaces-diagram

Workspace​

With workspaces, ML practitioners can develop ray applications using VS Code, JupyterLab Notebook or WebTerminal. Workspace has a Ray cluster attached. Persisted means that we save the code files and git state in the data plane account.

note

(Technical insight) Under the hood, workspaces are implemented as a combination of an Anyscale cluster, and directory in cloud storage where we store workspace snapshots.

Ray cluster​

Ray cluster consist of head node and worker nodes. Developed code runs on the Ray cluster. Every time you run your ray code, the head node sends the ray code and the runtime environment to every worker node. Worker nodes can be auto-scaled to handle computationally intensive jobs, like training multiple ML models in parallel. When creating a new workspace, use a compute config to specify the node types and numbers for the underlying cluster.

Storage​

Storage (Network File System (NFS)) is available out of the box in your workspace. There are four types of storage, that allows for logs, files artifacts and metadata management for user, cluster and for the entire Anyscale cloud. All storage is hosted in the Anyscale instance data plane.

Head node storage​

Instance local storage: /home/ray/<your_project_name> (for example /home/ray/default) - This is where your workspace files are placed (default directory of the WebTerminal). This storage is very fast, but note that we only persist files that are not included in the .gitignore. Files outside this directory will not be persisted.

Cluster local storage​

/mnt/cluster_storage - This is the NFS (so slower that Head node storage) mounted on every node of the Ray cluster and persisted along with the workspace. You might use this to store TensorBoard logs, for ML experiments run on the cluster. Note that when you clone a workspace, this does not clone cluster local storage.

note

(Technical insight) Implementation wise, this is a private directory on EFS.

Shared storage​

/mnt/shared_storage - This is a NFS (so slower that Head node storage) accessible to all the Anyscale users of the same Anyscale Cloud. Shared storage is storage mounted on every node of the Ray cluster, and is globally shared among all workspaces in the same Cloud. You might use this to store model checkpoints and other artifacts that you want to share with your team, use for batch inference, or deploy with Serve in a Service.

note

(Technical insight) Implementation wise, this is a shared directory on EFS.

User storage​

/mnt/user_storage - This is a NFS only accessible to the Anyscale user (from their workspaces, jobs, and services). Every user gets a different directory here, that all their workspaces share.

danger

It is not recommended to put large datasets (terabyte scale), in the storage. Best practice is to use an S3 bucket and access the data from there.

Git / Credentials​

When you first create a workspace, Anyscale generates a unique SSH key tied to your user account. This SSH key is available in ~/.ssh/id_rsa in your workspace cluster. It's up to you to authorize this generated key for access to GitHub or other services, consider it a convenience function.

danger

Be aware that this key is not currently stored securely in a key store (this will be fixed prior to release), so it is possible your coworkers could get access to this key. Make sure the key has appropriately scoped permissions.

Environment variable store​

You can store environment variables associated with your user in the workspace. These environment variables will be available to Ray jobs and to the WebTerminal, JupyterLab Notebooks and VS Code.

Commands and Jobs​

The best way to run Ray programs interactively in a workspace is directly as a command in the WebTerminal, as a JupyterLab Notebook, or in VSCode for example: python train_model.py --cycle_lr --large_batch.

When you run a Ray program in this way, workspace will make sure the files in your workspace directory are snapshotted and replicated to all worker nodes of the workspace cluster via the runtime_env infrastructure. You don't need to worry about configuring runtime_env. Ray programs will work out of the box with just ray.init() within a workspace.

This is exactly the same setup as what happens when running a command as a job, so you can easily go from interactive development to jobs for production.

You can also submit a job from a workspace using the Anyscale Jobs CLI. The job will run in a separate cluster from the workspace cluster. We recommend using this workflow if you want to run an experiment in a separate cluster decoupled from the workspace itself. If you want to run a job interactively, it is better to run it as a normal command. We will show a list of Jobs in the Workspace UI that were launched from the workspace.

Workspaces tutorial​

This tutorial will guide you - step by step - from setup, to files and environment management, to various options for running ray scripts and applications.

Create or start new workspace​

Follow these steps to start experimentation or development with workspaces.

  1. Log in to the Anyscale Console. log-in

  2. Go to the "Workspaces" tab on the left panel. workspace-panel

  3. Create a new Workspace by clicking on the "+ Create" button, or open an existing one. create-workspace

  4. If you open an existing workspace, start it. start-cluster

info

Your workspace will be ready in a few minutes.

Connect workspace with GitHub repo​

  1. When your workspace has started, in the WebTerminal find the ~/.ssh/id_rsa.pub key and authorize its access to your GitHub repository. Check connecting to GitHub with SSH for more help. github

  2. Clone a git repo (git clone <remote_repo> .) into the working directory. Don't forget the trailing dot . in the git clone command. Working directory is: /home/ray/<project_name>, for example /home/ray/default. Read more about projects.

  3. Test connection with GitHub repo. Run a sample Ray script (e.g., python main.py) in the WebTerminal, JupyterLab terminal or VSCode terminal and verify it's working as intended. You will see log messages in the panel below the WebTerminal.

  4. Terminate your workspace, and start it again. Verify that the git repository state is restored in the working directory of your workspace.

info

Your connection with GitHub repository is now ready

Set up Visual Studio Code​

Support for VS Code offers two options:

  1. Hosted VS Code available via browser.
  2. Remote experience with local VS Code installed on the laptop or workstation.

Hosted VS Code​

You can start working on your code with the hosted VS Code. Click on the VS Code icon to open the hosted VS Code. hosted-vs-code

Local VS Code - remote experience​

You can work on your local VS Code and - through remote experience - access and program your Anyscale cluster. You have two options to start a new session on your local VS Code.

Option 1: under Tools menu, select VS Code Desktop: local-vs-code

Option 2: In your local VS Code, open command palette ("CMD + P"), then type "> anyscale", and select "Open in VSCode Desktop": hosted-vs-code-cmd

JupyterLab users​

You can start experimenting with ML models in JupyterLab notebooks by selecting JupyterLab icon in the top menu. notebook

JupyterLab notebooks that you use in the workspace directory will be automatically persisted as part of the workspace files.

Manage files: upload and download​

The recommended way to upload or download small files in your workspace is to use the JupyterLab UI (click upload or download button in the file explorer). You can also commit the file to git and run git pull from the workspace. For large files, the best practice is to use an S3 bucket and access the data from there.

Manage environment​

Library dependencies​

Say your project depends on some pip package like pytorch, which isn't installed by default. Just running pip install only installs it temporarily on the head node of the current cluster. You have two options to enable libraries to be persisted and replicated to all nodes of the cluster:

Install to cluster local storage​

  1. Install your package with pip install --user and it will be put into cluster local storage: /mnt/cluster_storage/pypi.
  2. Since Workspaces sets PYTHONUSERBASE=/mnt/cluster_storage/pypi, the library will automatically be visible to all workers nodes in the cluster.

Build a new cluster environment​

  1. Test out pip install in the WebTerminal until you are satisfied with your environment.
  2. Build a new cluster environment.
  3. Update the cluster env in your workspace in "Edit" and "Save and Apply".
  4. Start the cluster from the workspaces page.

Using environment variable store​

You can store environment variables associated with your user in Workspaces. These environment variables will be available to Ray jobs and to the WebTerminal / JupyterLab. For example, this may be useful for storing credentials for W&B (note: similar to the SSH key management, this doesn't provide strong security yet).

Available commands (run from WebTerminal within the workspace):

`python -m anyscale.snapshot_util put_env <ENV_KEY> <ENV_VALUE>`
`python -m anyscale.snapshot_util get_env <ENV_KEY>`
`python -m anyscale.snapshot_util del_env <ENV_KEY>`

Notes:

  1. These variables are stored per-user. This means they apply to all clusters that you start, including Workspaces and Jobs clusters.
  2. You'll need to restart your Workspace for env vars to show up in the WebTerminal / JupyterLab. You can debug the env vars set by running cat ~/.bashrc. The env var export commands are appended at the end of this file.

Submit Anyscale Jobs​

You can submit Anyscale Jobs from your workspace using the WebTerminal, JupyterLab Terminal or VS Code Terminal. They will run in a separate cluster.

Example usage​

Store hello_world.yaml

name: "hello_world"
entrypoint: "python hello_world.py"

and hello_world.py

import ray
import anyscale

@ray.remote
def say_hi(message):
    return f"Hello, {message}."

ray.init()
print(ray.get(say_hi.remote("World")))

Then, in your workspace directory and run: anyscale job submit hello_world.yaml. It will automatically snapshot the working_dir and the job will inherit settings from the workspace (project, cluster environment, compute config) which can be overwritten in the YAML in the usual manner.

Custom workspace configuration​

Workspaces will source ~/.workspacerc in its .bashrc. If you want a custom .bashrc, put it in the ~/.workspacerc. This file is symlinked into /mnt/user_storage, so it persists for your user. Similarly, ~/.gitconfig is persisted in this way as well.

danger

Warning: be careful when editing your .workspacerc, since you can pollute workspaces with invalid commands.

Tutorial for Laptop Centric Users​

We support laptop-centric users via the "anyscale workspace" group of CLI commands for use from your laptop.

  1. Start by cloning a workspace with the following command. Note the workspace cluster must be RUNNING for these commands to be used:

    anyscale workspace clone -n <name>

  2. From within a locally cloned workspace directory:

    anyscale workspace pull [--pull-git-state]

    anyscale workspace push [--push-git-state]

    anyscale workspace run 'python main.py' [--no-push] [--as-job]

The main thing to know is that anyscale workspace run will run the command remotely in the workspace cluster, instead of on your local laptop. It will automatically push files prior to running, so no need to push explicitly beforehand. So you never need to install any dependencies, etc., locally, because command runs always occur remotely in the cluster container with anyscale workspace run. When run as a job you'll get an exact environment match to development.

Note that the git repo state isn't propagated by default on push/pull since it can add high overhead. Use the --push-git-state and --pull-git-state flags to enable syncing of the git state.

Quick Start for Laptop Centric Flow​

  1. Let's assume you already have your workspace set up, and it's called "my-workspace". Run: anyscale workspace clone -n my-workspace
  2. Go into your workspace directory: cd project-for-workspace-my-workspace
  3. Make some code changes to your workspace.
  4. Run a command: anyscale workspace run 'python my_file.py args'
    1. Notice that the command will be run with the latest file changes reflected in the cluster. This is because run does an implicit push unless you pass --no-push.

Hybrid Workflow​

Instead of using the run command, you can also use anyscale workspace push followed by running the desired command manually in the WebTerminal or JupyterLab terminal in the workspace. This is a hybrid laptop centric and cloud centric workflow, where edits happen locally but command runs go in the browser.

If you made edits in the browser and want to sync the files down to your laptop again, just run anyscale workspace pull again. However, we recommend trying to keep either the browser or laptop the source of truth for the code, even if you run commands from the other. Switching back and forth too much can get confusing.

Workspaces best practices​

When running ML experiments in Workspaces, you may want to pay attention to the following features:

  1. It may be useful to store your experiments results in /mnt/cluster_storage or /mnt/shared_storage by setting the TUNE_RESULT_DIR environment variable. This allows the results to be persisted across time and even shared with other collaborators.
  2. You may consider using the VSCode or JupyterLab Tensorboard extension, in order to view Tensorboard logs through these UIs.
  3. You can submit background jobs using anyscale job submit from within the workspace (see the Submit Anyscale Jobs).

Public Beta (temporary) limitations​

  • You must use Ray 2.0 or above
  • Workspaces is only enabled for customers using AWS for now, GCP support is coming soon.
  • At this moment we do not support git repos in subdirectories, or having multiple git repos per workspace. Either you should have no git repo at all, or clone it into the top-level directory.
  • Firefox users: Hosted VS Code service and JupyterLab Notebooks does not work on Firefox. This is due to the "Total Cookie Protection" enabled by default. Make sure to disable that to use the extension in the browser.
  • Install to cluster local storage: current limitations
    • We do not currently replicate /mnt/cluster_storage when you click "Clone" for a workspace, so packages will need to be manually reinstalled.
    • Installing in NFS will slow down initial import of the modules. For the best performance, we recommend installing packages in the cluster image.

Try Workspaces now​

Sign up to see and try workspaces in action!