Skip to main content

Distributed Debugging

Interactive distributed debugging with Visual Studio Code is now supported with Ray >= 2.9.0 in your workspaces. This feature allows you to break into any remote task running anywhere in your Ray cluster and perform post-mortem debugging when tasks hit an exception.

Configuration​

To enable debugging, set the RAY_DEBUG environment variable in your cluster environment or via runtime_env inside ray.init

# make sure ray is 2.9.0 or above
# assert ray.__version__ >= "2.9.0"
ray.init(runtime_env={
    "env_vars": {"RAY_DEBUG": "1"},
})

Breaking into Tasks with breakpoint()​

You can insert a breakpoint directly into your Ray tasks using the standard Python breakpoint() method. When your code hits the breakpoint, it will pause execution, allowing you to connect with VSCode and inspect or modify the state of the program.

How to Use:​

  • Ensure your environment is correctly set up by enabling the RAY_DEBUG variable and having a supported Ray version.
  • Insert breakpoint() at the desired point in your Ray task.
  • Run your Ray application.
  • When the breakpoint condition is hit, the task will enter a paused state. The terminal will clearly display this information.
(test pid=3459) Ray debugger is listening on 10.0.0.241:48101
(test pid=3459) Waiting for debugger to attach...
  • Open the Ray view in the side bar of VSCode, this is powered by the Anyscale extension installed by default, and you will see a list of paused tasks. Click on the paused task you wish to attach to.
connect-to-debugger
  • Use the VSCode debugger just like you do developing locally.
debug

Postmortem Debugging​

Postmortem debugging is crucial for diagnosing issues that cause uncaught exceptions in your tasks. When a task raises an exception, Ray will freeze its state and wait for a debugger to attach, allowing you to inspect the state of the program at the time of the error.

How to Use:​

  • Ensure your environment is correctly set up by enabling the RAY_DEBUG variable and having a supported Ray version.
  • Run your Ray application.
  • When a task throws an unhandled exception, it will enter a paused state. The terminal will clearly display this information.
(test pid=3459) Ray debugger is listening on 10.0.0.241:48101
(test pid=3459) Waiting for debugger to attach...
  • Open the Ray view in the side bar of VSCode, this is powered by the Anyscale extension installed by default, and you will see a list of paused tasks. Click on the paused task you wish to attach to.
  • Use the VSCode debugger just like you do developing locally.
debug-exception
caution

Ensure that the "unhandled exception" option is enabled in VSCode's debugger settings. This setting allows VSCode to break execution at the point where the exception occurred, enabling you to perform post-mortem analysis.