Data Scientists and Engineers have an array of third-party tools and libraries which help in their daily work. Some of the ones we have seen so far have been
Each of these tools have a different pattern for integrating with Anyscale and Ray. These details should assist in developing your integrations.
Fundamentally there are two integration types:
- Code-level Integrations in which you'll integrate with a particular tool by modifying your code
- Service-level Integrations in which you'll integrate with a particular tool by setting some configuration that will automatically log information.
Most of the code that data scientists and ML engineers use come from third-party libraries and are imported and leveraged from within the Python application. Many integrations with third-party tools are no different. With an API token in hand, all it takes for most integrations is to
- set your token in a runtime environment variable
- include the third party integration as a dependency
- use logging statements or other integrations
To enable sevice-level integrations, you may need to install corresponding agents or programs on every node of your clusters. The step may include
- install required libraries or packages in your cluster environments
- use post-build command to install or initialize the integrations with your API tokens.