JSON mode

Check your docs version

These docs are for the new Anyscale design. If you started using Anyscale before April 2024, use Version 1.0.0 of the docs. If you're transitioning to Anyscale Preview, see the guide for how to migrate.

JSON Mode enables the LLM to produce a JSON-formatted responses, useful when you want to integrate the LLM with other systems that expect a specific JSON schema.

You can set the JSON schema by setting the response_format = {"type": "json_object"} parameter in the POST request to the LLM. You can optionally provide a schema by setting the response_format = {"type": "json_object", "schema": <schema>} parameter. If you don't provide a schema, the LLM outputs any JSON object that fits.

Supported models

The following models support JSON mode:

• mistralai/Mistral-7B-Instruct-v0.1
• mistralai/Mixtral-8x7B-Instruct-v0.1

note

Support for mistralai/Mixtral-8x22B-Instruct-v0.1 is coming soon.

warning

top_p and top_k sampling parameters are incompatible with JSON mode.

Example

cURL

curl "$ANYSCALE_BASE_URL/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ANYSCALE_API_KEY" \
  -d '{
    "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
    "messages": [{"role": "system", "content": "You are a helpful assistant that outputs in JSON."}, {"role": "user", "content": "Who won the world series in 2020"}],
    "response_format": {"type": "json_object", "schema": {"type": "object", "properties": {"team_name": {"type": "string"}}, "required": ["team_name"]}},
    "temperature": 0.7
  }'

Python

import os
import requests

s = requests.Session()

api_base = os.getenv("ANYSCALE_BASE_URL")
token = os.getenv("ANYSCALE_API_KEY")
url = f"{api_base}/chat/completions"
body = {
    "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant that outputs in JSON."},
        {"role": "user", "content": "Who won the world series in 2020"}
    ],
    "response_format": {
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {
                "team_name": {"type": "string"}
            },
            "required": ["team_name"]
        },
    },
    "temperature": 0.7
}

with s.post(url, headers={"Authorization": f"Bearer {token}"}, json=body) as resp:
    print(resp.json())

OpenAI Python SDK

import openai

client = openai.OpenAI(
    base_url = "https://api.endpoints.anyscale.com/v1",
    api_key = "esecret_YOUR_API_KEY")

# Note: not all arguments are currently supported and will be ignored by the backend.
chat_completion = client.chat.completions.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    messages=[
        {"role": "system", "content": "You are a helpful assistant that outputs in JSON."},
        {"role": "user", "content": "Who won the world series in 2020"}
    ],
    response_format={
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {
                "team_name": {"type": "string"}
            },
            "required": ["team_name"]
        },
    },
    temperature=0.7
)
print(chat_completion.model_dump())

The output will look like:

{'choices': [{'finish_reason': 'stop',
              'index': 0,
              'message': {'content': ' {\n'
                                     '"team_name": " Los Angeles Dodgers "\n'
                                     '}',
                          'function_call': None,
                          'role': 'assistant',
                          'tool_calls': None}}],
'created': 1701933511,
'id': 'mistralai/Mixtral-8x7B-Instruct-v0.1-4-7sX-l4EqKMl0FxZu7f5HQ8DXIKMIQgCkT1sWGLwn8',
'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1',
'object': 'text_completion',
'system_fingerprint': None,
'usage': {'completion_tokens': 16, 'prompt_tokens': 30, 'total_tokens': 46}}

tip

Not explicitly instructing the model to output in JSON is prone to outputting a sting of whitespaces before and after the JSON object, costing more tokens and time. To prevent this issue Anyscale raises an error if the key json isn't found in your message, insensitive to case.

Streaming JSON mode completions

You can also use the streaming API to get JSON mode completions. Here's an example using the Python SDK:

import openai

client = openai.OpenAI(
    base_url = "https://api.endpoints.anyscale.com/v1",
    api_key = "esecret_YOUR_API_KEY")

# Note: not all arguments are currently supported and will be ignored by the backend.
chat_completion = client.chat.completions.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    messages=[
        {"role": "system", "content": "You are a helpful assistant that outputs in JSON."},
        {"role": "user", "content": "Who won the world series in 2020"}
    ],
    response_format={
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {
                "team_name": {"type": "string"}
            },
            "required": ["team_name"]
        },
    },
    temperature=0.7,
    stream=True
)

for chunk in chat_completion:
    print(chunk)

Example output:

ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role='assistant', tool_calls=None), finish_reason=None, index=0, logprobs=ChoiceLogprobs(content=[]))], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=' {', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='"', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='team', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='_', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='name', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='":', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=' "', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='Los', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=' Angeles', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=' Dod', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='gers', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='"', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='}', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809318, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809318, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809318, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=None), finish_reason='stop', index=0, logprobs=None)], created=1706809318, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage={'prompt_tokens': 30, 'completion_tokens': 28, 'total_tokens': 58})