Causal language modeling

The causal language modeling task computes the language modeling loss across all text tokens regardless of the role, "user" or "assistant."

Example config

model_id: meta-llama/Meta-Llama-3-8B-Instruct  # Any HF model ID.
task: "causal_lm"
generation_config:
    prompt_format: # Does nothing but concatenation.
        system: "{instruction}"
        user: "{instruction}"
        assistant: "{instruction}"
        system_in_user: False
...

Example dataset

You can specify all content for causal language modeling in just the user role.

{
    "messages": [
        {"role": "user", "content": "Once upon a time ..."},
    ],
},