Skip to main content
Deprecated

LLMForge is being deprecated: The Ray Team is consolidating around open source fine-tuning solutions. Llama Factory and Axolotl provide enhanced functionality (quantization, advanced algorithms) and native Ray support for scaling. See the migration guide for transitioning your workflows.

Causal language modeling

The causal language modeling task computes the language modeling loss across all text tokens regardless of the role, "user" or "assistant."

Example config​

model_id: meta-llama/Meta-Llama-3-8B-Instruct  # Any HF model ID.
task: "causal_lm"
generation_config:
    prompt_format: # Does nothing but concatenation.
        system: "{instruction}"
        user: "{instruction}"
        assistant: "{instruction}"
        system_in_user: False
...

Example dataset​

You can specify all content for causal language modeling in just the user role.

{
    "messages": [
        {"role": "user", "content": "Once upon a time ..."},
    ],
},