Integration guide

Use Tokelang Lite as a prompt compiler, not a black box.

Tokelang Lite returns a smaller compiled prompt when compression is safe and worth it, and returns passthrough when the prompt should stay exact. Always forward the returned `compact` field. To make compiled outputs useful, your model needs the decode system prompt.

1. Compile

Send the user prompt to `/v1/compile` with your API key.

2. Forward `compact`

The same field may contain compiled Tokelang text or unchanged plain text. Forward it with the decode prompt.

3. Set the decode prompt

Without the decode prompt, compiled Tokelang text will not decode correctly.

tokelang.com

curl -X POST https://tokelang.com/v1/compile \
  -H "Authorization: Bearer tk_live_***" \
  -H "Content-Type: application/json" \
  -d '{
  "prompt": "Analyze 10k rows of user data..."
}'

Typical response shape

{

"compact": "input\nsearch q1 sales data db simple\noutput\nsummarize emerging trends detail simple"

}

System prompts

Two prompt variants for two different jobs

`SP1` is the standard decode prompt for normal integrations. `SP2` is the stricter context-file variant for models that should also compress their own working context without dropping meaning.

Tokelang works best with the system prompt provided.

Token counts on this page use OpenAI tiktoken cl100k_base.

Warning

Token counts shown here use OpenAI tiktoken. Models with different tokenizers, such as Claude, Gemini, Llama, or Mistral, may report slightly different counts.

SP1

General decode prompt

Use this for standard Tokelang Lite integrations where the downstream model only needs to understand compiled output.

98 tokens

Interpret compressed lines semantically.

format:
blocks: input process output
line: [step] action payload modifiers
step = order
step refs if else gotoN return
action = first word
payload = middle words
modifiers = trailing style words; none = default
`default style` = block default; line-end modifier overrides
`...` = exact literal; keep unchanged
shape: report list summary comparison definition table

actions are ordinary verbs
modifiers are ordinary style words

SP2

Context-file compression prompt

Use this when the model should try to write its own context or memory files in Tokelang Lite when safe, or compress into structured English when compact syntax would lose meaning.

160 tokens

Interpret compressed lines semantically.
For context files, compress with this format when meaning stays clear.
Compress in a way that retains most of the meaning.

format:
blocks: input process output
line: [step] action payload modifiers
step = order
step refs if else gotoN return
action = first word
payload = middle words
modifiers = trailing style words; none = default
`default style` = block default; line-end modifier overrides
`...` = exact literal; keep unchanged
shape: report list summary comparison definition table

actions are ordinary verbs
modifiers are ordinary style words

Keep code, math, ids, paths, dates, counts, quotes, formulas, tables, and exact contracts literal.
If compression blurs meaning, use concise structured english instead.

Example

Representative compiled prompt from Tokelang Lite

Source prompt

First, search for the Q1 sales data in the database. Then summarize the emerging trends in detail.

Compiled output

input
search q1 sales data db simple
output
summarize emerging trends detail simple

API shape

What the compile endpoint returns

By default the compile endpoint returns only `compact`. That field may contain compiled Tokelang text or unchanged plain text. Always forward `compact` to the model with the decode prompt above.

{
  "compact": "input\nsearch q1 sales data db simple\noutput\nsummarize emerging trends detail simple"
}

Compiled output

The compiler found a smaller, safe Tokelang representation. The model should decode it with the prompt above.

Plain passthrough

The compiler chose fidelity over compression. Forward the unchanged prompt and answer it normally.

Optional diagnostics

`metrics`, `expand`, and `return_ir` are optional

For normal integrations, send only `prompt`. Set these flags to `true` only when you want extra metrics or debugging output from the compiler.

Diagnostics can add latency

`metrics: true`

Adds token counts and compiler timing. Leave it off when you want the leanest possible response.

{
  "prompt": "Analyze pH anomalies and list likely dumping windows.",
  "metrics": true
}

// Response adds:
{
  "metrics": {
    "original_tokens": 42,
    "compact_tokens": 18,
    "reduction_compact_pct": 57.1,
    "compile_ms": 1
  }
}

`expand: true`

Adds a rough natural-language expansion of the compiled prompt. Useful for sanity checks and eval work, not for production forwarding. Add `metrics: true` as well if you also want `expand_ms`.

{
  "prompt": "Analyze pH anomalies and list likely dumping windows.",
  "expand": true
}

// Response adds:
{
  "expanded": "analyze ph anomalies in simple terms list likely dumping windows in simple terms"
}

`return_ir: true`

Adds the internal structured program object. Useful for tooling and inspection, but usually unnecessary for app integrations.

{
  "prompt": "Analyze pH anomalies and list likely dumping windows.",
  "return_ir": true
}

// Response adds:
{
  "program": {
    "blocks": [
      {
        "kind": "process",
        "items": [/* internal IR items */]
      }
    ]
  }
}