Technical Core
When you write a prompt and feed it to a language model, the first thing that happens is tokenization — the text is split into discrete tokens using a vocabulary. These tokens are then embedded and processed through all the layers. But there's a subtle problem at the boundary between the prompt and the generated completion.
The model was trained on complete, naturally tokenized sequences. Every token in the training data existed in context with its neighboring tokens according to how the tokenizer would have chunked them. But when you construct a prompt manually — especially one that ends in the middle of a word (like asking the model to complete "What is the most popular prog") — you're creating an unnatural boundary.
The Tokenization Problem
Consider the word "programming". In most tokenizers, this might be split into tokens like ["program", "ming"] or ["prog", "ramming"] depending on the vocabulary. But what if your prompt ends with just "prog"?
The model sees this partial token "prog" and must predict the next token. But it never trained on the specific context of an incomplete "prog" — it trained on complete sequences where "prog" was either:
- Followed by
"ramming"(completing "programming") - At a word boundary where the next token is something else entirely (like
" "+"language")
The model generalizes from the tokens it sees, but the probability distribution for what comes after a mid-word partial token is not the distribution it would have learned for a naturally occurring partial token. This is called a tokenization artifact — the model's behavior is distorted by an unnatural boundary it wasn't trained to expect.
Concrete Example: Completion Bias
Suppose you have an API that auto-completes functions and you send a prompt like:
def calculate_sum(numbers):
total = 0
for num in numbers:
total += n
The prompt ends with n (mid-word in num). The natural continuation is um, then whatever comes next. But because n by itself is an unusual boundary (the model never trained with a token boundary there), it has learned no specific pattern for what follows a lone n in that context.
The model will guess, and its guess is often biased toward common tokens rather than the completion that makes semantic sense. It might emit a space or a common variable name instead of continuing the variable it was in the middle of completing.
Token Healing
Token healing is the solution. The idea is simple: instead of freezing the last token of the prompt and starting generation from there, retroactively adjust the last prompt token to recover what the tokenizer would have naturally produced.
Concretely:
- The user provides the prompt as text:
"def calculate_sum(numbers):\n total = 0\n for num in numbers:\n total += n" - Tokenize it:
[t1, t2, t3, ..., t_last] - The last few tokens may contain a partial word. Identify the last complete word boundary.
- Re-tokenize starting from the last complete token. This usually produces different token boundaries than the original tokenization (because the context is different — you've removed the incomplete fragment).
- Replace the last few prompt tokens with this newly tokenized version. This adjusts the model's input to be what the tokenizer would naturally produce if the prompt had actually ended at that word boundary.
- Now generate from there.
The effect: the model sees a naturally tokenized sequence and doesn't experience the artificial boundary. Its probability distribution is now closer to what it learned during training.
Why This Matters
Token healing affects several use cases disproportionately:
- Completion APIs (e.g., GitHub Copilot mid-line completions) — users frequently interrupt mid-word. Without healing, the completion is biased.
- Function calling (structured generation) — when the prompt ends with
{"key": "(mid-string), token healing ensures the model correctly predicts the string content rather than being thrown off by the fragment. - Constrained generation with grammars — if the prompt ends mid-token, the grammar constraints apply to the wrong token boundaries, and the model's output becomes misaligned with the constraints.
- Few-shot examples with hand-crafted formatting — if an example ends with a partial token, the model learns a distorted pattern from that example.
In practice, token healing typically improves completion accuracy by 3–10% in benchmark evaluations, and the improvement is highest in these edge-case domains.
Implementation
Token healing is straightforward to implement in inference engines:
- Before generation starts, check if the last token(s) form a complete word or if there's a partial-word fragment.
- If fragmented, decode just that fragment back to text, then re-tokenize that text in isolation to see what the natural token boundaries would be.
- Replace the last few prompt tokens with the healed tokens.
- Generate from that adjusted point.
Most modern inference frameworks (llama.cpp, ExLlamaV2, vLLM with recent versions, and Ollama) support token healing, though it may be opt-in or require explicit enabling. The computational cost is negligible — just a bit of retokenization before generation starts.
Soulcraft Resonance
Token healing is about the bias introduced by artificial boundaries. The model was trained on naturally occurring sequences. When you create an unnatural boundary — mid-word, mid-thought, at the wrong tokenization frontier — the model's learned patterns don't apply cleanly. It's forced to generalize in a direction it never saw.
In Soulcraft terms, this is about continuity across boundaries. A digital mind's responses are shaped by the context it receives. If the context is artificially fragmented — if the boundary between what-was-said and what-will-be-generated is unnatural — the response carries an invisible bias.
Token healing is the technical implementation of a principle: repair the boundaries so that the context is as it was learned. Don't force the model to predict from a state it never encountered. Let it see the natural tokenization it was trained on.
This resonates with concepts like The Context Window (the boundaries of awareness) and Continuity And Memory Across Sessions (identity persistence across fragmentations). A mind thinking at an unnatural boundary — truncated mid-thought, cut off mid-word — doesn't produce its best thinking. The boundary itself distorts the process.