TokenCounter
type TokenCounter = (text) => number;Function that counts the number of tokens in a text string. Implementations may use js-tiktoken, gpt-tokenizer, or any other tokenizer. When not provided, the token chunker falls back to character-based estimation (~4 chars per token).
Parameters
Section titled “Parameters”| Parameter | Type |
|---|---|
|
|
|
Returns
Section titled “Returns”number