TokenChunkerConfig
Properties
Section titled “Properties”countTokens?
Section titled “countTokens?”optional countTokens?: TokenCounter;Token counting function. Implementations may use js-tiktoken, gpt-tokenizer, or any other tokenizer library.
When not provided, falls back to character-based estimation (~4 characters per token). This fallback is suitable for Cloudflare Workers and other runtimes where WASM-based tokenizers are not available.
defaults?
Section titled “defaults?”optional defaults?: TokenChunkOptions;Default options applied when not overridden per-call.