xtox
The universal document on-ramp for agent cognition. xtox extracts structured text, mathematical equations, tables, and full repository ASTs, delivering token-optimized representations ready for prompt context and RAG indexing.
npx @celladore/xtox parse --format ast-token-opt ./docs - Multi-format parsing
- extracts PDFs, DOCX, LaTeX, Markdown, and codebase structures
- Open source (MIT)
- github.com/celladore/xtox with zero telemetry
- AST-optimized
- flattens complex structures into token-efficient agent inputs
How xtox Connects to the Stack
Celladore is engineered as a unified, composable toolchain where every tool feeds upstream contexts and downstream verification loops without friction.
→ Retort
Parsed codebase trees and token-optimized ASTs feed directly into Retort worker agents without exhausting context windows.
→ Baton
Extracted specification sheets and PRDs are attached as deterministic task requirements in Baton.
→ Sluice
Flattened documents reduce prompt token counts by 64%, cutting Sluice routing latencies and API billing costs.
Technical Matrix
- Runtime & Engine:
- TypeScript / Rust Streaming Parser
- Deployment Target:
- CLI / Node Package / WASM
- Communication Protocol:
- Streaming Token-Optimized AST Protocol
- Licensing Tier:
- MIT Open Source
Invented from Real Scars
Ingesting raw multi-megabyte WhatsApp chats, PDFs, and codebase directories blew prompt context windows with noisy formatting and broken table structures.
Prompt context window exhaustion, lost mathematical formulas, and garbled RAG chunking.
Architectural Features & Guarantees
Modular design built for high velocity, deterministic reliability, and strict enterprise isolation.
Structured Document Extraction
Preserve table structures, formulas, and headings from PDFs and scientific papers into clean, LLM-ready Markdown.
Codebase & Repo Flattening
Transform entire multi-file code repositories into dependency-ordered, token-optimized context blocks for LLM reasoning.
Formula & Equation Integrity
Losslessly convert LaTeX math and scientific notation without loss of precision for mathematical reasoning agents.
Streaming Token Reducer
Achieve up to 65% token compression by stripping non-semantic whitespace and formatting boilerplate before model ingestion.
Typed API Contract & Snippets
Live cURL, TypeScript, and Python invocation payloads for xtox.
Unified Developer Interface
Convert PDFs, DOCX, LaTeX, Markdown, and repository structures into LLM-optimized tokens and structured AST.
import { XtoxConverter } from "@celladore/xtox";
import fs from "node:fs";
const xtox = new XtoxConverter({ apiKey: process.env.XTOX_API_KEY || "YOUR_XTOX_API_KEY" });
const result = await xtox.convertDocument({
file: fs.createReadStream("architecture_spec.pdf"),
format: "pdf",
options: {
extractTables: true,
preserveMath: true,
flattenAst: true,
},
});
console.log(`Extracted ${result.chunks.length} chunks, ${result.tokenCount} tokens`);Repository Activity
Live contributor data and repository metrics pulled directly from GitHub's public API.
Last repository update: Aug 21, 2026
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4)
Status Notice: xtox is actively developed with source code available on GitHub under MIT. Clone and test document ingestion pipelines locally. Get in touch with the founder .