feyninc/chonkie: 🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines

Chonkie is a lightweight ingestion library for building fast, efficient, and robust RAG pipelines.
It is designed to fetch, chunk, refine, embed, and send data directly to vector databases.
The library emphasizes ease of use, speed, low overhead, and minimal installation.
Chonkie supports multilingual text out of the box for 56 languages.
It offers 32+ integrations, including tokenizers, embedding providers, LLMs, refineries, porters, and vector databases.
Users can install only the components they need, or install everything with an all-inclusive package extra.
The basic API centers on chunker classes such as RecursiveChunker, which can be imported and called directly on text.
Chonkie includes a Pipeline API for chaining chunking and refinement steps, with both synchronous and asynchronous execution.
It can also run as a self-hosted REST API, with reusable pipeline configurations stored in a local SQLite database.
Available chunkers include token, fast SIMD-accelerated, sentence, recursive, semantic, late, code, neural, slumber/agentic, table, and TeraflopAI-based chunking.

Your notes

Save this item to your library to add private notes.