Chunking & Embeddings

This guide walks through the two building blocks of the GPT chat knowledge base:

• Chunking – turning raw documents or text snippets into small, referenceable sections.
• Embeddings – storing those chunks inside a FAISS index so the conversational agent can search them.

The latest release introduces a dedicated chunk-only API. You can now preview how material will be segmented before deciding to embed it. All previously shipped embedding capabilities continue to work without any changes.

This server is running at https://app.laisky.com/ and is available to all registered users.

Authentication

Most endpoints in the GPT chat service use the same lightweight authentication scheme:

• Authorization: Bearer <app-key>
• X-Laisky-User-Id: <uid> (optional, defaults to the hashed app key)
• X-Laisky-User-Is-Free: true|false (absence or false marks the account as paid)
• X-Laisky-Openai-Api-Base: https://api.openai.com/ (optional override)

Clients that only need chunk previews can reuse these headers; no additional scopes are required.

Chunk Preview API

• Route: POST /gptchat/chunks
• Purpose: Split plain text or supported document formats (PDF, Word, PowerPoint, Markdown, HTML, TXT) into chunks and inspect the metadata that would later be embedded.
• Compatibility: Existing embedding endpoints (/gptchat/files, /gptchat/ctx/...) remain unchanged.

Accepted payloads

Content type	Description
application/json	Provide text directly as a string. Optional fields: metadata_name, chunk_size, chunk_overlap, max_chunks.
multipart/form-data	Include a file field for binary uploads and/or a text field. You may also pass metadata_name, chunk_size, chunk_overlap, max_chunks as simple form fields.
text/*	Raw text body. Optional X-Laisky-Metadata-Name header supplies the source name.
other binary types	Send the file body as-is and supply X-Laisky-File-Name (with extension) plus optional X-Laisky-Metadata-Name.

Chunking parameters

• chunk_size – Maximum tokens per chunk. Defaults to 500.
• chunk_overlap – Token overlap between adjacent chunks. Defaults to 30. Must be smaller than chunk_size.
• max_chunks – Cap on total chunks returned. Defaults to 600 for free users and 10000 for paid users. The service automatically clips any larger request to your account tier.
• metadata_name – Prefix for the generated source metadata. When omitted, the service uses the file name (for uploads) or inline-text.

JSON example (inline text)

curl -X POST "${HOST}/gptchat/chunks" \
	-H "Authorization: Bearer ${APP_KEY}" \
	-H "X-Laisky-User-Id: demo-user" \
	-H "Content-Type: application/json" \
	-d '{
		"text": "My doc intro...",
		"metadata_name": "project-notes",
		"chunk_size": 800,
		"chunk_overlap": 40
	}'

Multipart example (PDF upload)

curl -X POST "${HOST}/gptchat/chunks" \
	-H "Authorization: Bearer ${APP_KEY}" \
	-H "X-Laisky-User-Id: demo-user" \
	-F "file=@/path/to/report.pdf" \
	-F "metadata_name=report-2025" \
	-F "chunk_size=600" \
	-F "chunk_overlap=60"

Raw binary example (HTML body)

curl -X POST "${HOST}/gptchat/chunks" \
	-H "Authorization: Bearer ${APP_KEY}" \
	-H "X-Laisky-File-Name: public.html" \
	-H "X-Laisky-Metadata-Name: public-site" \
	--data-binary @page.html

Response payload

{
  "chunks": [
    {
      "text": "...",
      "metadata": {
        "source": "project-notes?chunk=1"
      }
    }
  ],
  "total": 4,
  "chunk_size": 800,
  "chunk_overlap": 40,
  "max_chunks": 600,
  "source": "project-notes",
  "origin": "text"
}

• chunks contains the raw text plus metadata that downstream embedding calls reuse.
• origin indicates whether the request came from text or file input.
• source is the canonical prefix applied to each chunk`s metadata.
• max_chunks reflects the resolved cap after applying account limits.

Errors are returned as 400 responses with a descriptive message when inputs are invalid (e.g., unsupported extension, chunk overlap larger than the size, missing file name for binary uploads).

Embedding pipeline (unchanged)

Once you are satisfied with the chunk preview, continue with the existing endpoints:

1. Upload & Index – POST /gptchat/files
   • Upload encrypted datasets for private indices. The worker encrypts the source document, builds embeddings, and stores everything in Minio.
2. Manage datasets – GET /gptchat/files
   • List uploaded datasets and the subset currently active in the user cache.
3. Build chatbot indices – POST /gptchat/ctx/build
   • Combine multiple datasets into an interactive chatbot index.
4. Query embeddings – GET /gptchat/ctx/chat or GET /gptchat/query/query
   • Retrieve answers from private or prebuilt indices.

All of these endpoints continue to rely on the same metadata emitted by the chunk preview service, so no client-side adjustments are required.

Best practices

• Use the preview API before uploading large files to ensure the chunk count fits within your quota.
• Keep metadata_name stable across uploads so references remain predictable.
• Free accounts can request a smaller max_chunks to trim responses without waiting for the auto-limit.
• After chunking, you can reuse the same payload with embedding_file workflows to build new datasets.

Local testing

• Run pytest tests/test_chunk_preview.py to exercise the chunk-only endpoint.
• Existing tests (tests/test_vector_store.py, etc.) still validate serialization of embedding stores.
• The test suite mocks storage credentials, so no external services are required.