Loading and cleaning documents

Askdocs indexes three sources: an internal wiki exported as HTML, a folder of PDF policies and a set of Markdown product docs. Each needs different handling, and the quality of this step sets a ceiling on everything after it.

Keep structure, drop noise

For every document, we want:

• The main text, without navigation menus, footers, cookie banners and repeated boilerplate.
• The structure: titles, headings, lists and tables.
• Metadata: source, URL or path, title, last modified date, owner and access group.

HTML

Extract the main content area and convert it to Markdown, which keeps headings and lists in a compact form that models read well. Libraries that strip boilerplate help, but check their output on a sample of pages. Wikis often hide important content in collapsible sections or tabs that some extractors drop.

PDFs

PDFs store positioned text and drawing instructions, not paragraphs. Simple text extraction often produces broken lines, merged columns, page headers repeated on every page and tables flattened into meaningless sequences of numbers.

Options, roughly in order of effort:

1. A good text extraction library with layout awareness, plus cleanup rules for headers and footers.
2. Specialised document parsing tools that detect headings and tables and output Markdown.
3. For scanned or very complex documents, OCR or vision capable models, which cost more and need spot checks.

Always open a sample of parsed output side by side with the original. Pay special attention to tables, because policies often hide the key details in them.

Deduplicate early

Knowledge bases contain many copies: the same policy in three versions, templated pages with tiny differences. Near duplicates waste index space and crowd search results. Hash normalised text to remove exact duplicates, and use similarity on embeddings or shingles to group near duplicates, keeping the most recent or most authoritative one.

Track versions and freshness

Store last_modified and a content hash for every document. On each sync, only reprocess documents whose hash changed, and delete chunks from removed documents. Stale chunks from deleted pages are a common source of wrong answers.

Record access rules

If some documents are restricted, for example HR policies visible only to managers, store the access group with the document now. Retrofitting permissions into an existing index is painful and risky.

Inspect the corpus

Before chunking, compute simple statistics: number of documents per source, length distribution, languages and the share of empty or near empty documents. Odd numbers here often reveal parsing problems that would otherwise show up much later as mysterious bad answers.