An AI knowledge base

Abstract

Engineers want documentation in git, beside the code and open in the editor they already use. Their colleagues quite reasonably want to edit a page in a browser without first learning version control. Most documentation platforms choose one of those experiences. Many also assume one repository and a public audience.

I built a platform that keeps the markdown in a GitHub organisation as the canonical content. A browser editor commits directly to those files. The database holds metadata, links and search embeddings, but never the document body. Workspaces decide which repositories each audience can see, and language models get access over the Model Context Protocol under the same rules as the web interface.

The deployment currently holds 1,778 documents from 109 repositories, split into 23,839 indexed chunks. The interesting decisions were mostly cheap ones made early. Most of the bugs came from equally tidy assumptions that worked for one tenant and quietly failed for several. Retrieval gets a short treatment here and a full one in the companion note.

1. Introduction

The original arrangement worked well for engineers and almost nobody else. Documentation lived under docs/ across dozens of repositories. Editing meant cloning a repository, committing a file and waiting for a deployment. The material was commercially sensitive, which ruled out the obvious public tools.

The alternatives had four practical problems. Browser editing often required moving the source of truth away from git. Static documentation generators expected one repository with one content tree. Scoping an audience to part of the material meant several deployments or a fork with authentication bolted on. The products that did support authentication charged per seat, and the price became difficult once everyone who needed read access was counted.

The brief was fairly direct: leave the source in git, add a browser editor that round-trips clean markdown, and decide which repositories each viewer can see.

What proved reusable

None of the parts is novel. Their arrangement left four patterns I would use again:

1. Keep content in files and use the database as an index. A content hash also makes a very effective cache key.
2. Add the workspace boundary before a second tenant makes it urgent. It groups members, repositories, document overrides, branding and visibility, then gives every authorisation check the same scope.
3. For a small team, a presence channel and soft edit lock solve most accidental overwrites without the cost of collaborative editing.
4. Treat every single-tenant default as suspect. Several bugs were queries or fallbacks that made perfect sense until a second workspace arrived.

2. The content model

Documentation systems tend to put content in a database or in files. A database makes browser editing immediate but duplicates state away from the engineers' working environment. Files keep the canonical copy beside the code, at the cost of an API call every time somebody presses save.

This system takes files. The database holds titles, frontmatter, file paths, content hashes, indexing history, and the resolved document-to-document link graph. It does not hold document content. Reading a document resolves its identifier to a repository and path, consults an in-memory cache keyed by content hash, and on a miss fetches the blob through a thin proxy function. Saving reverses the sequence: the editor submits markdown together with the content hash it began from, the server commits to GitHub, and the returned commit and blob identifiers update the stored hash.

The content hash makes a particularly useful cache key. Two readers of the same document share a cache entry; an edit changes the hash, which changes the cache key, so the next read computes a different key and cannot return stale bytes. There is no explicit invalidation step. A new hash simply asks for a different entry, so stale bytes cannot appear under the new key.

3. The system

3.1 Architecture

3.2 Workspaces

A workspace is the named scope behind everything else. It groups members, admitted either individually by address or collectively by email domain, each holding a viewer, editor, or admin role. It links repositories, and the link carries its own settings, so one repository may appear in several workspaces under different display names, sort orders, read-only flags, and parent groupings. It may additionally grant access to individual documents or folders beyond the repository-level grant. It carries branding, which changes the appearance of the site when that workspace is active. And it may be marked public, making it browsable without authentication.

I added this model before a second tenant could justify it. It cost an extra join in most queries and repaid that cost as soon as an outside audience needed its own scope. By then, adding the audience was configuration work.

3.3 The editor

I evaluated three editor libraries. Two were otherwise strong, but treated markdown as an export format rather than their native representation. A file could not reliably survive a round trip through the editor unchanged. That was enough to rule them out here.

Milkdown was chosen because its schema is markdown. Serialising through a CommonMark preset returns bytes that survive a round trip through a text editor, which means an engineer editing the same file in their own editor sees no gratuitous reformatting.

The editor carries custom plugins for structured components: admonitions in both common syntaxes, tabbed sections, step sequences, card grids, diagram rendering, video embeds with an allowlist applied to the frame source, and image uploads routed through a private bucket and served back behind session authentication.

One editor bug left a useful rule behind. I first rendered diagrams through a node view attached to the code block, which the framework also owns. The two renderers fought and produced a visible flicker. A decoration widget placed after the block fixed it. If an editor framework owns a node type, work around the node instead of competing for it.

3.4 Presence and soft locking

Showing which colleagues are viewing a page does not need a separate presence service. Supabase Realtime provides a channel primitive in which each participant broadcasts an opaque state object and receives everyone else's. There is no server-side state, no schema, and no expiry logic: when a client disconnects, the channel's own teardown removes its entry.

A soft edit lock is layered on the same channel. The first content change after mount marks the local presence state as editing and broadcasts it. Other clients display the editor as read-only with an option to take over. A successful save clears the flag, as does closing the tab.

This is only a soft lock. Two clients can race the take-over control. If both save, the later write wins. For a small team editing a shared document it prevents the common accidental overwrite, which was the actual problem. Proper collaborative editing over conflict-free replicated data types would be a substantial undertaking. The soft lock solves the accidental-overwrite problem without pretending to be a first step towards that system.

3.5 Indexing

Each repository is indexed by a background function. The pipeline resolves the repository's default branch, walks the git tree, and filters to markdown files. For each file it fetches the raw content, hashes it, and skips the file if the hash is unchanged since the last run. Surviving files have their frontmatter parsed and their outbound links extracted, are chunked into sections of roughly 500 tokens with 50 tokens of overlap, split preferentially at heading boundaries, and are embedded. Each chunk stores the full embedding and its truncated prefix, for the reasons set out in the companion note.

Two further stages follow. Link resolution attempts to resolve each extracted markdown link to a document identifier by relative-path arithmetic, marking those that resolve and retaining the others as candidates for later runs. Cleanup removes database rows for files no longer present in the tree. Every run is logged with counts of files indexed, skipped, removed, and errored, a duration, and a correlation identifier.

The function runs under a platform execution limit of 150 seconds and stops accepting new work at 140, recording unprocessed files for a subsequent run. Per-file failures are captured with context rather than aborting the run, which matters when one malformed file would otherwise prevent a repository from being indexed at all.

The indexing stays in the background. Linking a repository starts a run, the interface marks it as unindexed until completion, and the next page load picks up the new state. There is no progress bar for an operation the user does not need to watch.

3.6 Retrieval

Retrieval is covered in the companion note and summarised here. Each chunk carries two embeddings, the full 1,536-dimensional vector and its first 512 dimensions, both stored as 16-bit floats and both indexed. A database function shortlists against the truncated index and reranks the shortlist in memory against the full vectors. Results are then filtered to the repositories the caller may access and deduplicated so that one document does not occupy several result rows.

3.7 Scale

Figures taken from the production database at the time of writing.

Two figures need some context. More repositories are represented in the document table than are currently linked to a workspace, which is a consequence of indexing outliving the link that prompted it: unlinking a repository removes it from view without deleting its rows. That is defensible as caching and indefensible as data hygiene, and a reaping step is outstanding.

The link resolution figure is more useful. Slightly under two thirds of extracted markdown links resolve to a document the system holds. The remainder are largely links to source files, to external addresses, and to documentation that exists in a repository nobody has linked, so the figure is better read as a measure of how self-contained the corpus is than as a defect rate. It is nonetheless the number we would look at first if asked whether the corpus hangs together.

3.8 Access control

Authorisation is applied in three layers, cheapest first. A route gate establishes whether the user holds an editing role anywhere, which removes viewers and anonymous requests without a specific lookup. A resource gate establishes whether they may edit the named repository or document, taking account of document and ancestor-folder locks. A workspace-admin gate covers operations that mutate the repository-to-workspace relationship, and requires admin rights in every workspace whose configuration would change.

I added the third layer later than I should have. Without it, an administrator of one workspace could reorganise a repository as it appeared in another workspace where they held no role, because the operation was authorised against the repository rather than against every scope affected by it. The general lesson is that an operation should be authorised against everything it changes, which is not always the resource it appears to name.

Where the system uses a privileged database client that bypasses row-level security, the calling route performs its authorisation check explicitly before the call, and the client constructor fails loudly if its credentials are absent. An earlier arrangement that fell back silently to an unprivileged client produced failures that were tedious to diagnose, because the symptom appeared at the data layer and the cause was in configuration.

3.9 Agent access

The corpus is exposed over the Model Context Protocol, through an OAuth-gated function offering tools to list, search, and read documents, to find broken links, to list backlinks, and to report health. Each call resolves the caller and applies the same repository filtering as the web interface, so the agent surface and the human surface share one authorisation model rather than two implementations of it.

In practice, this has been the most useful part of the system. Retrieval quality is not the main reason. An engineer can ask a model a question and get an answer from the organisation's current documentation, instead of whatever the model absorbed during training. Vector search makes that quick enough to use, while workspace scoping keeps several audiences separate on one deployment.

4. What went wrong

The failures shared one shape: an assumption held for one repository, tenant or branch name, then broke when there were several.

The indexer originally assumed a default branch named main, and failed against every repository that used something else. Detecting the default branch through the repository API is a single additional call, and the assumption had simply never been examined.

Document URLs originally carried no repository component. Nearly every repository contains a file named README.md, so a URL resolving to that path matched many rows in a query written to expect at most one, and the resolver either failed or returned a document from the wrong repository. The fix was to add an explicit repository parameter as a disambiguator, retaining the broad search as a fallback so that existing links continue to resolve where the path happens to be unique. Any query that expects one row from a shared table while filtering only on a non-unique key contains the same defect. Put the tenant in the query where possible. Otherwise, make the ordering deterministic and document it.

Group folders were originally implicit, existing only insofar as some repository referred to them by name, with the consequence that an empty group disappeared. Making them a first-class table resolved it. Administrators granted their role by email domain rather than individually were rejected by a check that considered only the individual grant. A fallback that displayed all repositories when a user had none linked was removed entirely, on the principle that access should follow explicit configuration and an empty configuration should produce an empty result rather than a permissive one. A user-supplied accent colour, injected as inline styling, was constrained to a strict hexadecimal pattern at both the client and the database write. And a sidebar sort rule that grouped by kind before applying the user's ordering caused manual reordering to revert, which was resolved by sorting purely on user-controlled keys.

None of these bugs was difficult to fix once understood. Diagnosis took the time, usually because the original assumption had looked so reasonable.

5. What is worth taking

Several patterns carry over to other documentation systems. Git can remain the source of truth while the database acts as an index, provided the cache uses a content hash. An early workspace boundary costs a join and can save a later architectural migration. Presence channels make cheap soft locks. Tenant keys in cross-tenant URLs prevent a whole class of resolver bug. Authorisation can run from broad, cheap checks towards the specific ones without weakening the final decision.

I would not repeat two choices. The custom build output configuration was adopted to work around unrelated deployment issues, and cost an afternoon of debugging a routing table for no lasting benefit; the framework's supported deployment preset is the better default. And any sort rule that segregates by kind before applying a user-controlled ordering will fight the user, in this or any other reorderable interface.

6. Limitations

The system has no automated test coverage of the authorisation layer, which is the component where a regression would matter most and where the failures described above actually occurred. The soft lock is not concurrency control and will lose a write under contention. Optimistic concurrency is implemented on the read path through content hashing but is not currently enforced on save, which is a gap rather than a design decision. Search quality has not been evaluated against a judged test collection; the retrieval configuration is justified by cost measurements and by absence of complaint, neither of which is a recall figure. And the deployment serves a single organisation across two workspaces, so claims about the multi-tenant model rest on a model that has been exercised rather than one proven against genuinely independent tenants.

7. Conclusion

The finished system keeps canonical markdown in git, offers a browser editor that preserves it, scopes one deployment for several audiences and lets models query the corpus under the same rules as people. The parts that held up best were decided early and cheaply. The troublesome parts were reasonable assumptions that nobody revisited once they stopped being true.

The least sophisticated choice has aged best: keep the content where the engineers already work. Put only rebuildable state in the database, and keep that state cheap enough to throw away.