An AI Knowledge Base

Abstract

Documentation for an engineering organisation faces a structural tension. Engineers want the source of truth in git, next to the code, editable in the editor they already have open. Everyone else wants to edit a page in a browser without learning a version control system. The available platforms resolve this tension by choosing a side, and most of them additionally assume a single repository and a public audience.

This paper describes a documentation platform built to hold both sides at once. Markdown files in a GitHub organisation remain the canonical content; a browser-based editor commits to them directly; and a database stores metadata, the link graph, and the embeddings that make the corpus searchable, but never the content itself. A workspace layer scopes which repositories a given viewer may see, which allows one deployment to serve an internal team and a set of externally scoped audiences. The corpus is exposed to language models over the Model Context Protocol, under the same access rules as the web interface.

At the time of writing the deployment holds 1,778 documents drawn from 109 repositories, indexed into 23,839 chunks. We describe the architecture, the decisions that proved load-bearing, and a set of failures that were instructive: most of them arose where a design assumption held for a single tenant and stopped holding for several. Retrieval is treated only in outline here and in detail in the companion note.

1. Introduction

The platform began as a replacement for an arrangement that worked for engineers and for nobody else. Documentation lived as markdown under a docs/ directory across dozens of repositories, which suited the people who were already working in those repositories and excluded everyone who was not. Editing meant cloning, committing, and waiting for a deployment. The material itself was commercially sensitive, so the obvious public documentation platforms were unsuitable on those grounds alone.

Three established options were tried. Each failed on a different axis, and the pattern of failure defined the brief.

The first difficulty was the editing loop. Authoring in an editor, pushing, and waiting for a build is unremarkable to an engineer and prohibitive to a colleague who writes documentation occasionally. The second was the single-repository assumption: the static site generators expect one repository containing one content tree, and the requirement here was to present many repositories' documentation directories as a single navigable site. The third was the absence of any scoping concept. Restricting a given audience to a subset of the material meant either running many separate sites or maintaining a fork with authentication grafted on. A fourth consideration was commercial: the platforms that did offer authentication charged per seat, and the pricing did not survive contact with the number of people who needed read access.

The brief that emerged was therefore to keep git as the source of truth, to put a browser editor over it that produces faithful markdown, and to add a scoping layer that decides which repositories a viewer can see.

Contributions

The individual components are established technology and are not claimed as novel. The contribution is the arrangement, and the following patterns are offered as transferable:

1. A files-as-content model with the database as an index. Content lives in git; the database stores metadata, the link graph, and embeddings. A content-hash cache key makes invalidation implicit rather than something that has to be managed.
2. Workspaces as a multi-tenant primitive adopted before it was needed. A named scope groups members, repositories, per-document overrides, branding, and visibility, and every authorisation decision resolves through it.
3. Presence-channel soft locking in place of concurrency control. A broadcast presence channel provides most of the practical benefit of an edit lock at a small fraction of the cost of real collaborative editing.
4. An account of multi-tenancy failures. Several bugs shared one shape: a query or a default that was correct for a single tenant and silently wrong for several. That shape is worth naming.

2. The content model

Documentation systems generally place content either in a database or in files, and each choice carries a cost. Database-resident content makes editing immediate and duplicates state away from where engineers work. File-resident content keeps the canonical copy where it belongs and puts an API call between the author and the sensation of having saved.

This system takes files. The database holds titles, frontmatter, file paths, content hashes, indexing history, and the resolved document-to-document link graph. It does not hold document content. Reading a document resolves its identifier to a repository and path, consults an in-memory cache keyed by content hash, and on a miss fetches the blob through a thin proxy function. Saving reverses the sequence: the editor submits markdown together with the content hash it began from, the server commits to GitHub, and the returned commit and blob identifiers update the stored hash.

Keying the cache on the content hash does more work than its size suggests. Two readers of the same document share a cache entry; an edit changes the hash, which changes the cache key, so the next read computes a different key and cannot return stale bytes. There is no cache eviction logic anywhere in the system, because invalidation is a consequence of the key rather than an operation performed on the cache.

3. The system

3.1 Architecture

3.2 Workspaces

A workspace is a named scope, and it is the primitive on which everything else depends. It groups members, admitted either individually by address or collectively by email domain, each holding a viewer, editor, or admin role. It links repositories, and the link carries its own settings, so one repository may appear in several workspaces under different display names, sort orders, read-only flags, and parent groupings. It may additionally grant access to individual documents or folders beyond the repository-level grant. It carries branding, which changes the appearance of the site when that workspace is active. And it may be marked public, making it browsable without authentication.

Adopting this model before there was a second tenant to justify it was the single decision that repaid itself most clearly. The cost was an additional join in most queries. The benefit arrived the first time a scope was needed for an audience outside the organisation, at which point the work was configuration rather than architecture.

3.3 The editor

Three editor libraries were evaluated. Two of them, both otherwise strong, failed on the same criterion: their internal document model is not markdown-shaped, so markdown is an export format rather than the native representation, and a round trip through the editor does not reliably reproduce the source file. For a system whose entire premise is that the markdown in git remains canonical, that is disqualifying.

Milkdown was chosen because its schema is markdown. Serialising through a CommonMark preset returns bytes that survive a round trip through a text editor, which means an engineer editing the same file in their own editor sees no gratuitous reformatting.

The editor carries custom plugins for structured components: admonitions in both common syntaxes, tabbed sections, step sequences, card grids, diagram rendering, video embeds with an allowlist applied to the frame source, and image uploads routed through a private bucket and served back behind session authentication.

One implementation note generalises. Diagram rendering was first attempted as a node view attached to the code block node, which the editor framework also claims. Two components rendering the same node produced a visible flicker. The resolution was to attach a decoration widget after the code block rather than replacing it. Where an editor framework already owns a node type, decorating around it is more robust than competing for it.

3.4 Presence and soft locking

Showing which colleagues are viewing a page appears to require a presence service, and does not. Supabase Realtime provides a channel primitive in which each participant broadcasts an opaque state object and receives everyone else's. There is no server-side state, no schema, and no expiry logic: when a client disconnects, the channel's own teardown removes its entry.

A soft edit lock is layered on the same channel. The first content change after mount marks the local presence state as editing and broadcasts it. Other clients display the editor as read-only with an option to take over. A successful save clears the flag, as does closing the tab.

This is not a lock, and it is worth being precise about what it does not do. Two clients can race the take-over control. If both save, the later write wins. For a small team editing a shared document it prevents the common accidental overwrite, which was the actual problem. Proper collaborative editing over conflict-free replicated data types is a substantial undertaking, and the soft lock was adopted deliberately as the cheaper approximation rather than as a step towards it.

3.5 Indexing

Each repository is indexed by a background function. The pipeline resolves the repository's default branch, walks the git tree, and filters to markdown files. For each file it fetches the raw content, hashes it, and skips the file if the hash is unchanged since the last run. Surviving files have their frontmatter parsed and their outbound links extracted, are chunked into sections of roughly 500 tokens with 50 tokens of overlap, split preferentially at heading boundaries, and are embedded. Each chunk stores the full embedding and its truncated prefix, for the reasons set out in the companion note.

Two further stages follow. Link resolution attempts to resolve each extracted markdown link to a document identifier by relative-path arithmetic, marking those that resolve and retaining the others as candidates for later runs. Cleanup removes database rows for files no longer present in the tree. Every run is logged with counts of files indexed, skipped, removed, and errored, a duration, and a correlation identifier.

The function runs under a platform execution limit of 150 seconds and stops accepting new work at 140, recording unprocessed files for a subsequent run. Per-file failures are captured with context rather than aborting the run, which matters when one malformed file would otherwise prevent a repository from being indexed at all.

The process is deliberately invisible. Linking a repository triggers a background run; the interface shows the repository as unindexed until the run completes, and the next page load reflects the new state. There is no progress indicator, because the operation is not one the user needs to watch.

3.6 Retrieval

Retrieval is covered in the companion note and summarised here. Each chunk carries two embeddings, the full 1,536-dimensional vector and its first 512 dimensions, both stored as 16-bit floats and both indexed. A database function shortlists against the truncated index and reranks the shortlist in memory against the full vectors. Results are then filtered to the repositories the caller may access and deduplicated so that one document does not occupy several result rows.

3.7 Scale

Figures taken from the production database at the time of writing.

Two of these merit comment. Rather more repositories are represented in the document table than are currently linked to a workspace, which is a consequence of indexing outliving the link that prompted it: unlinking a repository removes it from view without deleting its rows. That is defensible as caching and indefensible as data hygiene, and a reaping step is outstanding.

The link resolution figure is the more interesting one. Slightly under two thirds of extracted markdown links resolve to a document the system holds. The remainder are largely links to source files, to external addresses, and to documentation that exists in a repository nobody has linked, so the figure is better read as a measure of how self-contained the corpus is than as a defect rate. It is nonetheless the number we would look at first if asked whether the corpus hangs together.

3.8 Access control

Authorisation is applied in three layers, cheapest first. A route gate establishes whether the user holds an editing role anywhere, which removes viewers and anonymous requests without a specific lookup. A resource gate establishes whether they may edit the named repository or document, taking account of document and ancestor-folder locks. A workspace-admin gate covers operations that mutate the repository-to-workspace relationship, and requires admin rights in every workspace whose configuration would change.

That third layer was added later than it should have been. Without it, an administrator of one workspace could reorganise a repository as it appeared in another workspace where they held no role, because the operation was authorised against the repository rather than against every scope affected by it. The general lesson is that an operation should be authorised against everything it changes, which is not always the resource it appears to name.

Where the system uses a privileged database client that bypasses row-level security, the calling route performs its authorisation check explicitly before the call, and the client constructor fails loudly if its credentials are absent. An earlier arrangement that fell back silently to an unprivileged client produced failures that were tedious to diagnose, because the symptom appeared at the data layer and the cause was in configuration.

3.9 Agent access

The corpus is exposed over the Model Context Protocol, through an OAuth-gated function offering tools to list, search, and read documents, to find broken links, to list backlinks, and to report health. Each call resolves the caller and applies the same repository filtering as the web interface, so the agent surface and the human surface share one authorisation model rather than two implementations of it.

This has been the most useful part of the system in practice, and the reason is not really about retrieval quality. An engineer asking a question of a language model gets an answer grounded in the organisation's current documentation rather than in whatever the model absorbed during training. The vector search makes that fast enough to be worth doing, and the workspace scoping makes it safe to do where several audiences share one deployment.

4. What went wrong

The failures worth recording share a shape. Each was an assumption that held while there was one repository, one tenant, or one branch name, and stopped holding when there were several.

The indexer originally assumed a default branch named main, and failed against every repository that used something else. Detecting the default branch through the repository API is a single additional call, and the assumption had simply never been examined.

Document URLs originally carried no repository component. Nearly every repository contains a file named README.md, so a URL resolving to that path matched many rows in a query written to expect at most one, and the resolver either failed or returned a document from the wrong repository. The fix was to add an explicit repository parameter as a disambiguator, retaining the broad search as a fallback so that existing links continue to resolve where the path happens to be unique. The general form of this bug is worth stating: a query that expects a single row, run against a table that is shared between tenants and filtered only by a non-unique key, is a latent defect. Either the tenant belongs in the query, which is preferable, or the ordering must be made deterministic and documented.

Group folders were originally implicit, existing only insofar as some repository referred to them by name, with the consequence that an empty group disappeared. Making them a first-class table resolved it. Administrators granted their role by email domain rather than individually were rejected by a check that considered only the individual grant. A fallback that displayed all repositories when a user had none linked was removed entirely, on the principle that access should follow explicit configuration and an empty configuration should produce an empty result rather than a permissive one. A user-supplied accent colour, injected as inline styling, was constrained to a strict hexadecimal pattern at both the client and the database write. And a sidebar sort rule that grouped by kind before applying the user's ordering caused manual reordering to revert, which was resolved by sorting purely on user-controlled keys.

The common thread is that these were not difficult to fix once characterised. The work was in the diagnosis, and the diagnosis was slow in proportion to how reasonable the original assumption had seemed.

5. What is worth taking

Several of these patterns transfer to anything documentation-shaped.

Keeping git as the source of truth with the database as an index avoids content duplication and the synchronisation problems that follow from it, provided the cache is keyed on a content hash so that invalidation is implicit. Adopting a workspace primitive before a second tenant exists costs one join and saves an architectural migration. Presence channels provide soft locking at a cost that is difficult to justify not paying. Explicit tenant keys in URLs that cross tenant boundaries prevent an entire class of resolver bug. And layering authorisation from cheap and broad to expensive and specific keeps the common path fast without weakening the specific checks.

Two things are not worth copying. The custom build output configuration was adopted to work around unrelated deployment issues, and cost an afternoon of debugging a routing table for no lasting benefit; the framework's supported deployment preset is the better default. And any sort rule that segregates by kind before applying a user-controlled ordering will fight the user, in this or any other reorderable interface.

6. Limitations

The system has no automated test coverage of the authorisation layer, which is the component where a regression would matter most and where the failures described above actually occurred. The soft lock is not concurrency control and will lose a write under contention. Optimistic concurrency is implemented on the read path through content hashing but is not currently enforced on save, which is a gap rather than a design decision. Search quality has not been evaluated against a judged test collection; the retrieval configuration is justified by cost measurements and by absence of complaint, neither of which is a recall figure. And the deployment serves a single organisation across two workspaces, so claims about the multi-tenant model rest on a model that has been exercised rather than one proven against genuinely independent tenants.

7. Conclusion

The system holds a position that the available platforms do not: canonical markdown in git, a browser editor that produces faithful markdown, scoping that allows one deployment to serve several audiences, and a corpus that language models can query under the same access rules as people. None of the components is novel. The arrangement is useful, and the parts of it that proved load-bearing were mostly decided early and cheaply, while the parts that caused trouble were mostly assumptions that nobody thought to examine because they were true at the time they were made.

The pattern that has aged best is the least sophisticated one: keep the content where the engineers already are, store everything else in a database that can be rebuilt from it, and make the derived state cheap enough to discard.