Why a read-only, permission-aware index over the storage you already have is a better foundation for search and AI than another copy of your data, what such a layer has to do, and how KDBL Context Lake does it.
The most valuable knowledge in most organisations is not in a database. It is in files: contracts, reports, spreadsheets, scans, slide decks and recordings, spread across object stores, file shares and document libraries, accumulated over decades and governed by permissions nobody fully remembers setting.
Every attempt to make that knowledge available to search and AI runs into the same choice. Either copy the files into a new platform, and inherit a second estate with its own permissions, cost and drift, or leave them where they are and accept that assistants cannot reach them. This paper argues for a third option: a context lake, a read-only layer that indexes what your storage holds and what is inside each file, honours the permissions the files already carry, and serves the result to people and assistants without moving anything.
We describe what such a layer must do, and how KDBL Context Lake (K-Lake) does it.
Ask a data team what the organisation holds and they will describe the warehouse. Ask an operations manager, a lawyer or an engineer and they will describe a share. Somewhere there is a folder with the answer to most questions the business asks, and the person who knows which folder is about to retire.
Three things make this estate hard to use.
Assistants have made the problem urgent. Once a team has used a capable model, the question "why can't it just read our contracts?" arrives within the week. The honest answer, for most estates, is that the contracts are on a share with permissions nobody wants to reason about, and the safest thing has been to say no.
The conventional answer is to copy. Export the files to a search platform, a vector database or a vendor's cloud, index them there, and put the assistant on top. It works in a demo and fails in production, for reasons that have nothing to do with the model.
The copy drifts from the original the moment it is made. Files change, are deleted, are moved; permissions change more often than content. Keeping the copy synchronised is an integration project that never ends, and the copy is a second attack surface with its own backups, retention and residency obligations.
Most copying pipelines carry content and drop access control, then attempt to rebuild it with a simpler model: a tag per collection, a group per folder. The result is either too open, so an assistant surfaces a salary spreadsheet to someone who could never have opened it, or too closed, so the assistant is useless. Rebuilding thirty years of ACLs by hand is not a project anyone finishes.
For regulated organisations, and for anyone with a data-residency commitment, sending file content to a hosted index or a model provider is not a configuration question. It is a contractual and sometimes legal one, and it is often simply not allowed.
The problem is not that the files are hard to reach. It is that reaching them safely means carrying their permissions, their location and their history with you. A copy carries none of those.
A context lake is a read-only layer over existing storage that does four things:
The name is deliberate. A data lake collects structured and semi-structured data into one place for analysis. A context lake does not collect anything. It builds context around the files where they already live: what they are, what they say, who they concern, and who may see them. The originals stay in the systems that own them.
From working with organisations on this problem, we have arrived at seven properties a context lake needs. Any product claiming the role should be measured against them.
The layer must never write to a source. Not to add a tag, not to normalise a name, not to move a stale file. Everything it derives is stored alongside the file record, never in the file. This is what makes it safe to point at a production share without a change board.
NTFS and NFSv4 ACLs, POSIX ownership and the equivalents on object stores must be captured at crawl time, correlated to the identities in the organisation's directory, and enforced on every query. Not filtered in the application after the fact; enforced where the data is stored, so that no code path can forget.
Full-text and semantic search over the extracted content of documents, spreadsheets, PDFs and scans, and over transcripts of audio and video. A result must land on the paragraph or the timecode, not the file.
Reading the files is where content is most exposed. Extraction, OCR and transcription must run on engines the organisation hosts, so file content never leaves its boundary to be indexed.
Every caller, human or agent, must be a named principal signed in through the organisation's own identity provider or holding a token tied to one user. Shared service accounts defeat the purpose of carrying permissions at all.
An assistant must connect as the individual user, inherit that user's trimming, cite the file every fact came from, and leave an audit record. Connecting AI must not widen anyone's access, and it must be possible to prove that after the fact.
A layer that reports a partial number as ground truth is worse than one that reports nothing. Where a source has not been extracted, or a facet depends on metadata only some sources capture, the layer must say so.
KDBL Context Lake was built to these requirements. It runs as a small set of services in the customer's Kubernetes cluster: an API, a web console, stateless crawl workers, a separately sized extraction fleet, an optional MCP server, a managed database and a work queue.
| Requirement | In K-Lake |
|---|---|
| Read-only | Sources are read with read-only access. Extracted text, redacted text and entities are stored alongside the file record. The MCP server serves extracted text and metadata, never original bytes. |
| Native permissions | ACLs and ownership are captured per source and correlated to directory identities by directory enrichment. Trimming is enforced by row-level security in the database on every query, across console, CLI, API and MCP. |
| Content | Hybrid lexical and semantic search over extracted text, with results grouped by document and deep-linked to the page or timecode. |
| Self-hosted extraction | A fast default engine, a document engine for scans and tables, a vision-model engine and a media engine, all running in the cluster and scaling out as a fleet. |
| Identity-first | Federated sign-in over OIDC and OAuth 2.1 to Entra ID, Okta, Google, Keycloak or any compliant provider. Personal access tokens act as exactly one user. |
| Assistants | A Model Context Protocol server that is itself an OAuth 2.1 authorization server. Assistants connect as the user, receive a citation for every fact, and every call is recorded with principal, tenant and files returned. |
| Honest coverage | Extraction and enrichment coverage are reported per source. Discover states the share of files a facet is based on wherever it depends on captured metadata. |
Beyond the baseline, two capabilities turn the index into something more than search. Entity enrichment builds a knowledge graph of the people, organisations, locations and agreements the documents name and the relationships they assert, distinguishing an asserted relation from mere co-occurrence. Discover aggregates the whole estate into a governance view: composition, searchability and exposure signals, with a click-through to the files behind every number.
Smart Actions let the layer act as well as observe. After each crawl, every file flows through a per-source pipeline: extract, redact sensitive patterns from what search and AI return, notify your own systems with a signed webhook, retry hard documents on a heavier engine. Change a rule and only that step re-runs; add a step and it backfills.
A context lake does not require a programme. The organisations that get the most from it start narrow and widen.
The files are not going anywhere, and neither are their permissions. An architecture that treats them as an obstacle to be copied around will always be fighting drift, residency and access control. An architecture that treats them as the system of record, and builds context around them where they are, has none of those fights to have.
That is what a context lake is for. It is what K-Lake does.
The full stack runs on a single VM in one command, with a 30-day evaluation licence. We will walk you through it on a representative estate.
Request a demo Explore K-Lake