AI prompts belong in settings

The problem with a prompt in the codebase

An AI feature is a question plus a model. Hide the question and users see only the answer, with no way to judge whether it was a fair one.

Then the question changes, quietly, in a release. Outputs shift. Nobody can tell whether the model got better, the data changed, or somebody edited a sentence.

And the people who understand the domain cannot touch it. The person who knows what a risky project meeting looks like is not the person with deployment access.

What we made visible instead

The prompts screen lists each prompt with its status, its version and the model it runs on. Ours reads meeting signals, active, version two, on a named small model.

Every generated result is stamped with the prompt version and model that produced it. An old signal carries the older version, and the screen says so rather than pretending the history is uniform.

There is a re-run action that re-analyses recent meetings after an edit, so a change can be applied to existing records instead of only affecting future ones.

• The prompt text, readable by an administrator.
• A version number that increments on edit.
• The model name, on the prompt and on every result.
• A backfill action to re-analyse recent records.
• A note that older results came from earlier versions.

Why a person still reads the result

A visible prompt is still a prompt. The model can be wrong, confidently, in the same way it always could.

Our meeting analysis produces a risk verdict and flags such as no client present. Those are signals for a human to look at, not findings, and they are deliberately admin-only rather than shown to everyone in the meeting.

The policy assistant follows the same principle and points people back to the published policy for anything important. Making a prompt editable makes it accountable, and a person still decides what to do with the answer.

Two AI features, each with a clear job

Meeting signals read a transcript and flag risk for leadership. Policy search answers a question from the company's own published policies.

Each does a narrow job a person can check against the source: the transcript in one case, the policy text in the other.

Two accountable AI features, with their prompts in the open, is a better claim than a long list nobody can inspect.

What to ask any vendor about their AI

The useful questions are about provenance, not capability. A demo shows what the feature can do on a good day. These show what happens on a normal one.

• Can an administrator read the instruction the model is given?
• Is each result stamped with the version that produced it?
• Who sees the output, and can the subject of it see it too?
• What happens when the model is wrong: correct it, or ignore it?
• Which provider does it depend on, and what breaks if that stops?

Who the output is for

Visibility is a design decision as much as prompt text, and it is the one most likely to cause harm.

Our meeting analysis is admin-only on purpose. A risk verdict about a client conversation, shown to everyone who attended it, changes how people behave in the next one.

There is a related guard worth knowing about. The channel that receives meeting digests refuses a name matching the pattern used for shared client channels, so a configuration slip cannot route a confidential assessment to a client.

That guard is not clever. It is just an admission that a text box next to something confidential needs more than a text box.

Versioning is what makes a result readable

Editability gets the attention, and versioning is the part that actually helps six months later.

Without it, a set of results is a single undifferentiated pile. Some came from one instruction, some from another, and no row says which.

With it, an old result carries its origin. You can tell that March looked different because the question changed, rather than assuming the projects did.

It also makes an edit reversible in the only sense that matters. Not undoing the text, which is easy, but knowing which records to re-run once you have.

Same reasoning as a migration or a schema change. The thing worth recording is not the current state. It is which state produced what.