What We Learned While Developing CLEVERINA Incident Assistant

CLEVERINA Incident Assistant is the AI-assisted investigation workflow we have been developing for Microsoft 365 environments. It is currently in Preview / MVP — not production-ready, deliberately narrow in scope, and being shaped by real investigations with a small number of design partners. This article shares what we have learned so far.

1. The hardest decision is what not to build

AI in security operations is a category where you can plausibly build almost anything: triage assistants, autonomous responders, threat-hunting copilots, report writers, tabletop simulators. The constraint that has served us best is asking, for every feature: 'is the human responder faster or more accurate because of this?' If the answer is uncertain, the feature does not ship.

2. Evidence provenance is the product

The first version of any AI feature in this space is tempted to produce smooth narrative. The version that survives contact with a real investigation produces narrative with citations — every claim linked back to a queryable evidence record. We rebuilt our evidence layer twice before getting it right, and we would still recommend starting there on day one.

3. Human oversight is not a checkbox

Every output the assistant produces is a draft. Every suggested action is reviewed by a named human before execution. This is not a compliance posture — it is what makes the tool trustworthy enough for analysts to use it on real incidents. The moment the assistant takes an autonomous action, the trust model has to be rebuilt from scratch.

4. Scope discipline beats model selection

Most of the gains we have measured come from scoping the assistant to a narrow, well-defined task — summarize this entity's activity, draft this section of the report, propose this set of queries — rather than from switching between models. The instinct to chase model upgrades is strong; the impact of scope discipline is larger and more durable.

5. Investigations have a real lifecycle and the tool has to fit it

Triage, containment, eradication, recovery, lessons-learned — each phase has different evidence needs, different audiences and different time pressures. A single 'AI for incidents' surface that ignores those phases ends up useful in none of them. We now design feature by feature against a specific phase.

6. Logging the tool is as important as logging the incident

Every prompt, every model response and every action taken on the assistant's suggestion is recorded. This serves three audiences at once: the analyst reviewing their own work, the team improving the assistant, and the regulator who may eventually ask how AI was used in handling an incident. Building this in from the first release is much cheaper than retrofitting it.

An anonymized example

An early design-partner walkthrough involved a simulated phishing-triggered token theft. The assistant produced a clear triage summary in under a minute and a draft narrative in under five. The most valuable feedback from the analyst was not about the narrative quality — it was about a missing column in the evidence view that made one cited record harder to verify. That feedback shaped the next sprint more than any model change would have. The lesson: time with real users on real workflows is the highest-leverage activity in this kind of product.

What is next

The Preview / MVP continues with a small set of design partners. Public availability, marketplace listing and broader rollout are tracked on the roadmap. We intend to keep the scope narrow for the foreseeable future — depth over breadth, with explicit human oversight and audit by default.

Conclusion

Building AI for security operations is a discipline of restraint as much as engineering. Narrow scope, cited evidence, human oversight and complete audit logging are the foundations we keep returning to. If the direction is relevant to your team, the product page describes Incident Assistant in more detail and the contact page is the right place to talk about Preview participation.