Model Explainability: Review the Reason, Not the Chart

Begin by asking who needs the explanation and what they will do with it. A developer debugging an unexpected score, a model validator investigating a dependency and an operator deciding whether to escalate a case need different evidence. The same feature-importance chart cannot be assumed to meet all three needs. Write down the intended reader, the question and the consequence of a misleading answer before selecting an explainer.

For a fictional document-routing model, the question might be why a harmless test document was sent to a specialist queue. The reviewer needs the served model version, the actual input representation, the predicted class and the routing rule. Explaining an earlier notebook model may be technically correct while failing to explain the live workflow. Keep prediction explanation separate from the business rule that used the prediction.

NISTIR 8312 discusses explanation, meaningfulness, explanation accuracy and knowledge limits. Those principles are useful background, not a ready-made acceptance test or a product endorsement. The procedure here is GovernAtlas's proposed operating method. It helps a team assemble and challenge evidence; it does not determine a legal right, certify a model or replace the responsible decision owner.

A feature attribution needs an interpretation context. Ask what reference or background data the chosen method uses, how it was selected and whether it represents the comparison the reviewer intends. Do not describe a contribution as an absolute reason independent of those choices. A feature can look different when compared with a broad population than when compared with a narrow operational cohort.

In a synthetic routing trial, keep the target record and served model fixed while comparing two deliberately documented background cohorts. Preserve both explanations and the reason for each cohort. If rankings change, investigate whether that reflects a changed comparison, an unstable approximation or a processing error. This diagnostic exercise does not establish a universal acceptable amount of variation; the reviewer must relate the difference to the intended use.

Separate a local explanation of one output from a global summary over many records. A global importance view can describe a chosen dataset while concealing cases where the behavior differs. If the decision concerns a particular case, inspect that case. If the decision concerns an operating population, document the sampling, exclusions and subgroup limits before generalizing from a small demonstration.

A counterfactual explanation asks what alternative inputs could change a model output under a chosen method and constraints. That is different from showing that a person can make the change, that the intervention would cause the desired real-world outcome or that the changed input would be accepted by the wider workflow. Record which variables are permitted to vary and which are fixed before interpreting the result.

For a fictional queue-routing example, a proposed change in document completeness may be actionable, while changing a historical event is impossible. Correlated fields can also make an independently varied combination unrealistic. Ask a domain reviewer to inspect the proposed alternative, its feasibility constraints and any omitted dependencies. Retain a rejected alternative with the reason for rejection rather than showing only the most persuasive example.

Do not translate a model-generated alternative into advice about a real person's employment, credit, insurance or health decision. This guide supplies no such determination. In an operational review, the authorized specialists must decide what can be communicated or acted upon. The explainer's technical ability to generate an input does not confer authority to recommend it.

The review packet should identify what was explained, the intended reader, the method and comparison assumptions, the observations, unresolved issues and the next responsible action. Distinguish evidence recorded from evidence reviewed and from an accepted decision. An artifact can exist while failing to answer the question that motivated the review.

Use the working record below to structure the conversation. Its five workstreams begin unreviewed. You can record an evidence reference, a gap or a reason a workstream does not apply; the tool does not validate those choices or calculate readiness. Download the plain-text record before leaving. Reloading clears it, and entries are not transmitted to GovernAtlas.

Revisit the explanation after a material model, feature, preprocessing, explainer, background-data or operating-population change. Retain the prior review so the team can identify which assumption changed. A useful explanation review narrows a decision with traceable evidence and honest limits. It should remain useful even when the outcome is to pause, obtain specialist input or reject the proposed use.