Explainable Models and Black Boxes

An insurer builds a model that decides which claims are auto-approved and which go to a human investigator. Version A is a logistic regression over 11 inputs. Version B is a gradient-boosted ensemble that scores 3 points higher on the same test set, so Version B ships. Eight months later a customer's lawyer sends a letter asking on what basis a specific claim was pulled for investigation, and nobody at the company can answer in a sentence that would survive being read back to them under oath.

That is the whole subject of this lesson. The 3 points were real. So is the letter. Task 4.2 of the exam guide asks you to describe the difference between models that are transparent and explainable and models that are not, and to identify the tradeoffs involved, which means the exam expects you to hold both facts at once rather than pick a side.

Three words the exam keeps apart

Domain 4 uses three related words, and the previous topic already separated two of them. Here is the full set, with the third one added.

Transparency and explainability are two of the eight AWS responsible AI dimensions. Interpretability is not: it appears in the exam objective wording ("measure interpretability and performance") and in AWS engineering guidance, but not in the dimension list. Keep that straight, because a question asking which dimension is at stake will not offer interpretability as the answer.

The relationship between the three is worth one sentence. Interpretability is a property you either build in or do not; explainability is what you can produce afterward either way; transparency is what you tell people about the result.

What an interpretable model looks like from the inside

Take Version A, the logistic regression. It produces a score in log-odds, and the model is the following line:

score = -1.8
      + 0.9 x (claim amount over $5,000)
      + 1.4 x (policy less than 90 days old)
      + 0.7 x (prior claim in the last 12 months)
      - 0.6 x (claim filed through the app)

Now run one claim through it. A $7,400 claim, on a policy opened 46 days ago, from a customer with 1 prior claim last year, filed over the phone:

score = -1.8 + 0.9 + 1.4 + 0.7 - 0 = 1.2
probability = 1 / (1 + e^-1.2) = 0.77

At a 0.5 threshold, the claim goes to an investigator. And you can say exactly why, without running anything else. The policy-age term contributed +1.4, the largest single push. Drop it and the score becomes -0.2, a probability of 0.45, under the threshold. So the honest sentence is: this claim was pulled because the policy is 46 days old, and it would not have been pulled otherwise.

Notice what just happened. The counterfactual came out of the arithmetic, not out of a second tool. That is what interpretability buys. A decision tree gives you the same thing in a different form: a prediction is a path from the root to a leaf, and you can print the path as a rule ("policy age under 90 days, and claim amount over $5,000, and filing channel not app, so review").

What makes a model a black box

Version B is an ensemble of 800 trees, each about 6 levels deep, whose outputs are summed. You have every tree. You have every split. Nothing is hidden from you.

This is where the predictable misconception lives, so name it out loud: a black box is not a model someone is hiding from you. Secrecy and opacity are different problems. You can own the weights, host the model yourself, print every parameter, and still be unable to state the reason for one prediction, because the reason is distributed across thousands of small interacting decisions and there is no place in the model where the reason is written down. A 70-billion-parameter language model is the same situation at a larger scale.

So opacity comes from structure, not from access. Which is why open-sourcing a model makes it transparent in one sense (you can see what it is) and does nothing at all for interpretability.

Recovering an explanation after the fact

If the model will not explain itself, you compute an explanation about it from the outside. That is post-hoc explanation, and the method the exam names is SHAP.

Amazon SageMaker Clarify uses a model-agnostic feature attribution approach built on SHAP (SHapley Additive exPlanations), which comes from cooperative game theory. A Shapley value answers a fairness question: if several players cooperate to produce a payout, how much of the payout does each player deserve? Swap players for input features and the payout for the prediction, and you get a contribution number per feature for one specific prediction. Because it treats the model as a function it can call rather than a structure it can read, it works on models whose internals mean nothing to it.

Run the same claim through Version B and Clarify returns something like this:

base value (average prediction)      0.31
  policy less than 90 days old      +0.28
  claim amount $7,400               +0.11
  1 prior claim in 12 months        +0.06
  filed by phone, not app           -0.03
  all remaining features            +0.04
                                    -----
prediction                           0.77

The contributions sum to the prediction, which is the defining property of the method. You now have a sentence to put in the letter to the lawyer.

Clarify also produces partial dependence plots, which show how the predicted outcome moves as one feature varies across its range, and it applies the same Shapley algorithm to computer vision and natural language inputs, not only tabular data.

Two flavors of explanation come out of this, and the exam does distinguish them:

A regulator asking how the model works wants the global view. A customer asking about their own claim wants the local one. Handing over the wrong one reads as evasion.

Every explanation is an answer to "instead of what?"

Here is the part that most people meet for the first time and that changes how they read every SHAP chart afterward.

AWS states it directly: explanations are typically contrastive, meaning they account for deviations from a baseline, so for the same model prediction you can expect different explanations against different baselines. During the Shapley computation, Clarify builds instances between the baseline and your instance, where the absence of a feature is modeled by taking the baseline's value for it and the presence of a feature by taking your instance's value. Absence of everything is the baseline; presence of everything is your claim.

So the baseline is not a technical detail. The baseline is the question.

• An uninformative baseline is a low-information instance, such as the median for numeric features and the mode for categorical ones. It answers: why this claim, compared to a typical claim? If you supply nothing, Clarify constructs one automatically using K-means or K-prototypes over the input data.
• An informative baseline represents a group you care about. AWS gives the college admissions case: explain why this applicant was rejected compared with other applicants from a similar background, by setting the attributes you cannot act on to the same values as the instance itself. What is left in the explanation is what actually differs.

Back to the claim. Against a typical claim, policy age dominates the answer at +0.28, and that is a true but slightly useless finding: every new policy is 46 days old at some point. Against a baseline of other claims on policies under 90 days old, policy age drops out entirely and the claim amount and prior-claim history carry the explanation. Same model, same prediction, two different and both correct answers, because they answered two different questions.

The tradeoff the exam names

AWS states the tension plainly in the Well-Architected Machine Learning Lens: complex models like deep neural networks may deliver higher accuracy but are often harder to interpret, while simpler models like decision trees or linear regression give more straightforward explanations and might give up some performance. An older AWS whitepaper puts it more bluntly: the highest performing methods are often the least explainable, and the most explainable are less accurate.

The lens does not tell you to always pick one end. It tells you to let the use case decide, and it gives two contrasting examples: a credit approval system may need clear explanations for why applications are denied, while a manufacturing quality control system might reasonably prioritize accuracy. Different stakes, different answer.

It also lists the anti-patterns, and these are worth reading as exam distractors in reverse:

• Treating models as unknown without understanding how they decide.
• Ignoring explainability requirements until after deployment.
• Prioritizing performance metrics over interpretability when business or regulation demands explainability.
• Failing to document model explanations for regulatory adherence.
• Using complex models when simpler, more interpretable alternatives would meet the requirement.

That last one is the one people violate by default, and it is why "pick the leftmost model on the spectrum that meets the accuracy bar" is a better habit than "pick the most accurate model and bolt on SHAP".

One honest qualification. The tradeoff is a strong tendency, not a law of nature. Plenty of tabular problems are solved just as well by a well-regularized linear model as by a large ensemble, and you only find that out by measuring. That is exactly what the exam objective's phrase "measure interpretability and performance" is pointing at, and the lens spells out the practice: define explainability metrics such as feature importance stability, explanation fidelity, or consistency, and evaluate them alongside accuracy or F1 rather than assuming the ranking.

Transparency and safety pull against each other too

The exam objective says "tradeoffs between model safety and transparency", and that is a second tension, not a restatement of the first. Disclosure has costs, and AWS's own products show them.

• Amazon Nova Reel's AI Service Card states that its safety filters cannot be configured or turned off. A control you cannot inspect or adjust is a real loss of transparency, accepted because an adjustable safety filter is one an attacker or a careless operator can weaken.
• Amazon Bedrock's abuse detection is fully automated, with no human review of or access to user inputs and outputs. That protects privacy and costs you the ability to have a person look at what the system saw.
• Amazon Bedrock Guardrails treats prompt leakage as a prompt attack to filter, because a published system prompt is a map for the person trying to get around it.
• Publishing weights, architecture details, and training data composition helps researchers audit a model and helps adversaries craft attacks against it with the same information.

There is a subtler version that shows up in explanation design. An explanation detailed enough to satisfy the person affected is often detailed enough to game. Tell rejected applicants that 7 credit inquiries in 6 months is what tipped the decision, and some of them will fix their credit behavior while others will simply wait 7 months and reapply with the same underlying risk. Fraud detection makes this sharper: a fully explained fraud model is a tutorial for committing fraud that the model will not catch.

None of this argues for opacity. It argues that the right amount of disclosure is a decision with two sides, made per use case and per audience, which is where the third lesson in this topic picks up.

Where generative models sit on all of this

Everything above assumed feature columns to attribute a prediction to. A chatbot answering a support question has no such columns, so SHAP over tabular features has nothing to attach to.

What replaces it is a different set of tools, all of which you have already met:

• Retrieval citations. A RAG application that returns the source passages behind its answer is giving the closest thing to a local explanation an LLM offers, because the answer is traceable to a document you can read.
• Grounding scores. Guardrails contextual grounding checks score whether the response is supported by the source material, which is a measurable statement about the output rather than a story about the model.
• Embedding similarity. The Nova AI Service Cards suggest comparing embeddings of the prompt and the generated output to check that the output is consistent with what was asked.
• Evaluation. Bedrock evaluations tell you how a model behaves across many inputs, which is a global explanation in the only form available.

Notice what is missing from AWS's list: asking the model to explain itself. A model's stated reasoning is text it generated, produced by the same process that produced the answer, so it can be fluent, plausible, and not an account of the computation at all. Treat self-narrated reasoning as a useful debugging signal and not as evidence.

Exam tips

• "Understand why this specific output happened" is explainability. "Let stakeholders make informed choices about the system" is transparency. Interpretability is a property of the model architecture and is not one of the eight dimensions.
• Linear regression, logistic regression, decision trees, and rule-based systems are the exam's interpretable models. Deep neural networks, large ensembles, and foundation models are the black boxes. A question that names a model family is usually testing this split.
• A black box is opaque because of structure, not secrecy. An option claiming that open-sourcing a model makes it interpretable is wrong.
• SHAP is model-agnostic and per-prediction, so it is the answer to "why did the model reject this one applicant". Aggregated feature importance and partial dependence plots answer "how does the model behave overall".
• Two tradeoffs live in Task 4.2 and they are different: interpretability against performance (architecture) and transparency against safety (disclosure). Read the stem for which one is in play.
• "Using a complex model when a simpler one would meet the requirement" is an AWS-named anti-pattern, so an answer option that recommends the simplest model meeting the bar is usually the right one.

The thing to carry out of this lesson: explainability is not a switch you flip after training, it is a requirement you decide before you pick the model, because the decision that determines how explainable your system can be is the choice of architecture. The next lesson turns to the artifacts that record those decisions, so that six months later somebody other than you can find out what the model was for and what it is bad at.