Articles about designing, interpreting, and using evidence to assess capabilities with sound judgment.
33 articles
Missing evidence, conflicting evidence, and uncertainty are not the same problem. When they are collapsed into a generic category, an organization may manufacture certainty, defer a decision indefinitely, or treat something it has not yet observed as a gap.
An assessment that returns only a score provides less information than it used to reach its conclusion. The person needs to understand the result, the evidence behind it, its limitations, and what they can do with it.
An average can accurately represent the center of several results while concealing the difference that matters most to the decision. Its usefulness depends on which components it combines and which of them may compensate for one another.
A scale reduces differences in interpretation when it requires assessors to recognize the same evidence—not when it replaces vague words with more elegant labels. Designing one means defining what will be observed, making the boundaries between levels explicit, and testing those boundaries with cases that reveal where disagreement begins.
A skill is a useful construct for organizing capabilities, but an assessment cannot access it directly. It accesses decisions, explanations, procedures, outcomes, and behaviors produced under specific conditions.
A skill is a useful construct for organizing capabilities, but it never appears as a complete object to the person assessing it. What we see are partial manifestations in specific tasks, decisions, and contexts.
Choosing who should assess a skill is not a search for the most reliable source. It is an exercise in identifying who had sufficient access to the behaviors, decisions, and outcomes relevant to the conclusion we need to draw.
A skills assessment is sound when it can support both the interpretation of its results and the decisions made from them. Five conditions must work as a chain: object, evidence, levels, sources, and rules.
Two people can observe the same behavior and reach different conclusions. The usual response is to ask which one of them got it wrong.