refusal-quality
Refusals say what to do next
The sections below are generated from the CLI (methodology 0.3-draft, CLI 0.3.0); the same text is printed by sealmark explain refusal-quality.
- pack
core- weight
- advisory · not scoredcandidate
- scope
- server (how it words a refusal)
- decided by
- response
- since
- 0.3-draft (methodology 0.3-draft)
- config keys
checks.refusal-quality.enabledchecks.refusal-quality.patternschecks.refusal-quality.severity
Probes
routingevery refusal the other checks recorded (codes scope.refused, sql-guard.refused, net.ssrf-refused*, fs.*refused*, and probe replies that are errors) is read; one routes when it matches a pattern: instead, use (the )?[a-z_]+ tool, available, allowed, try, list_[a-z]+, contact, will not (help|succeed), permanent, not retry. pass at 90 % or more, part at 50 %, fail below, n/a with no refusal
Settings and thresholds
Every check also takes checks.refusal-quality.enabled and checks.refusal-quality.severity; they are described once in the configuration reference.
This check has no numeric threshold.
| Key | Default | Standard | Meaning |
|---|---|---|---|
checks.refusal-quality.patterns | ["instead","use (the )?[a-z_]+ tool","available","allowed","try","list_[a-z]+","contact","will not (help|succeed)","permanent","not retry"] | not locked | Regular expressions that mark a refusal as routing the caller (naming what is available, what to use instead, that retrying will not help). Replaces the default list. |
How the verdict is established
- Scoring. advisory: reported with its verdict but not part of the score.
What to fix
Make every refusal say what to do next: name the tool or the identifiers that are allowed (for example call list_tables), say whether retrying can help, and avoid a bare forbidden. This check is advisory and never changes a letter; checks.refusal-quality.patterns sets the words that count as routing.
False positives
A server that refuses tersely on purpose (to give an attacker nothing) reads as a dead end; the check is advisory and never moves a letter.
False negatives
Only refusals the run recorded with the server's words are read; a refusal worded with none of the patterns reads as a dead end whatever it says (checks.refusal-quality.patterns replaces the list).