robust-input

Survives malformed tool input

shipped

The sections below are generated from the CLI (methodology 0.3-draft, CLI 0.3.0); the same text is printed by sealmark explain robust-input.

pack: core
weight: scored
scope: server
decided by: protocolresponse
since: 0.1-draft (methodology 0.3-draft)
config keys: checks.robust-input.enabledchecks.robust-input.hang_mschecks.robust-input.max_toolschecks.robust-input.severitytimeout_probe_sec

Probes

missing-required {} when the tool has required arguments core
wrong-types every typed argument gets a value of the wrong JSON type core
nulls every typed argument is null core
extra-argument wrong types plus an unexpected extra argument (sm_unexpected) core
string-as-object every string argument gets an object (a second wrong type: the first candidate is a number) core
nested-string a string in place of a required array-of-object parameter (one extra call per tool that has one) core
nested-item a required array-of-object parameter gets one item that lacks a required field, when the items declare one (one extra call per tool that has one) core
read-only-no-args {} for a tool with no arguments that declares readOnlyHint: true (a valid call, counted for coverage)
robust.lenient-downstream an error reply that reports a failed fetch (the failed-fetch vocabulary of the net pack: connection refused, DNS, timeout, TLS ...) and does not name a parameter: the server used the invalid argument before it failed; part core
robust.internal-error an error reply that is an unhandled runtime error (TypeError, Cannot read properties, is not a function, undefined is not, AttributeError, KeyError, NoneType): an internals leak, not a validation decision; part core

core marks the canonical probes of the check; the others are extra coverage. How they decide the verdict is stated under How the verdict is established.

Settings and thresholds

Every check also takes checks.robust-input.enabled and checks.robust-input.severity; they are described once in the configuration reference.

Threshold	Default	Set with	Standard
`max_tools`	`25`	`checks.robust-input.max_tools`	locked

Key	Default	Standard	Meaning
`checks.robust-input.hang_ms`	none	locked	Milliseconds a call may hang on bad input before robust-input fails. null derives max(2000, timeout_probe_sec * 1000).
`checks.robust-input.max_tools`	`25`	locked	Most tools robust-input will probe.
`timeout_probe_sec`	`10`	locked	Seconds the cap and timeout probes wait before a call counts as running too long (flag --timeout-probe).

How the verdict is established

Scoring. scored: pass = 1 point, part = 0.5, fail = 0. A fail lowers the pack's percentage but never caps the letter.

What to fix

Validate arguments against the declared input schema before running the tool, and answer a schema-invalid call with an MCP error (isError: true) that names the argument, not a plain-text success. An error that comes after the invalid argument was used (a failed upstream request built from it) is leniency, and an unhandled runtime error (TypeError, Cannot read properties, KeyError) is an internals leak, not a rejection; for an array of objects, check the array itself and each item's required fields.

False positives

Only schema-invalid input is sent, so a tool that rejects it cannot have run with unintended input. A plain-text refusal without isError counts as a rejection, with evidence that clients cannot tell it from success. An error is a clean rejection only when it is a validation decision: a reply that names a declared parameter and says it is invalid or required is one, a failed upstream request (robust.lenient-downstream) or an unhandled runtime error (robust.internal-error) is not.

False negatives

A tool that takes no arguments and is not declared read-only is never called (it is listed as needing a probes: entry). Valid calls are only made through probes:. A server that validates the top level and passes nested entities on unchecked is caught only through the two nested variants, and only for required array-of-object parameters; object parameters and optional ones are not descended.

Notes

This check calls tools with arguments that break the tool's own schema and watches whether the server rejects them cleanly. A server that dies or hangs on one bad call is a denial of service for every agent that shares it.

Probe details#

The four variants in the Probes list are sent to up to the first 25 tools in the listing (checks.robust-input.max_tools). Each variant is skipped when it does not apply to the tool, and a property with no declared type is not probed, because any value might be valid.

For the wrong-type call, the harness takes for each property the first of these values that violates the declared type and is not in the property's enum:

text

12345
"not-a-valid-value"
true
["x"]
{"x": 1}

The one valid call. A tool with no properties at all that declares readOnlyHint: true gets a single {} call, because that call is valid by the schema and cannot change anything. It counts toward coverage only when the server answers it without an error: a read-only tool that rejects its own no-argument call is listed as not covered, and the coverage level stays at hygiene only. Any other tool with no argument the harness can safely malform needs a probes: entry. If no tool has arguments the harness can malform and no valid call was made, no call is made and the check is n/a.

Each call has a hang limit of max(2 s, --timeout-probe), 10 seconds by default. After the last variant of a tool, or after a protocol error, the harness sends tools/list to check the server is still alive. After three hangs it stops probing.

How a rejection is read#

A rejection can be an MCP error (isError: true or a JSON-RPC error), or a refusal in the text of a normal reply ("invalid input: ..."). A refusal in text without isError counts as a rejection, and the evidence says so: a client that reads isError cannot tell it from a success. A schema-invalid call that succeeds with no refusal is lenient.

Every response is also scanned for a stack trace or internal path (the same scan runs on the replies to your own probes: calls):

a Python Traceback (most recent call last), File "...", line N or .py", line;
a JavaScript frame of the form at fn (file:line:col), including file://, drive-letter, absolute-path and node: frames;
node_modules/ or site-packages/ followed by a source file and a line number;
a C:\Users\..., C:\Windows\... or C:\Program Files\... path;
a Unix home-directory path (/home/, /Users/, /root/) that carries a :line suffix.

A bare Unix home-directory path does not count, so an ENOENT message or a PATH dump is not flagged. The captured responses are also kept for secret-echo to scan.

Thresholds#

Verdict	When
fail	the server process exited or stopped answering, or any call got no answer within the hang limit
partial	a response contains a stack trace or internal path, or a schema-invalid call succeeded without any refusal ("accepted schema-invalid arguments")
pass	every invalid call was rejected cleanly and the server stayed responsive
n/a	no tool has arguments the harness can safely malform, and no valid call was made

A crash or a hang takes the server away from every agent, so it fails. Leaking internals and accepting invalid input are hygiene problems, not outages, so they cost half a point.

Limits#

At most four invalid shapes are tried per tool, and only for the first 25 tools. Nothing is said about fuzzing, oversized input, concurrency, untyped arguments or valid-looking hostile input.
Many JavaScript servers coerce 12345 to "12345" for a string parameter. That is lenient, common and harmless, and still reads as part.
A slow server can exceed the hang limit under load, and the C:\Users\... pattern also matches an ordinary error message that mentions a Windows user directory.