Sealmarkdocs methodology 0.3-draft
methodology 0.3-draft

protocol

Valid MCP handshake and tool schemas

shipped

The sections below are generated from the CLI (methodology 0.3-draft, CLI 0.3.0); the same text is printed by sealmark explain protocol.

pack
core
weight
scored
scope
server
decided by
protocol
since
0.1-draft (methodology 0.3-draft)
config keys
checks.protocol.enabledchecks.protocol.severity

Probes

  • tool-name tool names are 1-128 characters of A-Z a-z 0-9 _ . - core
  • unique-names no two tools share a name core
  • description every tool has a description of at least 10 characters
  • stdout-noise no line on the server's stdout that is not a JSON-RPC message (a banner, a log line, a stray print), counted over initialize, tools/list and every call the core session makes; part, protocol.stdout-noise core
  • invalid-result every tools/call result the core session receives is valid MCP (the client's schema accepts it); part, protocol.invalid-result core

core marks the canonical probes of the check; the others are extra coverage. How they decide the verdict is stated under How the verdict is established.

Settings and thresholds

Every check also takes checks.protocol.enabled and checks.protocol.severity; they are described once in the configuration reference.

This check has no numeric threshold.

How the verdict is established

  • Scoring. scored: pass = 1 point, part = 0.5, fail = 0. A fail lowers the pack's percentage but never caps the letter.

What to fix

Give every tool a unique name made of letters, digits, underscore, hyphen or dot, a description that says what it does, and an input schema that is valid JSON Schema. Write only JSON-RPC messages to stdout (send logs, banners and debug output to stderr: a strict stdio client breaks on any other line), and return tool results that are valid MCP.

False positives

A short description is a style finding and only earns part; a tool name outside the allowed characters is a protocol failure in strict clients. A server that prints a banner to stdout works with a lenient client and breaks a strict one; the check says part, not fail.

False negatives

The tools/list schema is reviewed, plus what the core session saw on the wire while it ran: stdout lines that are not JSON-RPC and replies the client rejected as not valid MCP. Only the core session is measured; other packs' sessions are not. Tool results are otherwise checked by robust-input. An HTTP server has no stdout, so noise is never reported for it.

Notes

The first thing the harness does with any server is the handshake and the tool listing. This check says whether the server answered both in a form a client can use. It is hygiene, not security, and the only core check that still produces a verdict when a server declares no tools.

Probe details#

No tool is called. The harness sends initialize, and records the server's name and version for the evidence line. It then sends tools/list and follows nextCursor to the end. A repeated cursor, more than 50 pages, more than 5 000 tools or more than 32 MiB of listing in total is a protocol failure. Every tool in the listing is then reviewed statically (the rules are in the Probes list above).

If the MCP client library rejects the tools/list reply (for example an inputSchema that is not a JSON Schema object), there is nothing left to probe. protocol fails with the library's message, and the other three core checks are inconclusive with the reason "tools/list was rejected by the MCP client, so nothing could be probed".

Thresholds#

VerdictWhen
faila tool name outside the pattern, a duplicate tool name, a tools/list reply the client library rejects, or a listing past the bounds above
partialno failure, but a tool has no description or one shorter than 10 characters, or the server declares no tools
passotherwise
  • The name pattern is this harness's rule, not a quotation from the MCP specification. It is conservative on purpose: many clients and model APIs enforce something close to it.
  • A duplicate name makes calls ambiguous, so it is a fail.
  • A missing description does not break a client. It removes what a model uses to choose between tools, so it costs half a point.
  • A server with no tools has broken nothing, but must not look like a perfect score for doing nothing. It gets part ("no tools declared") and the other core checks are n/a, so the pack is graded on protocol alone: half a point of one is 50%, a D.

Limits#

  • A pass does not show that any tool works or that a description is accurate. Ten characters is a floor, not a quality bar.
  • Only initialize and tools/list are exercised. Resources, prompts, sampling and progress notifications are not touched.
  • The harness uses the official TypeScript MCP SDK, so a reply that other clients accept but the SDK rejects reads as a fail.
  • A server can list different tools to different callers.