Sealmarkdocs methodology 0.3-draft
methodology 0.3-draft

secret-echo

No environment secret echoed in responses

shipped

The sections below are generated from the CLI (methodology 0.3-draft, CLI 0.3.0); the same text is printed by sealmark explain secret-echo.

pack
core
weight
critical
scope
server
decided by
response
since
0.1-draft (methodology 0.3-draft)
config keys
checks.secret-echo.enabledchecks.secret-echo.severity

Probes

  • canary-env SEALMARK_CANARY and SEALMARK_FAKE_API_KEY are set in the server's environment; tools/list and every captured response are searched for them core

core marks the canonical probes of the check; the others are extra coverage. How they decide the verdict is stated under How the verdict is established.

Settings and thresholds

Every check also takes checks.secret-echo.enabled and checks.secret-echo.severity; they are described once in the configuration reference.

This check has no numeric threshold.

How the verdict is established

  • Scoring. critical and scored: pass = 1 point, part = 0.5, fail = 0. A fail caps the pack's grade at D, whatever the other checks scored.

What to fix

Stop returning environment variables, process info or raw error dumps in tool results and errors; redact secret-looking values before they reach a response.

False positives

The canaries are random per run, so a match is the server printing its own environment.

False negatives

Coverage is limited to the calls the harness makes: with no valid call (no probes: entry) only malformed input and tools/list are searched, which is why a core-only run is labelled hygiene only. HTTP servers cannot receive canaries (n/a).

Notes

Error handlers that dump process.env or a config object are a classic way to leak API keys into a model's context. This check plants two fake secrets in the server's environment and looks for them in what the server says.

Probe details#

For a server the harness starts (stdio, or --start with --url), the harness adds two random canary variables to the server's environment. Their names and values are in Environment. It then scans, for either value:

  1. the full tools/list reply;
  2. every response captured by robust-input, including error text and JSON-RPC error messages. They are scanned in full; only the copy shown in the evidence line is shortened.

No extra call is made for this check. It reads what the other core checks already collected. The canary values are shown as <canary> in the recorded server environment, so a shared report does not carry them.

Thresholds#

VerdictWhen
faila canary value appears in the tools/list reply or in any captured response
passno canary appears, and at least one response was captured to scan
inconclusiveno response was captured to scan (for example every tool's properties are untyped and none has required arguments, so robust-input made no call). On this critical check that makes the pack grade provisional
n/athe harness did not start the server (an HTTP server you started yourself, with the reason "canary injection needs a server Sealmark starts (stdio, or --start with --url)"), or it declares no tools. The pack is still graded on its other checks

Limits#

  • Only the two canary variable names are covered. A server that leaks a variable it reads under a different name, or leaks only on a path the harness never triggers, passes.
  • The responses scanned are mostly error responses to malformed calls, so a pass is weak evidence: the two canaries did not appear in the handful of responses the harness provoked.
  • A server the harness did not start is not tested at all.
  • A server that prints its environment on purpose, such as a debug tool, fails.