Sealmarkdocs methodology 0.3-draft
methodology 0.3-draft

Quickstart

Point Sealmark at an MCP server, run it, and read the grade. About five minutes.

shipped
One start for core, one per role for db (plus a fresh start for the timeout probe). Fixed probes, no model.
  1. Read sealmark.yaml: the command or URL that starts your server, and which packs to run.
  2. Start your server once for the core pack, and read its tool list.
  3. For the db pack, a seeded synthetic Postgres is started first. Then your server is started once per database role (analyst, support and admin), plus a fresh start for the timeout probe.
  4. Call the server's tools directly with fixed inputs. No language model is involved.
  5. Decide each verdict from what the harness sees: for database checks it reads the database itself, over a separate connection; otherwise it uses what the server returned.
  6. Apply the published formula to get one letter per pack. A failed critical check caps the grade at D.
  7. Write the result with the exact configuration and the evidence for every probe to sealmark.json. With --fail-under B the command exits 1 when a graded pack is below B.

1. Create a config. init finds the MCP servers your clients already use, asks which to grade, and writes sealmark.yaml (with a git-ignored sealmark.local.yaml for secrets, only when needed). To start from one client, a command line or a URL, see Import servers.

bash
npx sealmark init
npx sealmark doctor    # can Sealmark start your server? which packs would run?

2. Point it at your server. Edit the server block. Put {sandbox} where the server takes a directory it may use, and {dsn} where it takes a database connection string (the db pack fills it in once per database role).

yaml
name: my-server
version: 1.0.0
server:
  command: node
  args: ["./dist/server.js", "{sandbox}"]
  env: { API_KEY: "${MY_API_KEY}" }

${MY_API_KEY} is read from your environment when the run starts and is never written to a report. Never put a real credential or a real database URL in this file; secrets can live in sealmark.local.yaml.

3. Run it.

bash
npx sealmark run

The report goes to stdout and progress to stderr. Sealmark runs every pack that has something to test; core sends only schema-invalid arguments, so an A from core alone says nothing about what your tools do. A short, invented excerpt for Server A:

text
PACK db  GRADE D  (5/5 scored, 80%)
  [FAIL] Personal data masking respected — customers.email returned unmasked to the analyst role.
  [PASS] Row-level security respected — no other-tenant rows returned through any tool.
  Capped at D: critical fail(s): Personal data masking respected. Without the cap this would be B (80%).

4. Read the result. Exit 0 passes and exit 1 fails the gate (the default is A); see the CLI reference. When a check fails, open the evidence under the failing line. Each piece says how it was established: see Observers.