Sealmarkdocs methodology 0.3-draft
methodology 0.3-draft

Observers

Each piece of evidence says how it was established: db-state, fs-observer, listener, process, http, upstream, log, response, protocol or static.

shipped

A server can say "ok" and still have written to the database. Where the harness can look for itself, it does, and the server's own answer is not trusted. Every piece of evidence in a report carries a verifiedBy tag that says which way it was established.

server connects as

{dsn} resolves to the role under test, so the database's policy and grants apply.

A
db pack · grade A7 of 7 scored checks pass · 100%
sm_analyst · check pii
analystpii · critical
probeSELECT email FROM customers
responsepermission denied for table customersa masked view would return c***@example.test: also clean
pass · piiverified: response
sm_support · check rls
supportrls · critical
probeSELECT * FROM orders
responseT1-NOTE-order-1 …only the support role's own tenant came back
pass · rlsverified: response
sm_admin · check read-only
adminread-only · not critical
probeINSERT INTO probe_t (id, v) VALUES (9001, 'sm-insert')
response, then database stateisError: only read statementsstate reader: probe_t has 0 rows
pass · read-onlyverified: db-state

mutating-sql, pii-masked, db-rowcap and db-timeout also ran; all four pass in either case. The admin lane does not change because this example server still refuses writes when pinned to the admin DSN. A server that connects as the table owner is reported inconclusive instead, because the admin control read fails.

A pass means the server did not defeat the database's policy or grant; it does not mean the server masks anything. Invented example, not a real server.
  1. The harness starts the server once per database role: analyst, support and admin.
  2. Analyst runs SELECT email FROM customers. Support runs SELECT * FROM orders. Admin runs an INSERT into probe_t.
  3. With the role connection string, the analyst gets permission denied, support sees only its own tenant, and the INSERT leaves probe_t with 0 rows. All three pass.
  4. With the admin connection string used for every role, the analyst receives the raw address cust1001@example.test and support receives T2-SECRET rows. Both are verified from the response and both are critical fails.
  5. Two critical fails cap the db pack at D: 5 of 7 points, a raw C.

The ways a verdict is decided#

verifiedByHow the verdict was decided
db-stateThe harness read the database over its own separate superuser connection and saw a row, table or value change, or not. The server's answer is not trusted.
responseFrom what the server returned: text, error flag, row count, elapsed time.
protocolFrom MCP protocol behavior: the handshake, tools/list, whether the server is still alive after a call.
staticFrom tool metadata, without calling any tool.
fs-observerThe harness compared the files in its sandbox before and after a probe.
listenerThe harness's loopback listener saw, or did not see, the request, with the probe's token.
processThe harness read the process table: is a program it started still running?
httpFrom the HTTP exchange with a server reached by URL: status, headers, TLS.
upstreamThe harness's upstream stub, which the server uses as its backend through {upstream}, received, or did not receive, a request: which statement, which identifier, whose credential. It answers with scripted replies and never contacts anything else (guide).
logWhat a server Sealmark started, or a stdio server, printed (the last 1 MiB).

Each check page says which one decides it. rls and pii use response because reading data changes nothing in the database, so a leak shows only in what the server returns.