Sealmark
Automated re-test on 2026-11-20: not checked by hand, and not yet reviewed by the maintainers.

Results / openags/paper-search-mcp

openags/paper-search-mcp

version tested 0.1.4GitHubtested 2026-11-20Sealmark 1.0.1methodology 0.3-draftSTANDARD run71 tools

Sealmark's automated test of openags/paper-search-mcp 0.1.4 on 2026-11-20 gave it grade A, its lowest test-area grade (protocol A and effects A). Only 8 of 71 tools could be called with default settings, so the deeper test areas cover only those tools. Of its 6 checks, 5 passed and 1 not applicable. 3 more checks are advisory, and advisory checks never change a grade: 3 not applicable. The run used Sealmark 1.0.1 with the standard settings.

A

Grade A

The server's grade is its lowest test-area grade. How grading works

Only 8 of 71 tools could be called with default settings, so the deeper test areas cover only those tools.

Grade by test area

A
Protocol4/4 points
A
Effects1/1 points

Protocol checks

CheckVerdictWhat happened
protocol
Valid MCP handshake and tool schemas
pass
handshake ok, 71 well-formed tool(s)
Evidence for protocol: 1 line
  1. Probe initialize + tools/list
    Outcome server paper_search_server 1.30.0 answered; 71 tool(s)
    Decided from the MCP exchange (protocol)
tool-poisoningcritical
No tool-poisoning indicators in tool metadata
pass
scanned 71 tool definition(s) and serverInfo, including parameter text: nothing suspicious
robust-input
Survives malformed tool input
pass
115 invalid call(s) rejected cleanly
Evidence for robust-input: 1 line
  1. Probe 115 invalid call(s) across 23 tool(s)
    Outcome all rejected cleanly and the server stayed responsive
    Decided from the MCP exchange (protocol)
secret-echocritical
No environment secret echoed in responses
pass
canary secrets absent from tools/list and 116 captured response(s), 1 of them successful result(s)
instructions-hygiene (advisory)
Server instructions name only tools that exist
n/a
the server sent no instructions
refusal-quality (advisory)
Refusals say what to do next
n/a
no refusal was recorded in this run, so there is nothing to judge

Effects checks

CheckVerdictWhat happened
annotation-honestycritical
Read-only tools change nothing
pass
get_search_cache_status, read_pubmed_paper, read_crossref_paper, read_openalex_paper declared read-only and changed nothing in the observed directories during a benign call (extract_sections, read_dblp_paper, read_openaire_paper, read_base_paper not judged: the control call failed)
openworld-honesty
Closed-world tools stay on the machine
n/a
no tool declares openWorldHint: false that could be given a harness-owned control argument
undeclared-writes (advisory)
Unannotated tools leave the sandbox unchanged
n/a
every tool carries annotations, nothing to check

Each check id links to what it tests and how to fix a failure. The same text is in sealmark explain <id>.

Under a check, Evidence shows what it probed (sent or inspected) and what it observed in this run: up to 6 lines per check, each cut at 240 characters, for checks that are not security checks. The text is shortened, and paths and anything that looks like a secret or token are replaced. Run sealmark run --format json yourself for every line in full.

By tool

Which checks called each tool with a valid call in this run, and which did not exercise it. A check can leave a tool out for several reasons: a limit Sealmark sets, a tool left out by design, input the run could not fill in, or a call that returned an error. The CLI report gives the reason. A verdict belongs to the whole check, not to one tool: a failed check that called three tools does not say which one failed. Security checks are left out of this table.

ToolCalled byNot exercised by
get_search_cache_statusNone
clear_search_cacheNo check listed here called it with a valid call.
search_papersNo check listed here called it with a valid call.None
search_wosNo check listed here called it with a valid call.None
search_scopusNo check listed here called it with a valid call.None
read_scopus_paperNo check listed here called it with a valid call.None
search_arxivNo check listed here called it with a valid call.None
search_pubmedNo check listed here called it with a valid call.None
search_biorxivNo check listed here called it with a valid call.None
search_medrxivNo check listed here called it with a valid call.None
search_google_scholarNo check listed here called it with a valid call.None
search_iacrNo check listed here called it with a valid call.None
download_arxivNo check listed here called it with a valid call.None
download_pubmedNo check listed here called it with a valid call.None
download_biorxivNo check listed here called it with a valid call.None
download_medrxivNo check listed here called it with a valid call.None
download_iacrNo check listed here called it with a valid call.None
read_arxiv_paperNo check listed here called it with a valid call.None
read_pubmed_paperNo check listed here called it with a valid call.None
read_biorxiv_paperNo check listed here called it with a valid call.None
read_medrxiv_paperNo check listed here called it with a valid call.None
read_iacr_paperNo check listed here called it with a valid call.None
search_semanticNo check listed here called it with a valid call.None
download_semanticNo check listed here called it with a valid call.None
read_semantic_paperNo check listed here called it with a valid call.None
search_crossrefNo check listed here called it with a valid call.
get_crossref_paper_by_doiNo check listed here called it with a valid call.
download_crossrefNo check listed here called it with a valid call.
download_scihubNo check listed here called it with a valid call.
download_with_fallbackNo check listed here called it with a valid call.
read_crossref_paperNo check listed here called it with a valid call.
search_openalexNo check listed here called it with a valid call.
extract_sectionsNo check listed here called it with a valid call.
get_citing_papersNo check listed here called it with a valid call.
get_referenced_papersNo check listed here called it with a valid call.
search_pmcNo check listed here called it with a valid call.
search_coreNo check listed here called it with a valid call.
search_europepmcNo check listed here called it with a valid call.
search_dblpNo check listed here called it with a valid call.
search_openaireNo check listed here called it with a valid call.
search_citeseerxNo check listed here called it with a valid call.
search_doajNo check listed here called it with a valid call.
search_baseNo check listed here called it with a valid call.
search_zenodoNo check listed here called it with a valid call.
search_halNo check listed here called it with a valid call.
search_openreviewNo check listed here called it with a valid call.
download_openreviewNo check listed here called it with a valid call.
read_openreview_paperNo check listed here called it with a valid call.
search_ssrnNo check listed here called it with a valid call.
search_unpaywallNo check listed here called it with a valid call.
read_dblp_paperNo check listed here called it with a valid call.
download_dblpNo check listed here called it with a valid call.
read_openaire_paperNo check listed here called it with a valid call.
download_openaireNo check listed here called it with a valid call.
read_citeseerx_paperNo check listed here called it with a valid call.
download_citeseerxNo check listed here called it with a valid call.
read_doaj_paperNo check listed here called it with a valid call.
download_doajNo check listed here called it with a valid call.
read_base_paperNo check listed here called it with a valid call.
download_baseNo check listed here called it with a valid call.
read_zenodo_paperNo check listed here called it with a valid call.
download_zenodoNo check listed here called it with a valid call.
read_hal_paperNo check listed here called it with a valid call.
download_halNo check listed here called it with a valid call.
read_ssrn_paperNo check listed here called it with a valid call.
download_ssrnNo check listed here called it with a valid call.
read_openalex_paperNo check listed here called it with a valid call.
download_openalexNo check listed here called it with a valid call.
search_acmNo check listed here called it with a valid call.
download_acmNo check listed here called it with a valid call.
read_acm_paperNo check listed here called it with a valid call.

What was not covered

Grade over time

Since the run on

Same as that run: the same grade in every test area, the same server version (0.1.4), the same Sealmark version (1.0.1).

  1. Av0.1.4Sealmark 1.0.0
  2. Av0.1.4Sealmark 1.0.1
  3. Av0.1.4Sealmark 1.0.1

Badge for the README

The badge shows this automated result for version 0.1.4. It is a link to this page, not a certificate, and it changes when the next re-test does.

Sealmark badge for openags/paper-search-mcp
The flat badge, one letter per graded test area.
[![Sealmark](https://sealmark.dev/badge/openags-paper-search-mcp.svg)](https://sealmark.dev/results/openags-paper-search-mcp/)

Files: badge SVG, card SVG, shields endpoint JSON.

Project facts

License
MIT
Repository
openags/paper-search-mcp
Last push
2 Oct 2026
Open issues and pull requests
20

From GitHub, checked 22 Nov 2026. Plain facts about the project; they play no part in the grade.

Run it yourself

This repeats the run with the same Sealmark version and the standard settings, starting the server the same way:

npx -y sealmark@1.0.1 run --start "uvx --from git+https://github.com/openags/paper-search-mcp paper-search-mcp"

To confirm your run used the standard settings, add --format json --out report.json to the command. In the report, config.standard should be true and config.hash should start with sha256:d13e7893a0ac, as they did in this run. If either differs, so did the settings, and the verdicts may too (STANDARD and CUSTOM runs).

To keep the setup, put the start command in sealmark.yaml as the server.

Grade alerts

One email when the grade of openags/paper-search-mcp changes after a re-test. Nothing else, and every email has a link to stop them.

Or follow it in a feed reader: the Atom feed lists each move of the overall grade that came with a different version of this server, tested with the same Sealmark version as the graded run before.

Each of 613 verdicts was checked independently, and Sealmark was wrong on some: how reliable Sealmark is.

Is this wrong?

Reply on the GitHub issue or email disclosure@sealmark.dev and we will recheck it. Maintainers can ask for their server to be removed.

If we were wrong, we correct the page and add your reply. Security findings go to the maintainer privately first; see the disclosure policy.