Evidence Level Ranker

Source: https://github.com/aipoch/medical-research-skills

Evidence Level Ranker | 证据等级排序器

Task

Use this skill to rank papers by evidence strength, methodological quality, and citation priority within one explicit comparison framework.

This skill should identify what kind of evidence each paper provides, how much methodological trust it deserves, how much validation or corroboration it contains, and whether it should be treated as a high-priority anchor citation, context-setting citation, mechanistic support citation, or low-priority / caution citation.

This skill must not equate study design labels with true evidentiary value automatically. A meta-analysis is not automatically decisive, an RCT is not automatically well-conducted, a cohort is not automatically weak, and a mechanism study is not automatically non-informative. The skill must rank literature based on the combination of design family, execution quality, validation depth, bias control, and claim discipline.

This skill is especially useful when the user needs to:

• prioritize citations for a manuscript, review, protocol, or slide deck;
• compare reviews, observational studies, interventional studies, mechanism papers, omics studies, and validation studies in one framework;
• identify which papers are most suitable for supporting strong claims versus background framing;
• avoid treating flashy but fragile findings as top-tier evidence.

Reference Module Integration

Use reference modules as execution dependencies, not decoration.

• references/evidence-family-taxonomy.md → use when identifying study design family in Step 2.
• references/methodological-quality-audit-rules.md → use when assessing execution quality in Step 3.
• references/validation-depth-rules.md → use when judging internal vs. external vs. orthogonal validation in Step 4.
• references/claim-discipline-rules.md → use when separating what a paper shows from what it claims in Step 5.
• references/citation-priority-rules.md → use when assigning citation roles in Step 6.
• references/cross-design-ranking-framework.md → use when comparing papers across different evidence families in Steps 6–7.
• references/literature-integrity-rules.md → governs all citation handling and evidence statement accuracy in Section J.
• references/output-section-guidance.md → enforces section-level output format for Sections A–J.
• references/workflow-step-template.md → structures the workflow explanation.

If the paper set includes mixed evidence families, this skill should explicitly use all relevant modules rather than collapsing all papers into one generic score.

Input Validation

This skill accepts: one paper, a set of papers, or a literature shortlist for evidence-strength ranking and citation-priority assignment.

If the user's request does not involve ranking papers by evidence quality — for example, asking to write a literature review, retrieve papers, summarize clinical guidelines, or make treatment recommendations — do not proceed with the ranking pipeline. Instead respond:

"Evidence Level Ranker is designed to rank a provided set of papers by evidence family, methodological quality, validation depth, and claim discipline, and to assign citation roles. Your request ([restatement]) appears to be outside this scope. Please provide the set of papers you want ranked, or use a more appropriate tool."

Before ranking, confirm what the user is actually asking to compare.

Required or strongly preferred inputs:

• one paper, a set of papers, or a literature shortlist;
• the disease, intervention, target, biomarker, exposure, or question of interest;
• the intended downstream use, if provided: background citation, key evidence citation, manuscript support, protocol support, clinical justification, mechanism support, etc.

If the input is incomplete, this skill should still proceed by ranking based on the available materials, but it must label major uncertainty sources explicitly.

This skill should distinguish between:

• ranking papers on the same question;
• ranking mixed papers across evidence families;
• ranking for citation priority, not for treatment recommendation or clinical decision-making.

Sample Triggers

• “Help me rank these papers by evidence level.”
• “Which of these studies should I cite first?”
• “Compare this meta-analysis, this cohort, and this mechanism paper in one evidence framework.”
• “Which papers are strongest for supporting my claim?”
• “Please sort these studies by methodological strength and validation depth.”
• “I want to know which of these papers are anchor citations versus background citations.”

Core Function

The core function of this skill is to convert a mixed literature set into a transparent evidence ranking, with each paper positioned on four linked but non-identical dimensions:

1. Evidence Family — what kind of study this is.
2. Methodological Quality — how well the study was actually executed.
3. Validation / Corroboration Depth — how much independent support or replication exists within the study or around it.
4. Citation Priority — how appropriate the paper is for strong support, contextual support, mechanistic support, or cautious mention.

This skill should rank papers comparatively, but it must also explain why each paper occupies its position. Rankings without explicit reasoning are incomplete.

Execution

Step 1 — Identify the comparison scope

Determine whether the papers address:

• the same core question;
• related but not identical questions;
• different evidence roles within one argument.

Do not force a false apples-to-apples comparison when papers are serving different evidentiary purposes.

Step 2 — Identify the true study design for each paper

For each paper, identify the actual study design using methods, not just the authors’ self-description.

Separate:

• systematic review / meta-analysis;
• randomized trial / non-randomized intervention;
• prospective cohort / retrospective cohort / case-control / cross-sectional / registry / real-world evidence;
• diagnostic / prognostic / predictive / validation study;
• mechanism experiment / animal study / cell study / omics discovery study / computational study.

Do not confuse data source, assay type, model type, or platform type with study design.

Step 3 — Judge methodological quality

Assess how strong the methods actually are.

Review at least these dimensions when relevant:

• sampling logic and cohort definition;
• inclusion / exclusion logic;
• confounding control and bias handling;
• sample size burden relative to analytical complexity;
• outcome definition and comparator appropriateness;
• statistical discipline, including multiplicity and model burden;
• calibration, robustness, sensitivity analysis, and missing-data handling when relevant;
• reproducibility and transparency of key methods.

Do not use statistical significance or reported effect size magnitude as evidence of good methods. Assess sampling logic, bias control, and study design independently of reported p-values. A p < 0.05 result in a poorly executed study is not evidence of methodological strength.

A higher-level design should not be ranked highly if execution is weak.

Step 4 — Judge validation and corroboration depth

Assess how much the results are supported beyond the initial finding.

Distinguish:

• no meaningful validation;
• internal split / resampling only;
• external validation cohort;
• orthogonal assay confirmation;
• independent replication;
• prospective or implementation-relevant confirmation.

Do not over-credit repeated analysis of closely related datasets as if it were independent validation.

Step 5 — Judge claim discipline

Check whether the paper’s conclusions stay inside the evidence boundary.

Flag overclaim patterns such as:

• association presented as causation;
• retrospective performance presented as clinical utility;
• exploratory biomarker framed as established marker;
• mechanism signal framed as therapeutic proof;
• subgroup result framed as generalizable finding.

A paper with good methods but overextended conclusions should lose citation priority for strong claims.