Publication Trust Assessment

Evaluate the scientific integrity and reliability of publications before building analyses on their findings.

When to Use

• User wants to evaluate the reliability and reproducibility of a genomics publication
• User asks about "publication quality", "reproducibility", "trust assessment", or "paper evaluation"
• User needs to check if a paper follows ENCODE data standards and best practices
• User wants to verify that cited datasets, tools, and methods meet community standards
• Example queries: "is this paper's ChIP-seq analysis trustworthy?", "evaluate the methods in this genomics paper", "check if this study follows ENCODE standards"

Why This Matters

Not all published findings are reliable. Problems range from formal retractions (data fabrication, image manipulation) to informal contradictions where independent groups fail to reproduce key claims. In genomics and computational biology, building pipelines or analyses on unreliable findings wastes resources and propagates errors.

The problem is not always obvious. Formal retractions are rare compared to the number of problematic papers. More commonly:

• A subsequent study contradicts the key finding without triggering a retraction
• An erratum quietly corrects a critical result
• The original authors publish a "correction" that substantially changes conclusions
• Independent groups fail to replicate and publish their negative results

This skill provides a systematic approach to detecting these issues.

Trust Assessment Framework

Trust Levels

Level	Label	Meaning
5	High confidence	No issues found; replicated by independent groups
4	Standard	No issues found; not yet independently replicated
3	Caution advised	Minor corrections, errata, or partial contradictions exist
2	Reliability concerns	Key findings contradicted by independent study, or expression of concern issued
1	Compromised	Retracted, or key findings refuted with evidence of methodological problems

Default trust level is 4 (standard) — absence of evidence is not evidence of absence. A paper starts at "standard" and moves up with independent replication or down with identified issues.

Step 1: Gather Publication Metadata

For any publication being assessed, retrieve full metadata:

get_article_metadata(pmids=["PMID"])

Record:

• article_types: Check for "Retracted Publication", "Published Erratum", "Expression of Concern"
• authors: Note corresponding author and senior author (last position)
• journal: Impact factor context (high-profile journals ≠ automatic reliability)
• publication_date: Older papers have had more time for replication/refutation

If starting from a DOI:

convert_article_ids(ids=["DOI"], id_type="doi")

Step 2: Check Formal Integrity Markers

2a. Retraction Status

Search PubMed for retraction notices linked to this paper:

search_articles(query="PMID[PMID] AND (Retracted Publication[pt] OR Retraction of Publication[pt])")

Also check the article_types field from Step 1 — if it contains "Retracted Publication", the paper has been formally retracted.

If retracted → Trust Level 1 (Compromised)

2b. Errata and Corrections

Search for corrections:

search_articles(query="PMID[PMID] AND (Published Erratum[pt] OR Correction[pt])")

Errata can be minor (typo in a table) or major (recalculated results that change conclusions). Read the erratum to distinguish:

• Minor: Trust level unchanged
• Major (changes key results or conclusions): Trust Level 3 (Caution advised)

2c. Expression of Concern

search_articles(query="PMID[PMID] AND Expression of Concern[pt]")

An Expression of Concern from a journal editor indicates an active investigation. Trust Level 2 (Reliability concerns) until resolved.

Step 3: Check for Contradicting Publications

This is the most important and most nuanced step. Many problematic findings are never formally retracted — they are contradicted by subsequent independent work.

3a. Find Citing Articles

find_related_articles(pmids=["PMID"], link_type="pubmed_pubmed", max_results=50)

This returns computationally similar articles. From these, search for contradiction signals.

3b. Search for Contradiction Signals

Search PubMed for articles that cite the original AND contain contradiction language:

search_articles(query="\"[key claim from title]\" AND (\"fail to replicate\" OR \"unable to reproduce\" OR \"does not\" OR \"do not support\" OR \"contradicts\" OR \"challenges\" OR \"reanalysis\" OR \"re-analysis\" OR \"not reproducible\" OR \"could not confirm\")")

Also search with the first author's last name + key topic terms:

search_articles(query="[FirstAuthor] [KeyTopic] AND (Comment[pt] OR Letter[pt] OR \"fail\" OR \"does not\")")

3c. Search Academic Databases

Use Consensus to find contradicting evidence:

consensus_search(query="does [key claim from paper] replicate? contradiction evidence")

3d. Check bioRxiv for Preprints

Contradictions sometimes appear first as preprints:

search_preprints(category="[relevant category]", date_from="[pub date]", date_to="[today]")

3e. Evaluate Contradictions

Not every disagreement is a refutation. Assess:

Factor	Strengthens contradiction	Weakens contradiction
Independent lab	Yes — different group, different reagents	Same group correcting themselves (may be honest science)
Sample size	Larger sample in contradicting study	Smaller sample or different model system
Methodology	Direct replication attempt	Different methodology that may explain discrepancy
Specificity	Contradicts the specific key claim	Disagrees on secondary finding
Mechanism	Provides alternative explanation with evidence	Simply fails to replicate without explanation

If key finding is contradicted by independent group with stronger methodology → Trust Level 2

Step 4: Author and Group Assessment

4a. Check Author Track Record

If a contradiction or retraction is found, check whether the authors have other problematic publications:

search_articles(query="[SeniorAuthor][Author] AND Retracted Publication[pt]")

A pattern of retractions from the same group is a stronger signal than a single incident.

4b. Contextual Notes

When flagging author concerns, use measured language:

• "Publications from this group have been subject to independent contradiction"
• "A previous study from this corresponding author was contradicted by [citation]"
• "This group's findings in [area] have not been independently replicated"

Never use: fabrication, fraud, doctored, liar, shady, dishonest — unless a formal investigation has published findings using those terms. Stick to what the published record shows.

Step 5: Generate Trust Report

Present findings in a structured format:

## Publication Trust Assessment

**Paper**: [Title] ([Journal], [Year])
**PMID**: [PMID] | **DOI**: [DOI]
**Authors**: [First Author] ... [Senior Author]

### Trust Level: [X/5] — [Label]

### Formal Markers
- Retraction: None / Yes (date, reason)
- Corrections: None / [count] ([minor/major])
- Expression of Concern: None / Yes (date)

### Contradicting Evidence
- [Citation of contradicting paper] — [brief description of contradiction]
- Independent replication: Yes/No/Unknown

### Author Context
- [Any relevant notes about author track record]

### Recommendation
- [Clear guidance on whether to rely on this paper's findings]

Step 6: Integration with Other Skills

When this skill identifies a trust issue, the finding should propagate:

• cite-encode: Flag compromised papers in citation lists
• quality-assessment: Downweight methods from compromised sources
• variant-annotation: Note if variant-gene associations rely on compromised evidence
• disease-research: Flag if disease mechanisms depend on contradicted findings
• regulatory-elements: Note if regulatory element classifications cite problematic work
• data-provenance: Record trust assessment in provenance chain

Examples

Example 1: Informal Contradiction (Islet Biology)

Original: Li et al. 2016, Cell — "Artemisinins Target GABA Receptor Signaling and Impair Alpha Cell Identity" (Kubicek lab)

• Claimed artemisinins convert alpha cells to beta cells via GABA receptor signaling
• High-profile publication in Cell

Contradicting: van der Meulen et al. 2017, Cell Metabolism — "Artemether Does Not Turn Alpha Cells into Beta Cells" (Huising lab)

• Direct replication attempt by independent group
• Found no evidence of alpha-to-beta transdifferentiation
• Showed artemether actually suppresses Ins2 expression 100-fold and impairs beta cell function
• Concluded artemisinins cause general dedifferentiation, not directed transdifferentiation

Trust Level: 2 (Reliability concerns) — Key finding directly contradicted by independent group using primary cells (vs. cell lines in original).

Author context: When encountering other publications from this group, note that a previous high-profile finding was independently contradicted.

Example 2: Formal Retraction

Any paper with article_types containing "Retracted Publication":

• Trust Level: 1 (Compromised) — No further analysis needed
• Note retraction reason if available
• Check for other retractions from same group

Example 3: Statistical Reanalysis Leading to Retraction

Original: Potti et al. 2006, Nature Medicine — Genomic signatures predicting individual patient chemotherapy response. Used microarray data to build classifiers for drug sensitivity.

Contradicting: Baggerly & Coombes 2009, Annals of Applied Statistics — Independent reanalysis found simple off-by-one indexing errors, mislabeled cell lines, and reversed sensitive/resistant labels. The genomic signatures were artifacts of data handling mistakes, not biology.

Outcome: Clinical trials based on the signatures were halted. The paper was retracted in 2011. Senior author had multiple subsequent retractions.

Trust Level: 1 (Compromised) — Statistical reanalysis using the original data conclusively demonstrated methodological errors. This case illustrates why reanalysis (using the same data to reach different conclusions) is the strongest form of contradiction — it eliminates methodology differences as an explanation.

User Override

If the user reviews a flagged paper and determines the contradiction doesn't apply (e.g., different model system, different question), they can override:

• "Actually, that paper doesn't refute this finding because [reason]"
• Update trust level accordingly and note the user's reasoning in the assessment

Trust assessments are living documents — they should be updated as new evidence emerges.

Walkthrough: Verifying Literature Claims Before ENCODE Analysis

Goal: Validate key literature claims that inform your ENCODE analysis pipeline — ensuring you're building on solid scientific ground before investing analysis effort. Context: Genomics analysis choices (peak callers, normalization methods, quality thresholds) are often justified by citing papers. This skill verifies those citations are accurate and current.

Step 1: Identify claims to verify

Before running a ChIP-seq analysis, you might rely on these claims:

1. "MACS2 is the gold standard peak caller for ChIP-seq" — is this still true?
2. "FRiP >= 1% indicates acceptable ChIP-seq quality" — where did this threshold come from?
3. "IDR < 0.05 ensures reproducible peaks" — what's the original source?

Step 2: Verify claim 1 — MACS2 as gold standard

Search PubMed for the original MACS2 publication and recent benchmarks:

search_articles(query="MACS2 peak calling benchmark ChIP-seq", max_results=5, sort="relevance")

Key findings:

• Zhang et al. 2008 (PMID: 18798982) — original MACS paper, 12,000+ citations
• Recent benchmarks (2023-2024) confirm MACS2 remains competitive but note SEACR performs better for CUT&RUN data
• Verdict: Claim is VALID for ChIP-seq but NOT for CUT&RUN/CUT&Tag

Step 3: Verify claim 2 — FRiP threshold

search_articles(query="Landt 2012 ChIP-seq quality guidelines ENCODE", max_results=3)

Key findings:

• Landt et al. 2012 (PMID: 22955991) — ENCODE ChIP-seq guidelines
• FRiP >= 1% is indeed the ENCODE standard
• Verdict: Claim is VALID. Source: Landt et al. 2012, Genome Research

Step 4: Track verified experiments with literature provenance

encode_track_experiment(accession="ENCSR000AKA", notes="H3K27ac ChIP-seq - QC thresholds verified per Landt 2012 (PMID:22955991)")

Expected output (the notes you pass are stored, not echoed back; read them with encode_list_tracked):

{
  "tracking": {"accession": "ENCSR000AKA", "action": "tracked"},
  "auto_linked_references": [{"type": "geo_accession", "id": "GSE123456"}],
  "publications_found": 1,
  "publications": [
    {"pmid": "22955991", "doi": "10.1101/gr.136184.111", "title": "ChIP-seq guidelines and practices of the ENCODE and modENCODE consortia", "authors": "Landt SG, Marinov GK, Kundaje A", "journal": "Genome Research", "year": "2012", "abstract": ""}
  ],
  "pipelines_found": 0,
  "pipelines": []
}

Step 5: Document verification results

Record which claims were verified, sources found, and any caveats:

• Verified: MACS2 for ChIP-seq, FRiP >= 1%, IDR < 0.05
• Caveats: MACS2 not recommended for CUT&RUN (use SEACR instead)
• This verification log feeds into → data-provenance and → scientific-writing

Integration with downstream skills

• Verified QC thresholds inform → quality-assessment criteria
• Literature validation supports → scientific-writing methods sections
• Citation accuracy feeds into → cite-encode for proper attribution
• Verified pipeline choices guide → pipeline-guide recommendations

Code Examples

1. Verify an experiment's publication and quality

encode_get_experiment(accession="ENCSR123ABC")

Expected output (fields abridged):

{
  "accession": "ENCSR123ABC",
  "assay_title": "Histone ChIP-seq",
  "target": "H3K27ac-human",
  "status": "released",
  "audit_error_count": 0,
  "audit_not_compliant_count": 0,
  "audit_warning_count": 1,
  "audit_internal_action_count": 0,
  "bio_replicate_count": 2,
  "lab": "Bernstein, Broad",
  "date_released": "2020-03-15"
}

Trust check: audit_error_count 0, bio_replicate_count 2, released status — meets ENCODE standards.

2. Check citations for a tracked experiment

encode_get_citations(accession="ENCSR123ABC")

Expected output (export_format="bibtex" / "ris" return plain text instead of JSON):

{
  "publications": [
    {
      "id": 1,
      "experiment_accession": "ENCSR123ABC",
      "pmid": "32728249",
      "doi": "10.1038/s41586-020-2493-4",
      "title": "Expanded encyclopaedias of DNA elements in the human and mouse genomes",
      "authors": "ENCODE Project Consortium",
      "journal": "Nature",
      "year": "2020",
      "abstract": ""
    }
  ],
  "count": 1
}

3. Compare two experiments for consistency

Both experiments must already be tracked; otherwise the tool returns {"error": "Experiment ... not tracked. Track it first."}.

encode_compare_experiments(
  accession1="ENCSR123ABC",
  accession2="ENCSR456DEF"
)

Expected output:

{
  "experiment_1": {"accession": "ENCSR123ABC", "assay": "Histone ChIP-seq", "biosample": "GM12878"},
  "experiment_2": {"accession": "ENCSR456DEF", "assay": "Histone ChIP-seq", "biosample": "K562"},
  "verdict": "COMPATIBLE_WITH_CAVEATS",
  "recommendation": "These experiments can be compared, but the warnings should be addressed in your analysis.",
  "compatible_aspects": [
    "Same organism: Homo sapiens",
    "Same assembly: GRCh38",
    "Same assay: Histone ChIP-seq",
    "Same target: H3K27ac-human"
  ],
  "issues": [],
  "warnings": [
    "Different labs: Bernstein, Broad vs Snyder, Stanford. Batch effects possible."
  ]
}

Pitfalls & Edge Cases

• High citation count ≠ high quality: Highly cited papers can contain errors that propagate through the field. Always evaluate methods independently of citation count or journal prestige.
• "Standard pipeline" is vague: Papers claiming to use "standard" or "established" pipelines without naming specific tools and versions cannot be reproduced. Require exact tool + version + parameters.
• Missing replicate information: Papers that do not report the number of biological replicates, or confuse biological with technical replicates, cannot support differential claims. Minimum: 2 biological replicates.
• Genome build ambiguity: Papers that report coordinates without specifying the genome assembly (hg19 vs hg38) are a major reproducibility hazard. Coordinates can shift by megabases between builds.
• Incomplete data availability: Papers that claim "data available upon request" instead of depositing in public repositories (GEO, ENCODE) effectively prevent reproduction. FAIR principles require public deposition.
• Overinterpretation of motif analysis: Finding a motif in peaks does not prove direct binding. Background motif frequency, peak-to-motif distance, and orthogonal ChIP validation are all required.

Integration

This skill produces...	Feed into...	Purpose
Verified QC thresholds	quality-assessment	Use validated thresholds for ENCODE data QC
Citation verification reports	cite-encode	Ensure cited methods papers are accurate
Validated pipeline recommendations	pipeline-guide	Select pipelines backed by verified benchmarks
Literature provenance records	data-provenance	Document which papers justified analysis choices
Verified methods claims	scientific-writing	Write methods sections with verified citations
Updated tool recommendations	bioinformatics-installer	Install tools backed by current benchmarks
Verified regulatory annotations	regulatory-elements	Confirm annotation sources are authoritative
Literature search results	cross-reference	Link ENCODE experiments to verified publications

Related Skills

• cite-encode — Citation management with trust integration
• quality-assessment — ENCODE experiment quality (complementary to publication quality)
• data-provenance — Track trust assessments in provenance chains
• disease-research — Disease research workflows that depend on publication reliability

Presenting Results

• Present literature assessment as: claim | supporting_paper | citation_count | journal | year. Flag papers with <50 citations as lower confidence. Suggest: "Would you like to find the full text via PubMed?"

For the request: "$ARGUMENTS"