How EBPM.jp Researches and Verifies Policy Evidence

EBPM.jp does not evaluate policy claims simply by asking whether they have a citation.

We ask a different set of questions:

Where did the claim come from? What evidence supports it? What inferential steps connect the evidence to the policy claim? What was lost, omitted, or changed along the way?

Wherever possible, we trace claims back to primary sources and examine research design, study populations, causal identification constraints, external validity, uncertainty, funding, and conflicts of interest.

AI assists with source discovery and citation tracing. Substantive judgments remain human-reviewed.

Our objective is not to automatically determine the “correct” policy. It is to make the evidentiary chain transparent enough for others to inspect.

1. Tracing Policy Claims Back to Primary Sources

A policy document may cite research without fully preserving the conditions or limitations of that research.

Where feasible, EBPM.jp traces claims through the following chain:

Policy Claim → Government or Institutional Report → Review or Secondary Source → Original Research → Primary Data or Study Population

At each stage, we examine what information was preserved, omitted, or reinterpreted.

Relevant factors may include research design, study population and comparator, effect size, uncertainty, causal inference limitations, external validity, funding sources, conflicts of interest, and differences between the wording of the original research and the eventual policy claim.

2. What Is the Evidence Source Graph?

EBPM.jp is developing an Evidence Source Graph to make relationships between policy claims and their underlying evidence easier to trace.

It is intended to go beyond a conventional bibliography.

For a given policy claim, the graph is designed to help users inspect: Which report was cited? Which research did that report rely on? What did the original study actually find? Did the study population match the population targeted by the policy? Were important qualifications lost during citation or summarization?

The Evidence Source Graph is not an automated policy-ranking system.

It is a tool for making evidence provenance and transformation visible to human reviewers.

3. How We Use AI

Generative AI can dramatically reduce the time required to discover documents, identify candidate citations, and organize relationships between large numbers of sources.

It can also produce fabricated references, overstate findings, or repeat interpretations from secondary materials as though they were conclusions of the original research.

EBPM.jp therefore uses AI primarily to support source discovery, citation extraction, document organization, and comparative research.

Human review remains central to assessing the meaning of original research, the relationship between evidence and claims, causal validity, and policy interpretation.

The goal is not to delegate judgment to AI, but to use AI to increase human verifiability.

4. Separating Evidence From Policy Judgment

Evidence does not automatically determine a policy decision.

Research can inform us about observed effects, uncertainty, conditions, and plausible causal relationships.

Policy decisions also involve cost, feasibility, fairness, rights, ethics, institutional constraints, and political priorities.

EBPM.jp therefore tries to distinguish: What can reasonably be established from evidence? Where does policy or value judgment begin?

5. External Validity

Evidence that an intervention worked in one setting does not establish that it will produce the same result elsewhere.

Countries, institutions, populations, implementation capacity, culture, timing, and social conditions can differ substantially.

When examining international policy examples, EBPM.jp asks: Why did the intervention work in that setting? Do the relevant conditions also exist elsewhere? Which components may be transferable, and which may not be?

International comparison is therefore used not only to identify successful practices, but also to identify the conditions under which they succeed.

6. Staged, Modular Publication

EBPM.jp uses a staged, modular research and publication model.

Case studies, the Evidence Source Graph, methodological review, and bilingual dissemination are developed and released incrementally rather than depending on a single final deliverable.

A completed case study can be published and reviewed immediately.

A partial Evidence Source Graph can already be tested and improved.

Methodological feedback can be incorporated before the entire program is complete.

This structure allows useful, verifiable public knowledge to accumulate throughout the project rather than appearing only at the end.

It also reduces dependence on the success of any single final output.

7. Methodological Review and Corrections

EBPM.jp does not treat publication as the end of verification.

Findings may be updated when new research, data, institutional changes, or expert feedback become available.

Where appropriate, important cases may receive methodological review from researchers or subject-matter experts.

Material errors or omissions are corrected, with correction or update histories retained where appropriate.

The ability to correct a claim is treated as part of verifiability, not as a weakness.

8. Funding and Conflicts of Interest

Funding does not automatically determine whether evidence is valid or invalid.

However, funding relationships and conflicts of interest are relevant information for readers evaluating research, analysis, or policy advocacy.

When EBPM.jp receives organizational funding, we intend to disclose the funder, amount, period, and purpose where appropriate.

Funders, collaborators, and related organizations are not automatically excluded from scrutiny.

Editorial and methodological decisions — including topic selection, source selection, research methods, analysis, conclusions, publication, and corrections — should remain independent of funding relationships.

9. Bilingual Knowledge Exchange

EBPM.jp’s international work is not intended to move knowledge in only one direction.

We make international experience in policy evaluation and evidence use accessible to Japanese audiences while also making Japanese institutional conditions, administrative practice, implementation constraints, and lessons more accessible in English.

The objective is not: International knowledge → Japan but: Japan ⇄ International knowledge through reciprocal scrutiny and learning.

10. What We Are Building

EBPM.jp is not intended to be a website that tells readers which policy is “correct.”

We are working toward a public knowledge infrastructure in which people can trace where a policy claim came from, identify what the evidence can and cannot establish, and see where empirical evidence ends and judgment begins.

Better policy decisions require not only more information, but structures that make information verifiable.

FAQ

Does EBPM.jp decide whether a policy is right or wrong?

No. Our primary objective is to make the evidence and inferential pathway behind policy claims inspectable rather than to assign automatic policy verdicts.

What is the Evidence Source Graph?

It is a developing structure that links policy claims to institutional reports, secondary sources, original research, and underlying data, while helping users inspect how meaning or qualifications may change along the citation chain.

Does EBPM.jp use AI?

Yes. AI may assist with source discovery, citation tracing, and document organization. Substantive judgments about research findings, causal validity, and policy meaning remain subject to human review.

Why trace claims back to primary sources?

Because important qualifications concerning populations, uncertainty, study design, and limitations may be omitted in secondary sources or policy documents.

What is external validity?

External validity concerns whether findings from one study population or setting can reasonably be applied to another. It is particularly important when comparing policies across countries or institutions.

Why publish outputs incrementally?

Incremental publication allows completed components to be used, reviewed, corrected, and improved before the entire program is finished. The project therefore does not depend on a single final deliverable.

Can funders influence EBPM.jp’s conclusions?

Our intended governance model separates funding from editorial and methodological decision-making. Funders should not determine research methods, analysis, conclusions, publication, or corrections.

What happens when EBPM.jp finds an error?

Material errors are corrected. Where appropriate, correction and update histories are retained so that changes remain visible and reviewable.