Critical thinking: from a claim to a researchable question

Make the claim more specific first

“This method is very successful” sounds open to discussion, but it is not measurable. Which method, for whom, for which outcome and over what period? Rewrite the statement using these four questions. Make clear whether your disagreement concerns the meaning of a word, the quality of the data or the interpretation drawn from a result.

Four stages of review

1. Claim: Write one testable statement. Pay particular attention to expressions such as “always”.
2. Source: Follow the news report to the research, and the research to the data it uses. A widely shared post is not independent confirmation.
3. Method: Read how people were selected, how the comparison was made and what was measured.
4. Limit: Write down the question these data do not answer. Acknowledging uncertainty does not make a claim worthless; it corrects its scope.

Apply the same standard to yourself

Do not set a lower threshold of evidence for a result you like and a higher one for a result you dislike. Put observations that support and weaken the claim into two separate columns. Deciding what would change your mind before you begin is a useful working discipline.

Preregistration aims to distinguish analyses planned in advance from analyses developed later. Registration does not automatically establish that the results are correct. It makes the relationship between the plan and what was done open to inspection.

A five-minute exercise

Choose a news item you encountered today. Write its actual claim rather than its headline. Find the link to the first source. Then answer: “What other explanation could fit this result?” On the last line, state the limits of the available evidence rather than a definitive verdict.

Write two alternative explanations for an increase

Test the claim “a reminder message increased attendance” with a small example. Every number below is invented; this is not an ELYADAL study or a measured effect. The aim is to distinguish an observation from a causal claim.

1. Write the comparison first: how were the messaged and unmessaged groups formed, and do they concern the same event and period? Specify the outcome and denominators.
2. Create at least two competing explanations. First, selection: people already interested may have joined the message list. Second, timing: the messaged event may have taken place at a more convenient hour.
3. For each explanation, name the information that could distinguish it. If that evidence is absent, defer the causal conclusion. In a future study, settle comparison and analysis choices before seeing the results.

Worked record

Invented observation	12 of 20 messaged people attended, versus 8 of 20 others: 60% and 40%, a 20-percentage-point difference.
Selection question	Did groups already differ in interest, registration route or previous attendance? The list-entry record is missing.
Timing question	Were events at the same date/time and under comparable conditions? Schedule evidence is missing.
Supported conclusion	Attendance proportions differed in these two groups. The message has not been shown to cause that difference.
Working record	Claim __; observation/denominator __; alternative 1 and needed evidence __; alternative 2 and needed evidence __; deferred conclusion __.

Preregistration supports transparency of a plan; it does not automatically repair a poor comparison. The exercise claims no random assignment, statistical significance or generalisability.

Primary sources: Center for Open Science: preregistration