A public problem is not a research question. Research design narrows a contested problem into claims that evidence or interpretation can bear on while preserving the route back to the political question.
Part IV · Chapter 7
From Public Problem to Research Design
A public problem is not a research question. Research design narrows a contested problem into claims that evidence or interpretation can bear on while preserving the route back to the political question.
Learning objectives
What this chapter asks you to be able to do.
- Decompose a broad public problem into researchable claims without treating the decomposition as politically neutral.
- Distinguish descriptive, distributional, causal, predictive, interpretive, normative, and institutional or authority claims and identify what kind of reasoning each requires.
- Explain the difference between a concept and an indicator, and evaluate construct and measurement validity before interpreting precision.
- Distinguish unit of analysis, unit of observation, target population, sampling frame, observed sample, case boundary, and scope conditions.
- Analyze why evidence exists, who is absent from it, and how missingness or selection can distort a policy conclusion before any statistical model is fitted.
- State the comparison and inferential logic required by a causal claim without confusing association with causal effect.
- Design complementary multi-method research in which each component has a defined inferential job rather than serving as a ritual badge of methodological pluralism.
- Distinguish planned, exploratory, adaptive, and post hoc research choices and record redesign honestly.
- Integrate ethics, privacy, feasibility, and political risk into research design before collection begins.
- Evaluate AI-assisted research design as a source of useful analytical support and possible procedural power rather than as an authority that can choose the scientifically correct method.
- Produce Decision Dossier VII: a Research Design and Evidence Plan that remains auditable before results are known.
Chapter summary
The argument in inspectable form.
Descriptive, distributional, causal, predictive, interpretive, normative, and institutional/authority claims require different forms of support and must not borrow authority from one another.
A concept is not an indicator. Operationalization is necessary for empirical inquiry, but measurement validity depends on whether the indicator represents the concept relevant to the claim.
Precision is not validity. More observations can narrow uncertainty around the wrong quantity without repairing conceptual or population mismatch.
The unit of analysis, unit of observation, target population, sampling frame, observed sample, and analyzed cases can differ; those differences must be explicit.
Absence from data is not absence from the polity. Evidence is generated by social and institutional processes that can make policy exiters, nonparticipants, nonusers, or marginalized groups systematically hard to observe.
Causal language requires explicit comparison and counterfactual logic. Statistical sophistication cannot substitute for a design that distinguishes the preferred explanation from plausible alternatives.
Multi-method research is strongest when components perform complementary inferential work. Method variety alone does not create triangulation or independence.
Precommitment can improve transparency, but preregistration is not a universal hierarchy. Planned, exploratory, adaptive, and post hoc choices should be labeled honestly.
Research ethics, privacy, vulnerability, consent, and feasibility are part of the research design because they determine which evidence may legitimately be created.
AI can help generate, challenge, document, and audit research designs while also exercising methodological influence through rankings, templates, data access, cost, and standardization.
Current NousPolis architecture makes many research choices explicit and auditable, but provenance and schema completeness do not prove conceptual validity, methodological pluralism, institutional independence, or legitimacy.
Decision Dossier VII records the complete Research Design and Evidence Plan, including what would falsify, reinterpret, redesign, or reopen the inquiry and what the research can never authorize on its own.
Key terms
Concepts to carry forward.
- research design
- The explicit architecture connecting a research claim to concepts, cases or populations, observations, evidence, comparison, analysis, ethics, and rules for inference and revision.
- concept
- An abstract idea used to identify and organize a phenomenon, such as trust, representation, access, polarization, or institutional responsiveness.
- indicator
- An observable measure, classification, behavior, record, or judgment used as evidence about a concept; no indicator is automatically identical to the concept.
- operationalization
- The process of specifying how an abstract concept will be observed, measured, classified, or otherwise made empirically tractable.
- construct validity
- The degree to which an operational measure or set of observations adequately represents the theoretical concept relevant to the claim.
- unit of analysis
- The kind of entity about which the research seeks to make an inference, such as individuals, neighborhoods, institutions, elections, states, or policy episodes.
- unit of observation
- The entity or record from which evidence is actually collected; it may differ from the unit of analysis.
- target population
- The population or case universe to which the research claim is intended to apply.
- sampling frame
- The operational list, register, platform, archive, or other accessible set from which cases or observations can actually be selected.
- data-generating process
- The social, institutional, technical, or historical processes that cause an observation or record to exist in the form available to the researcher.
- missingness
- The condition in which relevant observations are absent; the reason for absence can be substantively informative rather than random noise.
- scope condition
- A stated boundary identifying the contexts, populations, periods, institutions, or mechanisms within which an inference is expected to hold.
- identification
- The reasoning that connects observed variation to a causal quantity or explanation by ruling out, controlling, or otherwise addressing competing explanations.
- precommitment
- A research decision recorded before outcome direction is known, such as a primary measure, inclusion rule, comparison, or analysis plan.
- triangulation
- The use of distinct evidence or methods to assess a claim from different inferential angles; agreement is informative only when the components are genuinely relevant and not simply correlated repetitions.
- redesign trigger
- A specified finding, ethical problem, feasibility failure, assumption change, or validity concern that requires the research design to be revised or reopened.
Review & discussion
Questions for seminar, revision, or assessment.
- Why can a public problem be politically important before it has been turned into a defensible research question?
- Take the claim “The Asterbridge reform reduced public trust.” Identify one descriptive, causal, interpretive, normative, and authority question hidden inside it.
- Why is trust in service reliability not automatically the same concept as trust in procedural fairness? Could both be valid dimensions of a broader concept?
- Construct an example of an extremely precise estimate with poor construct validity.
- Explain the difference between unit of analysis and unit of observation using an example that does not involve transport.
- How can the sampling frame exercise definitional power before a sample is selected?
- Give three reasons complaint data might decline after a policy change. Which additional evidence could distinguish them?
- Why can a very large dataset worsen political confidence in a misleading result rather than fix selection bias?
- What must be stated before a researcher can use the phrase “the reform caused” rather than “the reform was followed by”?
- When does adding a second method strengthen a design, and when does it merely decorate it?
- Why might three studies not count as three independent confirmations even if all are peer reviewed?
- Distinguish an exploratory discovery from undisclosed outcome switching. What should happen when an exploratory result is politically important?
- Under what conditions can redesign after seeing data be a sign of rigor rather than misconduct?
- How can an ethics or privacy restriction change the epistemic strength of a study? How should that limitation appear in the conclusion?
- What kinds of methodological bias could arise if an AI design system rewards speed, sample size, standardization, and reproducibility?
- Why does a complete provenance record not guarantee a valid research design?
- How could a challenger demonstrate that a NousPolis decomposition is conceptually wrong even if every later evidence artifact is accurate?
- Which Decision Dossier VII field would be easiest to fill mechanically and which requires the most substantive judgment? Defend your answer.
- Write one redesign trigger for a concept-validity failure, one for a population failure, and one for an ethics failure in the Asterbridge trust study.
- Complete the sentence: “Even perfect evidence about the effects of the reform cannot decide...” Give one normative and one authority answer.
Further reading
Continue into the literature.
For concept formation, begin with Sartori (1970) and Collier and Mahon (1993), then read Adcock and Collier (2001) on measurement validity as a problem shared across qualitative and quantitative research. Groves et al. (2009) provides a foundational account of target populations, sampling frames, coverage, and survey error. Geddes (1990) is a classic demonstration of how case selection can shape comparative conclusions, while Meng (2018) shows why sheer data quantity does not automatically overcome selection. For inference and research design, compare King, Keohane, and Verba (1994) with Brady and Collier (2010), Schwartz-Shea and Yanow (2012), and Goertz and Mahoney (2012). Lieberman (2005) offers one influential model of purposeful mixed-method integration. On preregistration and research transparency, read Nosek et al. (2018), Monogan (2015), Haven and Van Grootel (2019), and Kapiszewski and Karcher (2021) as a debate rather than a single rule. The Belmont Report (National Commission 1979) remains an important historical framework for thinking about respect, beneficence, justice, consent, risk, and subject selection. For AI-assisted social-science research, Ziems et al. (2024) surveys opportunities across computational social-science tasks, while Bisbee et al. (2024) provides a political-science warning against treating plausible synthetic survey responses as reliable replacements for human data.
Companion, not replacement
The textbook carries the complete argument.
This page reproduces the chapter's study and navigation layer from the current living manuscript. The Asterbridge narrative, historical and methodological argument, figures, Political Science Lens boxes, NousPolis Canon crosswalks, labs, red-team exercises, and N5 Practice boundaries remain in the canonical textbook rather than being republished wholesale here.