Designing a Program Evaluation That Stands Up to Scrutiny
A program has been running for a year. Leadership wants to know whether it worked, a funder wants a report, and someone has been asked to evaluate it, often without a dedicated research budget, a comparison group, or much lead time. This is the position many faculty researchers and organizational staff find themselves in, and it creates a familiar trap: the pressure to produce a clean answer ("the program works") outpaces what the available data and design can actually support. When that gap surfaces later, in a funder review or a board presentation, it damages the program's credibility more than an honest, appropriately scoped evaluation ever would have.
Start With the Question the Evaluation Is Actually Answering
Not every program evaluation is asking the same thing, and confusing the types leads directly to overclaiming. An implementation evaluation asks whether the program was delivered as intended — attendance, fidelity to the model, dosage, reach. An outcome evaluation asks whether the people or systems the program touched changed in some measurable way. An impact evaluation asks whether the program caused that change, as opposed to something else explaining it. These are progressively harder to answer and require progressively more rigorous designs. Most organizational evaluations, realistically, are implementation or outcome evaluations, not impact evaluations. That is perfectly legitimate, as long as the language used to report results matches the design. The credibility problem starts when an outcome evaluation gets written up using causal language it hasn't earned. Naming the evaluation type honestly at the outset, and holding to that framing through reporting, is the single most protective decision you can make.
Match Your Design to What You Can Actually Claim
Once the evaluation question is clear, the design has to be built around what it can genuinely support. A single pre/post measurement on program participants, with no comparison group, can tell you that scores changed — it cannot tell you the program caused that change, since maturation, seasonality, or outside events could explain the same pattern. If a causal claim is genuinely needed, the design needs some basis for comparison: a matched comparison group, a waitlist control, a staggered rollout that creates natural comparison points, or, where feasible, randomization. When a comparison group isn't available, that's not a reason to abandon rigor; however, it is a reason to be explicit about what the design can and cannot rule out. A well-designed pre/post study with a clearly stated limitation is far more defensible than a study that implies causality it can't demonstrate. Reviewers, funders, and methodologically literate stakeholders are far more forgiving of acknowledged limitations than of overreach they catch themselves.
Build Measures That Reflect the Program's Actual Theory of Change
A surprising number of evaluations fail not because of design flaws but because the measures don't actually reflect what the program was trying to do. This usually happens when convenience drives measurement choices. This might be using whatever data already exists, or an easy-to-administer satisfaction survey, rather than starting from the program's theory of change and asking what evidence would actually indicate success. Before selecting measures, map the program's logic: what inputs and activities are supposed to produce what short-term outcomes, and how those short-term outcomes are supposed to lead to the longer-term result the program ultimately cares about. Then choose measures tied to each step in that chain, not just the final outcome. This does two things. First, it lets you say something useful even if the long-term outcome hasn't materialized yet (e.g., perhaps the near-term steps show the program is on track). And, second, it protects the evaluation from being judged solely on a single distal outcome that may be influenced by many factors outside the program's control.
Anticipate the Questions Stakeholders and Funders Will Ask
A methodologically sound evaluation can still land poorly if the reporting doesn't anticipate how it will be read. Funders and boards are rarely trained methodologists, but they are often more sophisticated than evaluators assume, and they will ask the obvious questions: How do you know this wasn't just a good year? What would have happened without the program? Why this measure and not another one? Building answers to these questions into the report itself — rather than waiting to be asked — signals competence and heads off the moment where an unprepared answer undermines an otherwise solid evaluation (this works the same with dissertation committees and journal review boards!). This also means writing results in language appropriate to the audience. A methods section written for an academic journal will lose a program officer; a report that hides every limitation in a footnote will lose a methodologically careful reviewer.
The strongest evaluations state findings plainly, name limitations directly, and connect both back to concrete, actionable next steps for the program. A program evaluation earns trust the same way any research earns trust: by claiming exactly what the design supports, no more and no less. If you're currently scoping an evaluation, the most useful next step is to write down, in one sentence, the specific question you're trying to answer and then check whether your planned design could actually answer it. That single exercise catches most overclaiming before it becomes a problem.
Work With Matt
Program evaluations sit at the intersection of applied research and organizational accountability, and getting the design right the first time saves significant rework later. Matt works with faculty, researchers, and organizations to design evaluations with methodology, measures, and reporting that hold up to funder and stakeholder scrutiny. Learn more about Matt's consulting approach or schedule a consultation.
