Mike Frank
@mcxfrank
Cognitive scientist at Stanford. Open science advocate. Symbolic Systems Program director. Bluegrass picker, slow runner, dad.
If you’re at #cogsci2026, please come see presentations by some of the great folks collaborating with the Language and Cognition Lab at Stanford!
When you can, titrate your manipulation continuously rather than just treatment vs control. A dose-response design lets you estimate the shape of the relationship, not just whether an effect exists. Way more theoretically informative.
There are good reasons to go between (carryover effects, sensitization, practice effects), but they're often overstated. Across student replications, within-participants designs were the strongest predictor of replication success in our meta-analysis.
The big design decision we discuss is whether each participant experiences one level of a factor (between) or multiple levels (within). Within-participants designs are typically much more efficient, because they cancel out person-level variation that would otherwise be noise.
One piece of advice: estimate test-retest reliability whenever you can. Give the instrument twice, correlate. Test-retest is more reflective of real-world reliability than internal-consistency measures (e.g., Cronbach's alpha, which is widely reported and widely misinterpreted).
The bullseye visualization helps. High reliability + high validity = tight cluster on the bullseye. High reliability + low validity = tight cluster, just in the wrong place.
A common confusion: just because you can put something on the right side of a regression equation doesn't mean it's a causal control. Whether to "control for" a variable depends on the causal structure of your problem. Inappropriate adjustment can bias your estimate, not fix it.
Most of the statistical tests you learned in stats class — t-test, ANOVA, correlation — are actually special cases of a single thing: linear regression. Ch 7 of Experimentology argues that thinking in models, not tests, is more flexible and a better foundation for theory. 🧵 experimentology.io
The classic logic of NHST: simulate the null distribution, see where your observed effect lands, declare significance if it falls in the tails. Useful in narrow situations, but it forecloses on more interesting questions about how big and how reliable your effect really is.
If p < 0.05, you "reject the null." That's what many stats courses teach you to do. This week's Experimentology chapter argues that this binary, dichotomous frame can sometimes be useful but is the wrong default for psych experiments — there is a broader toolkit available! 🧵 experimentology.io
The New Children’s Museum in San Diego is truly special. I’ve seen, loved, worked in, and brought my kids to a lot of children’s museums over the years and this one is awesome.
How do we compare effects across studies that use different rating scales? One common move is the standardized effect size, like Cohen's d — divide the difference of means by the pooled standard deviation. That gives you a unitless number that's comparable across measurements.
In our modern example, we ask people to rate cups on a 1–7 quality scale. To estimate the average milk-first rating, you can just take the mean — that's actually maximum likelihood. Or you can use Bayesian estimation to combine the data with a prior, which helps when sample sizes are small.
Open science is, in part, a response: pre-registration limits p-hacking, materials sharing enables real replications, code and data sharing makes errors findable. Transparency at every step.
After author contact, the rate climbed to ~62%. The remaining papers included at least some values that nobody — neither us nor the original authors — could reproduce. pubmed.ncbi.nlm.nih.gov/33614084/
First, terms. Reproducibility = same data + same analysis → same numbers. Replicability = new data + same analysis → similar result. Robustness = same data + different analysis → similar result. They're related but distinct, and metascience asks different questions about each.
On this definition, much of "theory" in psychology is actuallys framework — broad sets of ideas that guide research without making contact with specific predictions. E.g., Bronfenbrenner's ecological systems "theory" is a great framework but a poor causal theory: everything connects to everything.
Theories let us make bets. Predicting "happiness goes up with money" is a roulette bet on red — half of outcomes confirm it. Predicting a specific log-linear relationship is more like betting on a single number. "Risky tests" (specific bets) are how we actually adjudicate between theories.
The web of constructs and assumptions is what Cronbach (1955) called a "nomological network." Constructs sit above the plane of observation; we only ever measure them indirectly through specific operationalizations.
Experiments help by "snipping" the confounds. If you randomly assign who gets money, then friends, education, sleep, and every other potential confound balance, on average, across the groups. The thing I find sort of magical is that this works even for confounds you didn't think to measure.
Take the classic question: does money make people happier? You could survey a bunch of people and you'd probably find a positive correlation. But that correlation is consistent with several stories — money causing happiness, happiness causing money, or some confound (friends, say) driving both.
We had a lot of fun writing this paper, not least because it helped us organize some of the vast theoretical literature on innateness. We'd love to hear your feedback!
For a year and a half, @carorowland.bsky.social, @lehersingh.bsky.social, Marisa Casillas, Shanley Allen, and I have been meeting to discuss whether innateness is still a useful concept to think about in studying language acquisition. Here's our take: osf.io/preprints/ps...
Still smarting from my experience failing to fix my bibtex file, I wondered if my problem was failing to use LLMs as the entry parsers. I think a solution of this type could still work but I'm really dispirited by how hard bibliographic data is for models. Take a look at this ChatGPT transcript!
Cool old study by Rachel Barr et al. (1996): deferred imitation at 12 - 24 months but not at 6 months! doi.org/10.1016/S016...
I always thought preschoolers were too egocentric to do well on communication tasks where they had to talk about novel referents. Old papers reported they'd say stuff like "this one looks like my uncle's hat." @vboyce.bsky.social shows that this is wrong! osf.io/preprints/ps...
Preparing for class and remembering how brilliant this essay is: persci.mit.edu/pub_pdfs/sha... - "The perception of shading and reflectance" (Adelson & Pentland, 1996). Early and beautiful description of Bayesian perception theories.
I am flabbergasted I am by how much vibe coding has expanded my capacities as a scientist and teacher. In the last few weeks, I've mocked up class demos of a live turing test, generated cross-references for an encyclopedia, and prototyped new tablet tasks for developmental psych. It's wild.