Alex Crenshaw

@crenshaw

Assistant Professor of Psychology. Couples, clinical trials, stats & methods. Views my own.

716 Folgt285 Follower74 Posts

Ist das dein Profil?Übernimm es — dann gehört dir crenshaw.dirty.social.Profil übernehmen

How much do these choices matter? Using a convenience sample of published psychometric data from common measures, differences in operationalizing the same JT formula produced RCI threshold discrepancies as large as 79%, or ~1 SD unit... (6/n)

Alex Crenshaw@crenshaw· 4d

We reviewed 226 clinical trials from four clinical psych journals (2020–2023). Twenty-nine formally assessed reliable change. Most (22/29; 76%) used the JT method. However, we found at least 5 distinct operationalizations of this same JT formula... (5/n)

Alex Crenshaw@crenshaw· 4d

New paper! We find that the Reliable Change Index (RCI), used to statistically evaluate whether an individual patient improved, is operationalized so inconsistently across clinical trials in psychology that results are often incomparable... (1/n)

Alex Crenshaw@crenshaw· Jul 9

We also need better reporting. Only 35% of studies provided enough info to know the operationalization, and 56% reported no information at all. Reporting recommendations:

1. Report how the mean difference was computed
2. Report the exact SD used. Citing a paper isn't enough b/c our review found that authors use terms like “Cohen’s d” or popular citations like Feingold (2009) to indicate different methods.
3. Use the term "SMD" instead of terms like Cohen's d, which no longer imply a specific formula in modern usage. Then report the exact operationalization
4. Prespecify the operationalization
5. Always report the mean difference in original units alongside the SMD

Alex Crenshaw@crenshaw· Jul 9

Using baseline SD meets 7 of 8. All criteria aren't equally important, but baseline SD stands far above other options, where main limitation is potential SMD inflation due to range restriction common at baseline in clinical trials. Second best is post-tx control group SD.

Table 1: Desirable properties of the SMD and extent to which each standardizer meets each property

Alex Crenshaw@crenshaw· Jul 9

My lab conducted a methods review of published clinical psych trials to see how they operationalized the SMD. Of 126 studies reporting an SMD, only 44 (35%) provided sufficient information. Of those, we identified at least 9 different ways the SMD was computed.

Figure 1: Sankey diagram showing broad and specific operationalizations of the SMD in reviewed clinical trials

Alex Crenshaw@crenshaw· Jul 9

Past work shows SMDs computed using different SDs can produce widely differing values for the same effect. Here is just one example:

Screenshot of text. Text reads: Finally, Luo et al. (2022) conducted perhaps the most thorough analysis by computing the SMD in three or more distinct ways for each individual trial that reported sufficient information to do so, finding the median difference in magnitude between the largest and smallest SMD was 0.30 (range = 0.02–1.61). This value is comparable to proposed values for the smallest worthwhile differences for psychotherapy (e.g., Sahker et al., 2025, Supplement 5). Therefore, individual researchers’ choice of standardizer alone, holding all other factors constant, has the potential to influence clinical decision making.

Alex Crenshaw@crenshaw· Jul 9

New commentary out in JCCP! In which I try to solve the problem of incomparable effect size estimates in clinical trials by recommending a standard, one-size-fits-most way to compute the standardized mean difference (SMD; e.g., Cohen’s d) in psychology trials. doi.org/10.1037/ccp0...

Abstract: The standardized mean difference (SMD) is widely used in psychology clinical trials to convert intervention effect estimates from raw units to a common scale, enabling comparison across settings. However, multiple operationalizations of the SMD are possible for clinical trial data, with large potential impacts on estimate magnitudes. In this commentary, I review practices for operationalizing the SMD in the Journal of Consulting and Clinical Psychology and other journals that publish results of clinical trials. I find that only one third of studies report sufficient information to determine the operationalization. Among those, the operationalization varies widely, threatening the SMD’s utility in providing a common scale. Given the common goals and design elements across many trials, it is feasible to establish a default operationalization for the SMD’s standardizer. I outline eight desirable SMD properties and evaluate each option against these properties. The pooled baseline standard deviation meets seven of eight, far more than any alternative, with its main limitation being susceptibility to inflation from restrictive inclusion criteria. I conclude that raw effect estimates should be divided by the pooled baseline standard deviation as a default, one-size-fits-most choice. Adopting this standard will help the SMD meet its promise as a standardized metric, improving comparability of results across different measures, outcomes, studies, and populations. I also provide reporting recommendations to improve transparency and enable recomputation using alternative standardizers when needed.

Alex Crenshaw@crenshaw· Jul 9

We also need better reporting. Only 35% of studies provided enough info to know the operationalization, and 56% reported no information at all. Reporting recommendations:

Alex Crenshaw@crenshaw· Jul 9

Using baseline SD meets 7 of 8, far more than other options, w/ main limitation being potential SMD inflation due to range restriction common at baseline in clinical trials. Second best is post-tx control group SD.

Table showing how well each operationalization meets desired SMD properties

Alex Crenshaw@crenshaw· Jul 9

Sankey diagram showing broad & specific operationalization of SMDs reported in clinical psychology trials

Alex Crenshaw@crenshaw· Jul 9

Past work shows SMDs computed using different SDs can produce widely differing values for the same effect. Here is just one example:

Finally, Luo et al. (2022) conducted perhaps the most thorough analysis by computing the SMD in three or more distinct ways for each individual trial that reported sufficient information to do so, finding the median difference in magnitude between the largest and smallest SMD was 0.30 (range = 0.02–1.61). This value is comparable to proposed values for the smallest worthwhile differences for psychotherapy (e.g., Sahker et al., 2025, Supplement 5). Therefore, individual researchers’ choice of standardizer alone, holding all other factors constant, has the potential to influence clinical decision making.

Alex Crenshaw@crenshaw· Apr 10

Across 24 tx-phase universes: PTSD symptom improvement was fully consistent: both groups improved, w/ no group differences. Relationship satisfaction was partially consistent. Fear of intimacy was highly dependent on analysis approach. The multiverse differentiated robust from fragile results

Figure showing multiverse results for PTSD symptoms during treatment phase

Figure showing multiverse results for fear of intimacy during treatment phase

Alex Crenshaw@crenshaw· Apr 10

We demonstrate the approach using a small RCT comparing prolonged exposure vs cognitive behavioral conjoint therapy for PTSD, which had serious real-world challenges: 65% dropout in one arm, only 4 people at 12-month follow-up, and a common challenge of txs differing in session count and length

Table 2 from manuscript detailing multiverse dimensions

Alex Crenshaw@crenshaw· Apr 10

We expand Del Giudice & Gangestad's (2021) framework to trials and introduce "Type D" decisions, where a best-practice option is itself flawed (e.g., ITT when dropout is severe and differential). All options are flawed, so converging evidence across them is stronger than a single result

Table 1 from manuscript detailing multiverse terminology, expanded decision types from Del Giudice & Gangestad (2021), and decision-making guidelines

Alex Crenshaw@crenshaw· Apr 10

1/7 Clinical trials should preregister analyses. But even careful plans leave room for defensible alternatives, and problems often arise that create decisions no one anticipated. Instead of picking just one defensible option, what if you ran them all? New paper in @collabrapsychology.bsky.social

Title page and abstract. Abstract text: "Making decisions regarding data processing and analysis are crucial steps toward extracting insights from data in clinical trials. Trial registries like clinicaltrials.gov promote transparency about these decisions and encourage making them in advance. However, clinical studies often face decisions with multiple reasonable options outside the bounds of preregistration, such as when studies conduct post hoc analyses, deviate from preregistered plans, or simply were not preregistered. Additionally, even a priori decisions often have multiple reasonable options from which to choose. Methods that maximize transparency and minimize bias in such situations are needed. This paper advocates for applying a “multiverse” approach to analyzing such data from clinical trials. The multiverse approach simultaneously selects and analyzes the various reasonable options for each decision and presents results across all analysis “universes.” We highlight common challenges and decisions when analyzing clinical trial data, review and expand upon the multiverse approach and show how it can address these challenges, and demonstrate the approach using data from a small randomized psychotherapy trial for posttraumatic stress disorder. In the example presented, results were fully consistent across the multiverse for one outcome (posttraumatic stress symptoms), partially consistent for another (relationship satisfaction), and mostly inconsistent for a third outcome (fear of intimacy). The multiverse approach is a flexible and transparent analysis option for clinical trials in the presence of uncertainty regarding data processing and analytic choices."

Alex Crenshaw@crenshaw· Apr 8

7/7 Recommendations: @tandfresearch.bsky.social

Detailed Recommendations for computing the reliable change Index: (1) Use a premade RCI threshold if one has been previously developed from a large sample of a comparable population. (2) If a premade RCI threshold is not available or there is a substantial population mismatch between the premade threshold and one’s study, (a) If N > 100, use the sample estimates of s and rxx to compute the RCI. (b) If N < 30, use estimates of s and rxx from a large-N study that best matches one’s population of study. Measure validation studies are a good default choice. (c) If 30 ≤ N ≤ 100, make a determination based on the study sample size and availability and representativeness of thresholds from a larger sample. (3) Do not use estimates of s and rxx from different samples. (4) Justify the choice of method for the RCI and its specific operationalization, and discuss limitations and biases that may result. Document such choices ahead of time in a study preregistration due to the impact that these choices may have on the strength of apparent evidence for change in a study.

Alex Crenshaw@crenshaw· Apr 8

6/7 An interesting nuance: Sampling error in the SD and reliability terms were positively correlated, so they partially cancelled each other out. The full RCI was less variable than either component alone. But this offsetting was incomplete and couldn't rescue small samples.

Figure 1. Percent of samples with [y] or greater deviation from the population value for the reliable change Index: simulated data. The y-axis represents the percent absolute deviation in the Reliable Change Index, and the x-axis represents the percent of samples that have a deviation as large or larger than the y-axis value. Text labels are provided for the worst 10% and 20% of samples, and for the median sample (50%). Example to aid interpretation: for the high reliability/more items measure (top left panel) when N = 10, 10% of samples (x-axis) have a deviation from the population value of 13% (y-axis) or greater, 20% of samples have a deviation of 10% or greater, and 50% of the samples have a deviation of 5% or greater.

Alex Crenshaw@crenshaw· Apr 8

1/7 The Reliable Change Index is supposed to be a common ruler for deciding whether a patient truly improved. But what if that ruler changes size from study to study? Recent paper in the International Journal of Social Research Methodology: doi.org/10.1080/13645579.2025.2585286