(Over)fitting like a glove: Adjusting regression models for better fit in L2 research
There are growing calls for more robust statistical methods in second language (L2) research (e.g., Larson-Hall & Herrington, 2010). One particular method is regression modeling (Plonsky & Ghanbar, 2018; Plonsky & Oswald, 2017), which provides a flexible, powerful alternative to more traditional procedures. Despite their attractiveness, regression models are susceptible to inflating effect sizes when the model fits the data too well. Such overfitting stems not only from outliers, but from influential data points (i.e., data that are not outliers but which exert a disproportionate influence on the overall model fit), especially when sample sizes are small, as is often the case in the field.
These issues in regression modeling can be alleviated via model validation, but this is rarely ever done (Plonsky & Ghanbar, 2018). This talk addresses the problem by providing an instructive look on how to validate regression models using the best-known method: bootstrap validation. Through discussion of regression fit, model validation, and a step-by-step tutorial with R scripts, we discover that simply taking the best fitting model for a data set may be a limiting analytical decision. Further, we show that bootstrap validation provides a relatively easy way to improve the validity of our effect size estimates.