Generalization Guide
How to test generalizability
This guide explains how to use the image distribution splits available in LAION-fMRI to test whether the findings you replicated hold up beyond the specific set of images used in the original study. You don't need to test generalizability to participate in the initiative, but only teams that test it are eligible for the generalization award.
There are three methods available. If you conduct a generalization, please use Method 1 and Method 2. Additionally, you can use Method 3, which involves our OOD images.
Below we elaborate on how these image sets can be used for findings that use a train-test split (i.e. encoding models) and those not using a train-test split (i.e. representational similarity analysis).
All splits and image sets are accessible via our dataloader.
If the study you replicated uses only a very specific subset of images (e.g. only images of food or faces), these generalization methods are likely not applicable. In that case, stick to the replication.
Independent within-distribution
What are these splits?
LAION-fMRI provides an 80/20 train-test split designed to maximally cover the image space while minimising overlap between splits. Unlike random sampling, these splits explicitly control for leakage caused by overly similar images, giving you a clean test of whether findings hold on genuinely unseen image content.
For findings using a train-test split
Retrain on the LAION-fMRI train split (80% of images).
Evaluate on the held-out test split (20% of images).
Report results alongside your replication. Consistent performance indicates generalization.
For findings not using a train-test split
Run your full analysis on the LAION-fMRI train split (80% of images).
Repeat the analysis unchanged on the test split (20%), without adjusting any parameters.
Consistent findings across both splits constitute evidence for generalization.
Out-of-distribution clusters
What are these splits?
LAION-fMRI provides cluster-based splits that partition the image set into 5 distinct regions of the embedding space. By training on a subset of clusters and testing on the held-out one, you can evaluate whether findings generalize to genuinely different types of image content.
For findings using a train-test split
Retrain on 4 clusters and evaluate on the held-out one.
Repeat for all 5 clusters, each time holding out a different cluster.
Average results across all 5 folds and report the mean.
For findings not using a train-test split
Run your full analysis on 4 clusters, excluding the 5th one.
Repeat for all 5 clusters, each time holding out a different cluster.
Average results across all 5 folds and report the mean.
Out-of-distribution images
What are these images?
LAION-fMRI includes neural responses to a curated set of out-of-distribution images that fall outside the main image distribution. These span several unconventional categories and can be used to probe whether findings extend to genuinely novel image content, well beyond what within-distribution splits can test.
For findings using a train-test split
Select the OOD image types most applicable to your finding (see below).
Train on the standard LAION-fMRI images and evaluate on the selected OOD images.
Report OOD performance alongside your replication results.
For findings not using a train-test split
Select the OOD image types most applicable to your finding (see below).
Run your full analysis on the selected OOD images.
Consistent results indicate the finding extends to novel image content.
OOD image types
