Replication
Research Question
Can we replicate findings from image-related fMRI neuroscience?
The main goal of this initiative is to replicate findings of papers that use condition-rich fMRI datasets. We put together a list of potential studies here, but you can also suggest a study yourself. These can be studies using large public datasets, or smaller, private datasets as long as the studies are, in theory replicable with LAION-fMRI. From each study we want you to replicate the main findings mentioned in the title and abstract of the paper. You only need to replicate findings that relate to fMRI data in some way. If there are findings that are exclusively based on behavior, modelling, or other types of data, you do not need to replicate those. In your replication please stay as close as possible to the methods of the original study. This also includes using the original code if it's public. If you have any questions about how to run the code or generate metadata necessary for the analyses, we encourage you to get in contact with the original authors.
Statistics
Stay as close as possible to the statistics of the original study. But since LAION-fMRI only contains 5 subjects, all analyses must use subject-level statistical tests. Studies that used between-subject statistics must switch to non-parametric permutation testing. See our dataset section for a tutorial.
Generalization
Research Question
Do these results generalize to different image distributions?
Conducting the generalization analysis is optional.
Additionally, we want to find out how robust these findings are. To assess this, we want to test if the findings generalize to different image distributions within LAION-fMRI. You can find different ways to test generalization below. All the recourses can be accessed through our dataloader.
Independent within-distribution
LAION-fMRI includes generalization splits that each maximally capture the available image space on a global level. At the same time they are maximally different from each other on a local level to avoid any overlap or leakage. In contrast to random sampling, they explicitly control for leakage effects caused by overly similar images between train/test splits.
Out-of-distribution clusters
LAION-fMRI provides cluster-based splits that partition the main image set into 5 distinct regions of the embedding space. By training on a subset of clusters and testing on held-out ones, models can be rigorously evaluated for generalization to unseen parts of the distribution.
Out-of-distribution images
LAION-fMRI includes responses to different types of out-of-distribution images. These can be used to test generalizability to unconventional parts of the image space. The types of OOD images available in LAION-fMRI can be seen below.