EEG participant leakage: audit the split before training
Published 2026-10-25 · PhyPA methods editorial desk · Research methods guide
A split can contain different EEG windows and still contain the same people on both sides. If the question is whether a model works on a new participant, checking window IDs alone misses that problem. Start with a manifest of which participant, session and trial produced each window, then audit the groups inside each evaluation fold.
This guide adds a small, runnable check to the EEG benchmark documentation checklist. It uses fictional identifiers and no EEG signals. It does not evaluate a classifier, establish clinical validity or certify an entire analysis as free of leakage.
Choose the unit that your claim holds out
Write the generalization question before choosing a split. For an unseen-participant claim, a participant belongs to only one partition within a fold. For an unseen-session question, a session is the held-out unit; the same participant may deliberately appear elsewhere. A trial-level split answers a narrower question again. Record the difference rather than calling all three “cross-validation.”
The original study Data leakage in deep learning studies of translational EEG compares segment and participant holdouts and shows why performance on new participants can be overstated. Its results concern the evaluated datasets and procedures; they are not a numerical correction to apply to another experiment.
A worked example with four fictional participants
Our toy manifest has P01–P04, each with two windows from one session and trial. In the window-split example, the first window goes to training and the second to testing for every person. The window IDs are distinct, but all four participant IDs cross the boundary.
In the participant-split example, both windows from P01 and P02 go to training; both windows from P03 and P04 go to testing. At the participant level, the first manifest produces four overlap findings and the second produces none. These are deliberately small teaching examples, not adequate sample sizes for an EEG performance study.
Download the standalone split-audit script. With Python 3, run python3 audit_split.py window-split.csv --group participant, then repeat with participant-split.csv. The first command exits with status 1 and reports four overlaps. The second exits with status 0 and reports no overlap in the selected group IDs. The script and these outputs were tested on the supplied synthetic files before publication.
What the manifest must identify
- fold: the evaluation fold. Overlap is checked within a fold, not across the whole cross-validation run.
- partition: train, validation or test. The script checks boundaries between all three.
- participant, session, trial: non-empty source identifiers. Session and trial grouping use composite keys so a repeated “session 1” label in different people is not treated as one session.
- window: a source window identifier within its participant/session/trial. A repeated window across partitions is also reported.
Use stable pseudonymous identifiers consistent across every input file. If different datasets reuse P01 for different people, namespace those IDs first. Conversely, two aliases for the same person must map to one identifier. This script cannot discover aliases, family relationships, overlapping time intervals or duplicate signals from an arbitrary CSV.
Use grouped splitters, then inspect what they produced
GroupKFold keeps the supplied groups separate in each train/test fold. Supplying participant IDs can therefore support a participant-level split. It cannot repair missing or incorrect identities. Save the generated assignments and audit them rather than relying only on the splitter's name.
The downloadable script uses Python's standard library, not scikit-learn. It audits existing assignments; it does not generate folds, enforce class balance or choose the right group for your question. For a nested evaluation, keep a record of the outer split and each inner split. Run the audit within those evaluations and verify that tuning never reaches the outer test partition.
A clean ID check is only one gate
Next, inspect preprocessing. Scikit-learn's data-leakage guidance explains why learned transformations must use training data only. Fitting normalization or feature selection on all the data can leak information even when participant IDs are perfectly separated.
Our example leaves the original manifest untouched and does not silently move rows to make it pass. If a check fails, revisit the intended evaluation, generate a new version of the split and record why it changed. Keep the original as evidence. A zero-overlap message is a result about the identifiers you supplied, not proof of a reliable or clinically useful model.
Next reading and a compact audit record
Save the manifest version, file hash, grouping unit, tool version and output with your experiment. Use the benchmark notes guide to document dataset versions, labels and preprocessing as well. The site's archived benchmarking section supplies historical context; this independently maintained guide reports no new PhyPA research study.