2.1 Data and script availability

All data, as well as all presentation and analysis scripts, will be made freely available online upon publication, at the Donders Repository.

2.2 Participants

Forty-two adult volunteers (16 male, average 27 years) participated in the experiment. Volunteers were excluded when they had more than 20% of no response trials (n = 7) or had a dental wire (n = 1). Thirty-four were included for the behavioural and MEG analyses. This sample size was determined a priori to ensure 80% power to detect a within-participant effect of medium size (d ≥ 0.5, paired t test). This effect size should not be interpreted as an expected effect size, but we instead decided on this particular effect size as the minimum value we wanted to be sensitive for. All participants had normal or corrected-to-normal visual acuity, normal hearing, and health conditions consistent with the experiment. This study was approved under the general ethics approval (“Imaging Human Cognition”, CMO 2014/288) by CMO Arnhem-Nijmegen, Radboud University Medical Centre. All participants provided written informed consent.

2.3 Apparatus

Computational routines were generated in MATLAB (The MathWorks) and stimuli were presented using “Psychtoolbox” (Brainard, 1997). A PROpixx projector (VPixx Technologies, Saint-Bruno, QC Canada) was used to project the visual stimuli on the screen, with a resolution of 1920 × 1080 and a refresh rate of 120 Hz, and the audio stimuli were presented through MEG-compatible ear tubes. Behavioural responses were collected via a MEG-compatible response box.

MEG was recorded from a whole-head MEG system with 275 axial gradiometers (VSM/CTF Systems, Coquitlam, BC, Canada) in a magnetically shielded room and digitized at 1200 Hz. Eye position data was recorded during the experiment using an Eyelink 1000 eye tracker (EyeLink, SR Research Ltd., Mississauga, Ontario, Canada) for further eye blink and saccade artefact rejection. During the session, head position was recorded and monitored online (Stolk et al., 2013) by coils placed at the nasion, left and right ear. At the end of each block, participants were asked to reposition the head in case they moved more than 5 mm away from the initial position. MEG analyses were performed using FieldTrip software (Oostenveld et al., 2011) and repeated measures ANOVA were performed in JASP, Version 0.9.0 (JASP Team, 2018).

2.4 Stimuli and general task

Participants performed auditory and visual discrimination tasks. In visual trials, the target was a grating of 3 degrees of visual angle, with a spatial frequency of 2 cycles per degree (cpd), random phase, and with one of six possible orientations (15, 45, 75, 105, 135, 165 degrees) surrounded by a magenta circle and presented centrally for 100 ms. In auditory trials, one of six possible pure tones (501, 661, 871, 1,148, 1,514, 1995 Hz) was presented for 100 ms as a target.

The target was always followed by a delay period, after which a probe stimulus was presented for 100 ms. The probe was similar to the target with the exception of the pitch/orientation feature. Participants had to judge whether the probe was tilted clockwise (CW) or counterclockwise (CCW) relative to the target in visual trials or whether the probe had a frequency higher or lower than the target in auditory trials. They always responded with a button press of either the index (lower/CCW) or middle finger (higher/CW) of their right hand.

Stimulus presentation timing was either nonrhythmic or rhythmic, in order to manipulate temporal expectations. The experimental session started with the nonrhythmic trials and the rhythmic trials were presented subsequently.

2.5 Nonrhythmic trials

The nonrhythmic trials began with a central white fixation point (0.4 degrees of visual angle) with a surrounding cyan circle (3 degrees), and after a random intertrial interval (ITI) chosen from a uniform distribution between 0.6 s and 1.6 s, a target was presented for 100 ms. Three seconds after target onset, the probe was presented, and participants had to indicate their response. There was no time limit for the responses. Participants received performance feedback for 400 ms and a new ITI started immediately (Figure 1a). There were 24 trials in each block. Six blocks were randomly presented to participants (three auditory and three visual), resulting in a total of 72 trials per attended sensory modality. To ensure participants understood the task, they performed at least six easy practice trials for each condition before the procedure. Practice trials were not included in analysis.

Performance in nonrhythmic trials was also used to calibrate visual and auditory parameters for the following rhythmic experimental manipulation. The staircase method was QUEST as implemented in the Palamedes Toolbox (Prins & Kingdom, 2018), which we set to achieve a performance around 75%. For both conditions, a cumulative normal function with a lapse rate of 0.1 and the mean of the posterior were used as the staircase parameters. For the visual condition, the beta value (slope) was 1 and the prior alpha (threshold) range was a normal distribution with mean of 10 degrees, standard deviation of 5, ranging from 0 to 20 degrees. During the experiment, the chosen alpha value plus a random value from a normal distribution function (mean 0, std 1) was added or subtracted from the target orientation value. For the auditory condition, the beta value was 100, and the prior alpha range was a normal distribution with mean of 0.1, standard deviation of 0.1, ranging from 0 to 0.2. During the experiment, the chosen alpha value plus a random value from a normal distribution function (mean 0, std 0.01) were multiplied with the target pitch value and the resulting value was added or subtracted from the target value. Beta and alpha prior values were based on prior piloting results.

2.6 Rhythmic trials

In rhythmic blocks, each trial started with a fixation point and a cyan circular border (Figure 1 A). Half a second later, the first attended-modality stimulus was presented for 0.1 s. This stimulus was presented several times (3 to 6, balanced and randomly chosen per trial) with a fixed interval between them (1 s) to create a rhythmic stream. Interleaved and irrelevant to the task, a second stream of stimuli in the other sensory modality stimuli was presented (unattended stream). Therefore, the interval between the onset of adjacent stimuli was 0.5 s. The last attended stimulus of the sequence (target) was marked by a change in an irrelevant feature to warn participants that the next presented stimulus would be the probe. The design with repeated stimuli was important to induce the rhythmic expectations. We used a variable number of repetitions in order to induce an uncertainty in the exact time point of target presentation, thereby making the rhythm itself (rather than one particular time point) all the more relevant to attend. The warning signal was a higher volume sound (auditory trials, equal to nonrhythmic target volume) or a magenta outline (visual trials, instead of cyan outline). Half a second after target onset (400 ms after the target offset), an unattended modality stimulus was always presented for 100 ms. Probes had a positive or negative difference in orientation or pitch based on the output from the previous staircase procedure and were also marked by the magenta outline or by the volume increase. The interval between target and probe (SOA) could be 1, 1.5, 2, 2.5 or 3 s with the respective probabilities: 25%, 12.5%, 25%, 12.5%, 25%. Participants were informed at the beginning of the experimental session that the timing of the probe was most likely to follow the attended rhythm, that is, it would likely to occur in phase with it. A new trial with a new modality-relevant stimulus would appear 2 s after the probe. Each block consisted of 12 trials.

There were 16 auditory and 16 visual rhythmic blocks. There were 192 trials for each sensory modality. Volunteers performed 48 trials for each in-phase delays (1, 2, 3 s) and 24 trials for each antiphase SOAs (1.5, 2.5 s) in each sensory modality condition. The blocks were always presented in a pseudo-random order, where no more than 3 same type blocks could be presented in a row, and participants were instructed about the block-type before its beginning.

2.7 Behavioural analysis

Trials where participants did not respond within 3 s postprobe range were treated as incorrect trials in accuracy analyses and were excluded from reaction time (RT) analyses. Accuracy scores were arcsin-transformed before all statistical tests to improve normality. Both measures were submitted to a 2 × 5 repeated measures ANOVA with modality (auditory or visual) and the five SOAs as factors. Mauchly's test of sphericity was performed, and Greenhouse–Geisser correction was applied in case of sphericity violation. Holm correction for multiple comparisons was performed for all post hoc analyses, when applicable.

2.8 MEG preprocessing

An antialiasing low-pass filter at 600 Hz was used during the online MEG recordings. Nonrhythmic trials were segmented between 0.2 s before the target until 0.5 s after the probe. Rhythmic trials were segmented between 0.2 s before the first stream stimulus until 0.5 s after the probe. After segmentation, synthetic third-order gradient correction was applied and the channel- and trial-wise mean was subtracted from the traces. Trials with eye movements, muscular activity and with an unusually high variance were excluded from the further analyses using a semi-automatic procedure (rejected trials: mean = 7.2%, SD = 3.3%). Sensors showing an unusually high variance were rejected following the same procedure (rejected sensors: mean = 2.8%, SD = 1.2%). After artifact rejection, data were off-line down sampled from 1,200 Hz to 400 Hz to speed up analyses, followed by an independent component analysis to identify and remove residual eye, heart and other muscular components. A discrete Fourier transform was used to suppress line noise at 50 Hz and its harmonics, 100 Hz and 150 Hz.

2.9 Planar combined event-related fields

For the analysis of event-related fields (Figure 2), all trials were low pass filtered at 35 Hz and baseline corrected from −0.1 to 0 s. All nonrhythmic trials (number of visual trials ranged from 54 to 72, median = 69; number of auditory trials ranged from 65 to 72, median = 71) were considered from −0.2 s to 2.5. To better illustrate the activity during the delay of the rhythmic trials, only 2.5 and 3 SOAs trials (number of visual trials ranged from 52 to 72, median = 64.5; number of auditory trials ranged from 60 to 72, median = 68) were considered from −2.2 s to 2.5 s, including the 3 repetitions of attended and unattended stimuli in a trial. We chose to include only longer SOA trials for the evoked-field analysis to have a sufficiently long “clean” window (i.e., a period without any stimulation). For each participant, trials were time-lock averaged. MEG axial gradiometers were then transformed to planar configuration (Bastiaansen & Knösche, 2000) and combined as the root-mean-square of horizontal and vertical sensors. The combined planar activity from participants were averaged in the end.

2.10 Multivariate pattern analyses

MVPA were performed using linear discriminant analysis (LDA) as implemented in MVPA-Light toolbox (https://github.com/treder/MVPA-Light). Features consisted of activity in the MEG sensors (267 ± 3 sensors). Feature scaling was performed as preprocessing step in all analyses: data were normalized for each time point, across trials, using z-score transformation based only on the training set. Final scores were calculated based on the distances estimated by LDA from the six classes' centroids in multiclass classification or from the hyperplane in the two-class classification.

2.10.2 Specific feature classification: Temporal decoding

To potentially increase sensitivity to stimulus-specific activity during the delay period, we also performed a decoding analysis while training on those time points of maximal stimulus-evoked activity. Trials were locked to the target and segmented until the probe moment. Subsampling by averaging 32.5 ms temporal windows (13 points in time) was applied to improve the signal-to-noise ratio. We performed classification in a leave-one-trial-out cross-validation approach. Accordingly, excluding the rhythmic test trial, all stimuli segments from rhythmic and nonrhythmic trials were used for training. Segments were baseline corrected based on pretarget window (−0.1 to 0 s). The activity used in the training set was the average activity of each axial sensor from 0.1 to 0.2 s after a stimulus presentation. This training time period was chosen based on a previous study showing that the visual template effect (Kok et al., 2017) resembles the ERF peak activity (Figure 3b). Depending on the test trial task, the auditory and visual scores were assigned an attended or unattended label. Trial length was SOA-condition dependent.

2.11 Analysis of the rhythmicity of sensory modality nonspecific classification

To evaluate the rhythmicity of classification scores during the delay (1 to 2.5 s), two models were fit for each participant and each condition (nonrhythmic and rhythmic). The first model was a linear model with intercept and slope as free parameters. The second model combined a linear function with a 1 Hz sine and 1 Hz cosine function (which is equivalent to a 1 Hz sinusoid with phase as a free parameter) (Zoefel et al., 2019). The amplitude of the 1 Hz sinusoid was calculated as the square root of the summed squares of the sine and cosine coefficients. The phase was calculated as the four-quadrant inverse tangent between the coefficients of the sine and cosine.

We compared the improvement in model fit at the group level that resulted from adding the sinusoid term between the rhythmic and nonrhythmic trials by performing a Wilcoxon signed-rank test. The same approach was used to compare the sinusoid amplitudes between the rhythmic and nonrhythmic trials. To evaluate the phase consistency of the sinusoid term across participants, Rayleigh tests were performed for each condition.

3.1 Behavioural performance

We first tested whether, in the present task, rhythmic presentation of targets resulted in a rhythmic modulation of perceptual performance. We measured performance based on accuracy and reaction time (RT) (Figure 1c). Accuracy was lowest for the earliest SOA (mean = 76.7%, SEM = 1.8%), an effect most pronounced for the attend-auditory blocks. This was backed up by a significant main effect of SOA (F(4,132) = 6.37, p = 6.172e-04, ω² = 0.03), as well as an interaction of SOA and attended modality (F(4,132) = 5.02, p = 8.451e-04, ω² = 0.02). Overall accuracy was not different between the modalities (main effect of sensory modality: F(1,33) = 0.02, p = 0.886, ω² = 0), and despite the interaction, SOA affected accuracy in both the attend-visual and attend-auditory blocks (simple main effects analysis of SOA, auditory: F(1) = 7.45, p = 1.92e-05; visual: F(1) = 2.89, p = 0.025). However, only the first SOA differed from the other intervals (1.5 s: 81.3 ± 1.8%, t(33) = −3.39, p = 0.015, d = −0.582; 2 s: 81.8 ± 1.8%, t(33) = −4.76, p = 3.711e-04, d = −0.82; 3 s: 80.7 ± 1.8%, t(33) = −3.93, p = 0.004, d = −0.67), except from 2.5 s (79.9 ± 1.9%, t(33) = −2.34; p = 0.181, d = −0.4). Additionally, Bayes factor analysis shows that the data were about four times more likely under the null than under the alternative hypothesis (BF10 = 0.246, Bayesian t test) when computing in-phase (2 and 3 s) and antiphase (1.5 and 2.5 s) accuracy scores from the four remaining SOAs.

Reaction times decreased with increasing SOA (1 s: 896 ± 38 ms; 1.5 s: 780 ± 28.5 ms; 2 s: 743 ± 24.5 ms; 2.5 s: 730 ± 23.5 ms; 3 s: 735 ± 23.6 ms; main effect of SOA F(4,132) = 32.27, p = 2.234e-09, ω² = 0.12). Reaction times were not significantly different between attended sensory modalities (F(1,33) = 0.09, p = 0.763, ω² = 0). We did observe an interaction of SOA and attended modality (F(4,132) = 7.57, p = 0.001, ω² = 0.02), while SOA affected reaction time in both the attend-visual and attend-auditory blocks (simple main effects analysis of SOA, auditory: F(1) = 28.98, p = 2.725e-17; visual: F(1) = 12.36, p = 1.438e-08). Responses for the shortest SOA (1 s) were slower than for the other SOAs (post hoc t tests, 6.05 < t(33) < 7.32, 2.14e-7 < p < 5.779e-6, 1.04 < d < 1.26), and responses for the 1.5 s SOA were slower than those for the longer SOAs (2 s: t(33) = 2.82, p = 0.032, d = 0.48; 2.5 s: t(33) = 3.29, p = 0.014, d = 0.56; 3 s: t(33) = 3.18, p = 0.016, d = 0.55), while response times for the SOAs >1.5 s did not differ among one another (−0.59 < t(33) < 1.31, 0.6 < p < 0.88, −0.1 < d < 0.22). When computing in-phase (2 and 3 s) and anti-phase (1.5 and 2.5 s) reaction times from the four last SOAs, there was no evidence for any hypothesis (BF10 = 1.833, Bayesian t test). Taken together, behavioural performance provides no evidence for a significant rhythmic modulation of perceptual performance, but instead points toward a hazard rate effect.

3.2 Stimulus-specific information is decodable from MEG sensors during stimulation only

Next, we turned our attention to the neural consequences of interleaved multisensory rhythmic stimulation. Figure 2 shows the event-related fields for MEG sensors approximately overlying auditory and visual cortices, in all different conditions. As expected, auditory and visual stimuli elicited pronounced event-related fields at the auditory and visual associated sensors. The evoked activity returned to baseline levels approximately 1 s after stimulus presentation.

To quantify whether the neural signals contained stimulus-specific information (i.e., information about which of the six auditory pitches or visual orientations was present), we performed a multivariate pattern analysis. Specifically, we trained classifiers on the period (−100 to 500 ms) from all stimuli presented in rhythmic trials with SOAs 1, 1.5, and 2 s, and quantified how well these generalised to both the stimulus and delay periods of the rhythmic trials with the longest SOAs (2.5 s and 3 s), as well as to the nonrhythmic trials. Train and test data here are thus fully independent. We investigated the cross-temporal generalisation of these signals between the full stimulus training period to the combined stimulus and delay testing period.

We observed a strong feature-specific signal when training and testing the classifier on similar time points (Figure 3a). In all conditions, there were high levels of stimulus information in the diagonal (nonrhythmic: from 55 ms to 305 ms poststimulus, p < 0.001; attended rhythmic: from 50 ms to 295 ms poststimulus p < 0.001; unattended rhythmic: from 75 ms to 275 ms poststimulus, p = 0.002; p values estimated using cluster-based permutation tests). However, we found no evidence of a generalisation of this activity to other time points in the delay period or in anticipation of events.

This first analysis suggested that sensory representations elicited by a specific feature did not generalise to the delay period. We found a momentary and transient feature-specific signal that peaked after stimulus presentation. To test whether increasing the size of the training set might increase our sensitivity to a potentially missed result, we repeated the classification procedure within the rhythmic conditions only. In this new analysis: (1) data from all trials (excluding a single trial) were used as a training set (i.e., we used a leave-one-trial-out procedure); (2) we used as the training time the period around the ERF peak activity (100 to 200 ms). Similar to our previous analysis, we found that only periods around stimulus presentation had scores higher than chance (auditory attended: from 155 ms to 220 ms, p < 0.001; auditory unattended: 143 ms to 240 ms, p < 0.001; visual attended: from 123 ms to 155 ms, p = 0.008; visual unattended: from 143 ms to 175 ms, p = 0.004; Figure 3b). We did not find feature-specific sensory activation during the delay period and patterns evoked by specific orientations and tones were restricted to periods of stimulus-driven activity.

3.3 Sensory cortices preactivate rhythmically during delay periods

Contrary to our expectations, we did not find stimulus-specific anticipatory sensory templates using the present experimental design. It is, however, known that rhythmic stimulation can entrain neural activity in related sensory areas. Even though, given the above, this entrainment is likely not feature-specific, we next investigated whether multimodal rhythmic stimulation induced nonspecific, yet modality-specific, rhythmic activation of sensory cortices during the delay, as an exploratory analysis. We again used a temporal generalisation approach, this time to decode the attended modality (visual or auditory). The modality decoding reflects how strongly a pattern corresponding to visual activation is present in the sensors versus how strongly a pattern corresponding to auditory activation is present considering the attentional label as the direction of evidence. A significant positive decoding means that neural sources typically processing information of the attended sensory modality are more active than neural sources typically processing information of the unattended sensory modality (in reverse for significant negative decoding scores).

The attended sensory modality was significantly decodable from the signal across several time points in both rhythmic and nonrhythmic trials (cluster-based permutation tests: nonrhythmic p < 0.001; rhythmic p < 0.001, Figure 4a). Importantly, the modality signal extended throughout the delay periods in both types of trials (Figure 4a). Its temporal evolution differed between conditions: the modality signal showed clear rhythmicity only in the rhythmic trials, indicating a preactivation signal related to temporal expectation that was not immediately driven by any stimulus. The negative values during early training periods in rhythmic trials can be explained by noting that the unattended (i.e., different modality) stimulus was presented at those times.

In both types of trials, we observed an early pattern of activity (training time 0.08 s to 0.13 s) that was strong and generalised to different testing times throughout the delay period (Figure 4a). To study the temporal dynamics of this activity in more detail, we further analysed performance over time for a classifier trained on this time window, in both types of trials. Figure 4b shows how modality activity evolves. We observed a clear oscillatory modulation of decoding scores, which can be seen during the delay period for rhythmic trials, and which was absent for nonrhythmic trials. Critically, the last stimulus presented in this period was at 0.5 s, with no other stimulation after that. Additionally, 400 ms after the offset of the unattended stimulus (at 1 s), the modality signal reverses to the attended modality by staying largely positive and significant during the delay. This suggests that such modality signal during the delay reflects preparatory sensory activity rather than the previous stimulus evoked activity.

An oscillatory modulation of the modality signal was clearly present in the grand-averaged data (Figure 4b). We next assessed whether this rhythmicity was reliably present across participants, by fitting two models to the activity in the delay period (1 to 2.5 s) for each participant. The first was a linear model with intercept and slope as free parameters. The second model was a combination of a linear function with a 1 Hz sine and 1 Hz cosine function (which is equivalent to a 1 Hz sinusoid with phase as a free parameter) (Zoefel et al., 2019). The combined 1 Hz-linear model provided a significantly better fit of the data for 26 out of 34 participants in the nonrhythmic condition (Wald test controlling for the extra degree of freedom; F values range: 0.88 to 76.72, critical F(2,295): 3.03), and for 31 out of 34 in the rhythmic condition (F values range: 0.61 to 189.47, critical F(2,295): 3.03). As the critical test of whether stimulus periodicity induced a rhythmic modulation of sensory cortex activation during delay periods, we compared the improvement in model fit that resulted from adding the sinusoid term between the rhythmic and nonrhythmic trials. The improvement of adding an oscillatory function was considerably higher for rhythmic than nonrhythmic trials (Wilcoxon signed-rank test of relative F values across participants; Z = 3.77, p = 1.634e-04; Figure 4c, top left). Furthermore, the model fits for the rhythmic trials had significantly higher 1 Hz amplitudes than those for nonrhythmic (Wilcoxon signed-rank test of 1 Hz amplitudes across participants; Z = 3.86, p = 1.156e-04; Figure 4c, top right).

If the delay-period oscillatory modality signal is the result of entrainment by the rhythmic stimuli, one would expect the phases of this signal to be consistent across participants, specifically for the rhythmic (and not the nonrhythmic) condition. This is indeed what we observed: phases were not significantly different from uniform in nonrhythmic trials (Rayleigh test; Z(33) = 0.07, p = 0.93), but we observed a clear phase concentration in rhythmic trials (average phase = 0.525 rad, Z(33) = 9.37, p = 4.783e-05; Figure 4c, bottom panels). To rule out that the evoked activity of the irrelevant stimulus in the rhythmic condition at 0.5 s could be an explanation for a stronger phase-locking compared to the nonrhythmic, we also tested the phase consistency at later latencies. Phases were still consistent (Z(33) = 8.708, p = 1.016e-04) in the rhythmic condition when we considered the analysis window from 1.5 to 2.5 s (1 s after the last presented stimulus), while there was no phase consistency in the nonrhythmic condition, already in the window from 1 to 2 s (so also 1 s after the last presented stimulus; Z(33) = 0.5, p = 0.611).

Taken together, these results demonstrate that the rhythmic stimulation resulted in a rhythmic preactivation of sensory cortices, which was consistent across participants. Importantly, this preactivation was observed during the delay period, that is, without any ongoing sensory stimulation, suggesting a true entrainment of endogenous neural signals.