The same phase creates a unique visual rhythm unifying moving elements in time

INTRODUCTION

Each object in the visual world is characterized by visual features like color, orientation, shape, or location, which can exogenously guide the deployment of attention (Theeuwes, 2018; Utochkin et al., 2020; Zhang et al., 2020). Besides such stationary characteristics, dynamic features of moving objects also influence the selection of attention. For example, one may increase vigilance when branches start swinging in the wind. The periodic movements with flowing information (e.g., the synchronous walking of a group, violinists playing together in an orchestra, or the bouncing movements of a ball) may all induce the perception of a visual rhythm with certain frequency and duration. Although previous studies have demonstrated that a variety of features can capture our attention, it is still unclear whether temporal attributes of moving objects can also guide our attention automatically (Wolfe & Horowitz, 2017; Wolfe & Utochkin, 2019). Hence, we decided to address this open question with periodically moving objects to clarify which temporal parameters may be important in visual rhythm perception.

Rhythms are essential features both in the environment and living organisms (Kolers & Brewster, 1985). They carry an evolutionary heritage in our “temporal genes” (Zhou et al., 2014). Harmonious rhythmic movements may signal a peaceful situation, or a unique visual rhythm may signal an approaching predator. We sense these rhythms with specific sensitivity to keep track of the dynamic information unfolding in time. Previous studies indicated a crucial role of audition and a subordinate role of vision in rhythm perception (Grahn, 2012; Grahn et al., 2011; Silva & Castro, 2016), discrimination (Barakat et al., 2015; Villalonga et al., 2020), reproduction (Glenberg & Jona, 1991), or synchronization tasks (Colley et al., 2018; Comstock et al., 2018; Su, 2014a; Sugano et al., 2017). ERP and fMRI studies also found modality specificity in auditory and visual rhythm processing (Araneda et al., 2017; Comstock & Balasubramaniam, 2018; Kang et al., 2018; Pasinski et al., 2016). Thus, questions are accumulated as to why rhythm perception appears to be modality-specific and how rhythms and interruptions of such rhythms are perceptually extracted from the visual environment.

To address these questions, most previous studies induced visual rhythms by a sequence of blinking circular shapes like flashes, which imitate the serial pattern of sequential tones (Celma-Miralles et al., 2016; Levitin et al., 2018; Patel et al., 2005). Such periodic changes of stationary objects are qualitatively different from the perceived visual rhythms in natural environments, as they are always accompanied by motion cues. Thus, the ecological relevance of periodically moving visual stimuli is essential for rhythm perception (Huang et al., 2018). For example, one must determine the periodic movement to catch a bouncing ball, follow the walking pace of a queue, or detect unusually swinging branches. Initial experiments usually used discrete visual stimuli to present visual rhythms, but the unique spatial characteristics of visual rhythms make it questionable to apply the same experimental approaches as in auditory rhythm studies. Recent studies challenged the initial paradigm and indicated that visual rhythms derived from apparent movements have more potential to induce a beat perception, for example, a moving bar (Grahn, 2012; Hove et al., 2010), a human point-light moving figure (Su, 2014b; Wang et al., 2021), or a bouncing ball (Gan et al., 2015; Hove et al., 2013; Huang et al., 2017; Iversen et al., 2015). Indeed, unlike auditory rhythms, which typically arise from a stationary source, visual rhythms are experienced with dynamic spatial information.

Just imagine catching an approaching bouncing ball. We can perceive temporally organized patterns via their spatial movements. For the first time, Li et al. (2014) applied a visual search paradigm to investigate the visual rhythm perception of bouncing balls. They found that the unique visual rhythm did not capture attention automatically since the reaction time became slower as the number of distractors increased. Silva and Castro (2016) later used a perception task and asked the subjects to judge whether the final part of a temporal sequence was compatible with the previous beat structure. They suggested that nonstationary visual stimuli outperforming static ones may be ground in action rather than perception. However, the rhythmic movements are combinations of temporal features, such as phases and amplitudes of the motion. Previous studies have not investigated how these factors might influence visual rhythm processing. Thus, in the current study we performed three experiments to systematically modulate the moving patterns of the bouncing balls and examine their effects on participants' rhythm perception. In the previous experiment of Li et al. (2014), all the balls started bouncing at randomized phases (different initial positions within their trajectories), and their amplitudes were randomized within a predefined range, making the overall visual display seemingly chaotic and complex. It is unknown whether the search efficiency would be improved when the searching background is more transparent and less complex. In the following visual search experiments, we applied the same visual search paradigm and asked the participants to search for a target ball with either a longer or shorter bouncing period that differed from the distractor balls that were sharing the same period. We modulated amplitudes and phases of the balls systematically to see whether the performance would be influenced.

EXPERIMENT 1: MODULATING AMPLITUDE

Researchers found that searching for the bouncing ball of a unique period is not a pop-out process but a serial one (Li et al., 2014). In addition, they indicated that the perception of a visual rhythm demands considerable attention. In their experiments, all the balls started bouncing from randomized phases (different initial positions within their trajectories), and the amplitudes were randomized within a predefined range, making the full visual display look chaotic. Thus, we attempted in Experiment 1 to test the search efficiency when the search background was less complex, that is, making the distractor balls not only sharing the same period, but also sharing the same amplitude.

METHOD

Equipment and stimuli materials

The experiment was conducted in a dark behavioral chamber. With their heads stabilized on a chin rest, participants were seated 60 cm from a 21-in. CRT monitor (Trinitron) with a display resolution of 1024 × 768 pixels and a refresh rate of 90 Hz. The visual search task was presented via MATLAB (Version 2012a, MathWorks) using Psychotoolbox-3 (Kleiner et al., 2007).

The search display contained a gray cross centering on the screen and a set of gray dots (set size: 4, 6, or 8) against a black background within 18° × 18° around the fixation. Each ball subtended 0.8° in diameter and moved independently of the others in a vertical manner with a velocity varying according to a rectified sinusoid (demonstrated in Figure 1). This pattern resembled a bouncing ball, neglecting the influence of gravitation in the real world. The distance between the movement trajectory of each ball was fixed at 2.25°. All the distractor dots bounced with the same period (1500 ms), while the target bounced either faster (period = 1000 ms) or slower (2300 ms) than the distractors. In this experimental design, the bouncing amplitude of the balls was fixed at 4.5°, while phases were randomized between 0 and 2π (the initial position from the bottom to the top of the trajectory).

Results

The average accuracy rate was 94.5%. RTs for each participant in correct trials within the three standard deviations were reserved. Individual response time (RT) patterns in each experimental condition were then averaged to obtain a group-level pattern. A 3 × 2 repeated-measure analysis of variance (ANOVA) was conducted with set size and target type as two independent variables. The results showed a significant main effect of set (F[2,30]) = 27.015, p < .001), indicating an increased RT with the number of distractors. The main effect of target type was also significant (F[1,15] = 46.558, p < .001), demonstrating a longer RT for the slower target (longer period) than for the faster target (shorter period). Besides, the interaction between set size and target type was also significant (F[2,30] = 8.769, p < .01). Further analyses revealed the different search efficiencies for the faster and slower targets. Specifically, the search slope for faster targets was 336 ms per item, while the search slope for slower targets was 522 ms per item.

It has been indicated that the visual rhythm was surprisingly not processed in a parallel manner (Li et al., 2014). Our results confirmed that the reaction time needed for perceiving a unique visual movement rhythm increased as the number of distractors grew. These findings suggested that reconciling the amplitude of all distractors that share the same period would not change the serial search pattern. The results of Experiment 1 also demonstrated an asymmetry of visual rhythm search, that is, searching for the faster target was more efficient than for the slower target as the search slopes indicated (see Figure 2, left section).

EXPERIMENT 2: MODULATING PHASES

In Experiment 2, we reconciled the phases of the distractor balls in order to find out whether and how the searching efficiency would be influenced. The bouncing phases were set to zero at the beginning of each trial. After that, all balls started moving from the bottom of the trajectory. The amplitude was randomized between 2.5° and 6.5°. All other aspects of the methodological details were the same as in Experiment 1. The same group of subjects participated in this experiment.

EXPERIMENT 3: MODULATING BOTH AMPLITUDE AND PHASES

The first two experiments had revealed the different consequences of separately reconciling the amplitude and the phase of the bouncing ball movements. In Experiment 3 we further reduced the background complexity, making all bouncing amplitudes of the balls fixed at 4.5°, and the bouncing phases of all balls were set to zero at the beginning of each trial. All other parts were the same as in Experiment 1. The same group of subjects participated in this experiment.

GENERAL DISCUSSION

One important finding of the current study is the critical role of phases in visual rhythm processing. If all distractor balls with the same period are phase synchronized, the unique visual rhythm of the target ball, which is “out of phase,” pops out immediately. In other words, the search time does not change as the number of distractors increases. This finding is inconsistent with the study of Li et al. (2014). In their study, search efficiency for a faster or a slower periodically moving target decreased as the number of distractors increased. The response time increased against the set size of the searching display, which indicated that visual rhythm is not processed in a parallel mode.

In the current study, when all bouncing balls started at the same relative position on their trajectories, representing the same initial phase, the unique visual rhythm could pop out immediately. This observation suggested that participants were relatively quick to detect the target that did not follow the common fate, and that the flat reaction time function against the set size indicated an efficient search or parallel processing. This finding demonstrated that phase synchronization in the visual search display could be a guiding component since the unique visual rhythm captured attention automatically. In everyday life, we often experience the pop-out of unique visual rhythms. For instance, we see birds landing on swinging branches in natural scenes, whose movements compose the visual rhythm in the dynamic environment. This visual rhythm, like swaying branches, can be different in amplitude like length, area, or volume; the same material creates the same phase, and a different rhythm is somewhat discordant. It may be helpful to think about a parade where a line of people steps at a unified pace. If someone is stepping earlier or later, the entire rhythm in its gestalt is interrupted, and we can recognize this disruption immediately.

In Experiment 1, where the amplitude was reconciled, the result demonstrated a similar search pattern to that described by Li et al. (2014), suggesting that considerable attentional resource is required for searching the outlier in the rhythmical movement environment. However, this does not imply that the amplitude reconcilement had no benefit. Comparing the results of Experiments 2 and 3, we found a decrease in overall response time. These results suggested an improvement brought by amplitude reconcilement, which was insufficient to switch the search pattern from serial search to parallel search. Thus, while the amplitude is not as essential as the phase in rhythm perception, it also contributes to visual rhythm search. The performance improvement was a slight one but valid, which might be caused by modulating the overall entropy of the searching background. As the current study demonstrated, systematically reducing the background noise would significantly improve the search efficiency. Furthermore, aligning the amplitude would accelerate the response, and aligning the phases would even change the search pattern from serial search to a parallel one. In our experiments, distractors with the same phase are grouped and segmented from a target with a different phase. We can thus explain why a unique visual rhythm does not pop out in the previous study (Li et al., 2014). The visual rhythms are defined by the conjunction of the features like amplitude, phase, and period, but it is the phase not the amplitude that makes a parallel search of a unique visual rhythm.

Another important finding in the current study was the asymmetry of searching efficiency for a short-period target compared to a long-period target. This result is consistent with the observation in a previous study (Li et al., 2014). Nevertheless, the gap between distractors and the short-period target was smaller than that between the long-period target. If the search is symmetrical, the results should show the long-period target being comparable to the short-period one; but this was not the case in all three experiments. This asymmetry of visual rhythm search in which a higher-frequency rhythmic movement captures our attention in a more straight forward manner than a lower-frequency component suggested an essential mechanism in visual processing. It has to be stressed that this is different from the motion pop-out phenomenon (Ivry & Cohen, 1992). The short-period target did not necessarily have a higher speed as the amplitudes were random. During the periodic movement of a ball, the real-time speed changes all the time. Hence the subjects could not pick out the target by speed difference.

Instead, this asymmetry tendency is rooted in rhythm, in lockstep with our rhythmic sensory processes. Evidence suggests that animals and humans sense the environment in snapshots, rather than continuously, with the brain rhythms of enhanced sensitivity to sensory input cycling-specific frequencies (Baumgarten et al., 2015; VanRullen, 2016; VanRullen et al., 2014). The rhythms of the sensory system facilitate the sampling of such information. Meanwhile, the rhythms of the motor system generate outputs that are likely relevant for other organisms. For example, rhythmic patterns may signal an approaching predator or a conspecific trying to communicate (Lakatos et al., 2019). Therefore, a foundation for this critical search asymmetry tendency may be an evolutionary adaptation for field survival (Patel, 2014; Yang et al., 2018). In addition, rhythm perception may be anchored in social synchronization across domains and species (Bao, 2017; Bao et al., 2015). In the natural environment, a peaceful situation mainly consists of slow rhythmic movements or a tranquil atmosphere, like swinging grasses, branches, or ripples raised by the breeze. We only need to pay a little attention to these regularities. If regularities are broken, something must have happened, and one had better pay more attention.

Just imagine a situation of high-frequency and low-frequency interruption in a peaceful environment. The swinging grasses are dipping and swaying peacefully at a slow frequency when a snake suddenly intervenes. The slow rhythmic movements of the grassers can be illustrated as sinusoidal curves with a long period and the moving snake inducing a relaxation oscillation with a shorter period. If we view them from the time domain, it is rather hard to point out the curve of the snake, because space, amplitudes, and time coincide. However, the difference is clear enough if we compute a spectrum of these waves and view from the frequency domain. In simple terms, the sudden disturbance (e.g., a bird landing on branches or a snake zigzagging through the underbrush) would always result in a higher temporal frequency component. No matter at what phase the high-frequency interruption appears or whether the high-frequency interruption amplitude is large or small, this component is always distinct in the frequency domain (see Figure 3, left).

On the contrary, low-frequency interruption does not stand out in the spectrum (see Figure 3, right). Thus, we have a higher sensitivity and a lower threshold for the high-frequency components as they indicate unexpectedness. We may notice such objects with faster tempi in the visual search scene even if we are not explicitly looking for them. Such effortless processing can explain the relatively high search efficiency for the high-frequency target (or shorter period target), which is the essential requirement for precisely predicting the occurrence of the possible danger and timing the initiation of the respective motor programs adequately. Recent studies suggest animals that can imitate sound, for example, humans, chimpanzees, and parrots, have evolved an ability to move with a beat (Kotz et al., 2018). Concerning the evolutionary development of behavioral synchrony in humans, it might be interesting to test this effect in our closest living relatives, like chimpanzees, who can perform rhythm tapping tasks (Hattori et al., 2015; Takeya et al., 2017).

To conclude, we integrate temporal and spatial information of a periodic movement to perceive a visual rhythm, which is characterized by a conjunction of different components, that is, period, phase, and amplitude. Searching a unique visual rhythm generally requires serial deployment of attention. However, the target can suddenly “pop out” as an “oddball” when the phase is violated. This effortless parallel search pattern reflects the critical role of the phase in visual rhythm perception.

CONFLICT OF INTEREST STATEMENT

The authors declare that they have no conflicts of interest with respect to their authorship or the publication of this article.

ETHICS STATEMENT

The study was approved by the local Ethics Committee of the School of Psychological and Cognitive Sciences, Peking University. Informed consent was given before experiments, and a reasonable honorarium was bestowed after experiments.