Typically developing (TD) infants adapt to the social world in part by shifting the focus of their processing resources to the relevant aspects of a visual scene. Any impairment in visual orienting may therefore constrain learning and development in domains such as language. However, although something is known about visual orienting in infants at risk of autism, very little is known about it in infants/toddlers with other neurodevelopmental disorders. This is partly because previous studies focused on older children and rarely compared the children to both chronological age (CA)- and mental age (MA)-matched TD controls. Yet, if visual orienting is important for learning and development, then it is imperative to investigate it early in development and ascertain whether it relates to higher level cognitive functions such as language. We used eye-tracking technology to directly compare visual orienting in infants/toddlers with one of three neurodevelopmental disorders—Down syndrome (DS), fragile X syndrome (FXS) and Williams syndrome (WS)—matched on CA or MA to TD controls (~15 months). We also measured language ability using the Mullen Scales of Early Learning (MSEL). We found that the ability to disengage attention from a visual stimulus in order to shift it to another visual stimulus is related to language ability in infants/toddlers irrespective of group affiliation. We also found that, contrary to the literature, infants and toddlers with DS (but not WS) are slow at disengaging attention. Our data suggest that orienting attention constrains language development and is impaired in DS.