Johns Hopkins University – Homewood Campus – (410-516-5250/office phone)
Thursday, October 5, 2006
3:30 p.m.
Indiana University
“Multimodal Statistical Learning: Linking Words to World”
There are an infinite number of possible word-to-world pairings in naturalistic learning environments. Previous proposals to solve this mapping problem focus on linguistic, social, and representational constraints at a single moment. We examined an alternative account -- a cross-situational learning strategy based on computing distributional statistics across words, across referents, and most importantly across the co-occurrences of these two at multiple moments. The results from a series of experiments showed that cross-situational statistical learning is with the repertoire of human learners. Moreover, we suggest that social cues, as embodied multimodal interactions between children and their caregivers, can be integrated in the cross-situational statistical framework. Toward this goal, we developed a multimodal learning system that learns language from natural environments. In our experiment, adult subjects were asked to act as a caregiver and perform some everyday activities, one of which was reading a picture book to a young child. We collected speech data in concert with multisensory information from non-speech modalities, including visual data, gaze positions, head directions and hand movements. Our computational model (as a young language learner) was then able to identify the sound patterns of individual words from continuous speech using non-linguistic contextual information and employ body movements as deictic references to associate action verbs and object names with their perceptually grounded meanings.
Visit our website: http://www.cogsci.jhu.edu (menu header: Events)