PI: Leah Windsor (IIS/English)
Funder: NSF, $450,000
This project examines multimodal communication among world leaders from a multicultural, interdisciplinary perspective using the Embodied Cognition (EC) framework. This framework demonstrates how multimodal processes, such as discourse, the acoustic production of speech, and nonverbal signals such as gestures and facial expressions, are integrated into human communication. The EC approach will be especially beneficial for understanding politics in opaque environments, like authoritarian regimes, where it is difficult to observe true preferences and priorities of leaders. The PIs' proposed research represents a novel combination of computational linguistic analysis, multimodal signals derived from audio and video analysis, and biometric audience data. To date, multimodal analyses have not been deployed comprehensively on a representative international corpus and extant research relies primarily on corpora from the English-speaking world. Given seismic changes in the international system, this analysis provides a way for scholars to better understand problems like state stability, conflict processes, and regime transition. A multimodal approach to analyzing the international system will help us know more than what leaders’ words alone convey.
This project facilitates the understanding of the importance of multimodal signals by providing a data set, pedagogical materials, and academic publications to translate the meaning of multimodal inputs through the Security and Preparedness program. Differences in culture and language may lead to misunderstandings that can escalate to security crises. By evaluating different communication channels, the PIs propose a clearer picture of countries’ motivations and deeper insight into their preferences and beliefs. Using biophysical indicators alongside language, this project substantially improves the scientific community’s understanding of how multimodal channels work in concert with language to convey meaning, especially in international and multicultural contexts. To accomplish this work the collaborators in this project prioritize hiring and training women and minorities in social and computational social sciences, and STEM fields.