Speech and gesture alignment in immersive communication: Human-Human vs. Human-Computer interaction
https://doi.org/10.18384/2949-5075-2024-2-40-51
Abstract
Aim. Based on a pilot multimodal study, this paper explores the alignment patterns of speech and gesture distribution elicited from participants in two regimes of immersive communication, Human–Human and Human–Computer interaction.
Methodology. A multimodal experiment was carried out in two interactional regimes, with the participants discoursing with a human partner and to a computer (with a computer stimulus exposure). Individual variance in discourse schemata in speech and gesture types distribution was established based on hierarchical cluster analysis that enabled to identify groups of participants exhibiting discursive recurrencies in immersive communication.
Results. The obtained results showed that the co-occurrence of gesture and discourse schemata helped reinforce the communicative intent in Human-Human communication. Using regression analysis, the study confirmed that individual variance in discourse schemata distribution was observed in the use of descriptive schemata irrespective of the interactional regime at play.
Research implications. The research findings suffice to assume that the extent of immersion in multimodal communication can be measured, predicted, and even deliberately preset. Overall, the article attests to the applicability of multimodal methodology to exploring immersion
About the Author
V. Potekhin
Diplomatic Academy of the Ministry of Foreign Affairs of the Russian Federation
Russian Federation
Russian Federation
Vadim O. Potekhin – Senior Lecturer, Department of European Languages
ulitsa Ostozhenka 53/2 build. 1, Moscow 119021
References
1. Efimenko, T. N. & Ivanova, Yu. E. (2020). Verbal and nonverbal means of communication as functional operators of the meaning in English business discourse (exemplified in public speeches). In: Bulletin of Moscow Region State University. Series: Linguistics, 1, 15–28. DOI: 10.18384/2310-712X-2020-1-15–28 (in Russ.).
2. Karray, F., Alemzadeh, M., Saleh, J. A. & Arab, M. N. (2008). Human-computer interaction: Overview on state of the art. In: International Journal of Smart Sensing and Intelligent Systems, 1 (1), 137–159. DOI: 10.21307/ijssis-2017-283.
3. Valtolina, S., Matamoros, R. A. & Epifania, F. (2023). Methods for Evaluating Conversational Agents’ Communicability, Acceptability and Accessibility Degree. In: Human-Computer Interaction – INTERACT 2023. Proceedings of 19th IFIP TC13 International Conference. Part II. Cham: Springer. pp. 372–382. DOI: 10.1007/978-3-031-42283-6_21 (Series: Lecture Notes in Computer Science (LNCS). Vol. 14143).
4. Dittrikh, A. G. (2023). The modern fields of pragmalinguistics research. In: Key Issues of Contemporary Linguistics, 4, 42–52. DOI: 10.18384/2949-5075-2023-4-42-52 (in Russ.).
5. Mcneill, D., Levy, E. T. & Duncan, S. D. (2014). Gesture in Discourse. In: Tannen, D., Hamilton, H. E. & Schiffrin, D. (eds.). The Handbook of Discourse Analysis. Chichester, UK: John Wiley & Sons, Inc. pp. 262–289. DOI: 10.1002/9781118584194.ch12.
6. Iriskhanova, O. K. & Cienki, A. (2018). The semiotics of gestures in cognitive linguistics: contribution and challenges. In: Issues of Cognitive Linguistics, 4 (57), 25–36. DOI: 10.20916/1812-3228-2018-4-25-36.
7. Kiose, M., Leonteva, A. & Agafonova, O. (2022). Aesthetic multimodality of speech and gesture: Towards its functional framework. In: Languages and Modalities, 2, 1–17. DOI: 10.3897/lamo.@.78840.
8. Apostolopoulos, J. G., Chou, P. A., Culbertson, B., Kalker, T., Trott, M. D. & Wee, S. (2012). The road to immersive communication. In: Proceedings of IEEE, 100 (4), 974–990. DOI: 10.1109/JPROC.2011.2182069.
9. Shilina, M. G. & Wirth, J. (2021). Immersive technologies in media: Towards the concept of generative mediatization? In: RUDN Journal of Studies in Literature and Journalism, 26 (4), 672–680. DOI: 10.22363/2312-9220-2021-26-4-672-680.
10. Holzapfel, H., Nickel, K. & Stiefelhagen, R. (2004). Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures. In: ICIM ʹ04: Proceedings of the 6th International Conference on Multimodal Interfaces. New York: Association for Computing Machinery. pp. 175–182. DOI: 10.1145/1027933.1027964.
11. Gunes, H. & Piccardi, M. (2007). Bi-modal emotion recognition from expressive face and body gestures. In: Journal of Network and Computer Applications, 30 (4), 1334–1345. DOI: 10.1016/j.jnca.2006.09.007.
12. Kotov, A., Zinina, A., Zaidelman, L. & Arinkin, N. (2021). Generative Classification of Elements of Nonverbal Communication. In: Computational Linguistics and Intellectual Technologies: papers from the Annual International Conference “Dialogue” (2021), 20S, 1095–1105. Moscow: Russian State University for the Humanities. DOI: 10.28995/2075-7182-2021-20-1095-1105 (in Russ.).
13. Khan, T., Barthel, H., Amrhein, K., Ebert, A. & Liggesmeyer, P. (2015). An Interactive Approach for Inspecting Software System Measurements. In: Human-Computer Interaction – INTERACT 2015. Proceedings of 15th IFIP TC.13 International Conference on Human-Computer Interaction (INTERACT’ 2015). Cham: Springer. pp. 1–8. DOI: 10.1007/978-3-319-22698-9_1 (Series: Lecture Notes in Computer Science (LNCS). Vol. 9298).
14. Malakhova, V. L. (2022). Analysis of the formation of English discourse multimodal sense space from the perspective of integrative functional methodology. In: Bulletin of the Moscow Region State University. Series: Linguistics, 1, 60–69. DOI: 10 .18384/2310-712X-2022-1-60-69 (in Russ.).
15. Werlich, E. (1976). A Text Grammar of English. Heidelberg: Quelle and Meyer.
16. Longacre, R. E. (1983). The Grammar of Discourse. New York: Plenum.
17. Virtanen, T. (1992). Issues of text typology: Narrative – a ‘basic’ type of text? In: Text – Interdisciplinary Journal for the Study of Discourse, 12 (2), 293–310. DOI: https://doi.org/10.1515/text.1.1992.12.2.293.
18. Kibrik, А. А. (2009). Modus, genre and other parameters of discourse classification. In: Topics in the study of language, 2, 3–21 (in Russ.).
19. Van Dijk, T. A. (1980). Macrostructures: An interdisciplinary study of global structures in discourse, interaction, and cognition. Hillsdale, NJ: Lawrence Erlbaum Associates.
20. Mann, W. C. & Thompson, S. A. (1988). Rhetorical Structure Theory: Toward a functional theory of text organization. In: Text – Interdisciplinary Journal for the Study of Discourse, 8 (3), 243–281. DOI: https://doi.org/10.1515/text.1.1988.8.3.243.
21. Mandler, J. M. & Johnson, N. S. (1977). Remembrance of things parsed: Story structure and recall. In: Cognitive Psychology, 9 (1), 111–151. DOI: https://doi.org/10.1016/0010-0285(77)90006-8.
22. MacSaveny, T. (2010). Towards a Description of Descriptive Discourse. In: GIALens, 4 (3). URL: https://www.diu.edu/documents/gialens/Vol4-3/Tim-MacSaveny-Description-of-Descriptive-Discourse.pdf (accessed: 21.10.2023).
23. Nippold, M. A. & Scott, C. M. (eds.) (2010). Expository discourse in children, adolescents, and adults: Development and disorders. New York, NY: Psychology Press.
24. van Eemeren, F. H. (2010). Strategic maneuvering. Extending the pragma-dialectical theory of argumentation. Amsterdam-Philadelphia: John Benjamins. DOI: https://doi.org/10.1075/aic.2 (Series: Argumentation in Context. Vol. 2).
25. McNeill, D. & Duncan, S. D. (2000). Growth points in thinking for speaking. In: McNeill, D. (ed.). Language and Gesture. Cambridge: Cambridge University Press. pp. 141–161.
Review
Views:
102
Email this article 
