No.030 Multi-activity in interaction: a multimodal perspective on the complexity of human action

Icon

NII Shonan Meeting Seminar 030

Organizers

Lorenza Mondada, Basel University, Swizerland
Mayumi Bono, National Institute of Informatics, Japan
Aug Nishizaka, Meiji-Gakuin University, Japan

Participants

Satomi Kuroshima, Meiji-Gakuin University
Eiko Yasui, Nagoya University
Kaoru Hayano, Ochanomizu University, Center for Foreign Language Education
Katsuya Takanashi, JST/ Kyoto University
Hidehito Honda, NII
Nobuhiro Furuyama, NII
Kuniyoshi Kataoka, Aichi University
Hiroaki Tanaka, Kyoto Institute of Technology
Kouhei Kikuchi, NII
Takeshi Hiramoto, Kyoto University
Ayami Joh, JSPS / University of Shiga Prefecture
Akira Takada, Kyoto University
Daiji Kimura, Kyoto University
Eric Hauser, University of Electro-Communications
Lefebvre Augustin, Université Sorbonne Nouvelle Paris III
David Aline, Kanagawa University
Domenic Berducci, Toyama Prefectural University
Adam Brandt, Kansai University
Misao Okada, Hokusei Gakuen University
Ikuyo Morimoto, Kwansei Gakuin University
Gustav Lymer, University of Gothenberg
Tomoko Ikeda, J. F. Oberlin University
Hiro Yuki Nisisawa, Tokiwa University
Motoko Igarashi, Joetsu University of Education
Yoshifumi Mizukawa, Hokusei Gakuen University

Schedule

Sunday, Feb. 17th,
15:00 check-in
19:00 welcome banquet

Monday, Feb. 18th,
9:00-10:30 Lecture 1 (Prof. Lorenza Mondada)
11:00-12:00 Data Session 1 (Kaoru Hayano)
Lunch
14:00-14:45 Presentation (45min.) 1 (Aug Nishizaka)
14:45-15:30 Presentation (45min.) 2 (Akira Takada)
Break (30min.)
16:00-17:00 Data Session 2 (Domenic Berducci)
17:00-18:00 Data Session 3 (Eiko Yasui)
Dinner

Tuesday, Feb. 19th,
9:00-10:30 Lecture 2 (Prof. Lorenza Mondada)
11:00-12:00 Data Session 4 (Misao Okada)
Lunch
13:30-14:10 Presentation (40min.) 3 (Satomi Kuroshima)
14:10-14:50 Presentation (40min.) 4 (Augustin Lefebvre)
14:50-15:30 Presentation (40min.) 5 (Kuniyoshi Kataoka)
Break (30min.)
16:00-16:40 Presentation (40min.) 6 (Hidehito Honda)
16:40-17:20 Presentation (40min.) 7 (Hiroaki Tanaka)
17:20-18:00 Presentation (40min.) 8 (Daiji Kimura)
Dinner

Wednesday, Feb. 20th,
9:00-10:30 Lecture 3 (Prof. Lorenza Mondada)
11:00-12:00 Data Session 5 (Adam Brandt)
Lunch
13:30-14:10 Presentation (40min.) 9 (Ikuyo Morimoto)
14:10-14:50 Presentation (40min.) 10 (Takeshi Hiramoto)
14:50-15:30 Presentation (40min.) 11 (Mayumi Bono)
Break (30min.)
16:00-16:40 Presentation (40min.) 12 (Gustav Lymer)
16:40-17:20 Presentation (40min.) 13 (Kouhei Kikuchi)
17:20-18:00 Presentation (40min.) 14 (Eric Hauser)

Overview

The workshop intends to bring together researchers in ethnomethodology, conversation analysis, workplace studies and invite them to contribute to the study of multi-activity.

Multi-activity is a pervasive feature of contemporary work spaces: people are often engaged in more than one activity at a time, manage concurrent courses of action, overhear other conversations and phone calls while working, pay attention at different events happening at the same time. These features characterize very different professional settings; they are particularly salient in computer supported work environments. People use different screens at the same time, engaging in different activities such as writing a report, writing emails, skyping, checking the news; they often phone while reading or writing at their PC; they engage in other activities while they continue to work on their computer, etc. These forms of multi-activity are spread over many types of professional settings ? just to give a few examples, in call centers, call takers speak at the phone with customers and use their computer to record and search for information, as well as for dispatching the service asked for; in doctor-patient consultations, the physician both attends the patient and uses the computer to access the patient’s file and write his report; in surgical theatres, the surgeon operates on a patient, discusses with colleagues about the last technologies or the insurance policies, while looking at a screen the endoscopic image supporting the procedure; in control rooms, professionals are constantly monitoring various screens while taking decisions and coordinating action at distance.

These phenomena raise an important set of questions: How are these activities concurrently organized? How are multiple temporalities intertwined and embedded one with the other? Which kind of shared attention, focus of attention, focal or peripherical monitoring are involved in these activities? How do participants rely on the ecology of the activity to organize their multi-activity? How are objects, artifacts, and technologies mobilized in these contexts? How do participants interact together while attending different courses of action? Which effect has multi-activity in the emergent and situated organization of talk? How are multiple participation frameworks superposed and managed at the same time? These questions concern both the organization of social action, the temporality of action, and the role of artifacts in interaction. These questions are both crucial from a theoretical perspective on human action, technology and interaction, but also from an applied perspective, since they concern a multiplicity of practical problems encountered at the workplace.

Even if there exist a very rich literature about multitasking, our understanding of the finely tuned and situated way in which people manage several courses of action at the same time ? what we call multi-activity -is still scarce. Previous studies have mainly focused on the cognitive management and stress characterizing these settings as well as on individual skills ? but less research has been carried out on the situated organization of multi-activity, as it is seen and managed moment by moment by the participants themselves and coordinated in a socially coordinated way. Moreover, previous studies are often based on experimental procedures and questionnaires or post-hoc interviews, but less work has been done on video documented naturalistic settings in which multi-activity is actually carried out.
In order to respond to the questions sketched above and to develop a conceptual and analytical approach of multi-activity that is based on the actual embodied situated practice of social actors, the workshop will draw on recent work in ethnomethodology, workplace studies, multimodal conversation analysis and video analysis. Ethnomethodology (Garfinkel, 1967) has shown the constitutive dimension of situated action and Conversation Analysis (Sacks, 1992; Schegloff, 2007) has demonstrated in the last decades the systematic organization of talk as it is orchestrates sequentially and temporally complex arrays of linguistic and embodied resources. These insights are fundamental for developing an in depth comprehension of simple actions but also for more complex courses of action. They have been consequential for the development of workplace studies, which have revealed the complexity of the ecologies characterizing professional activities ? such as control rooms in airports and underground transportation (see the seminal work of Suchman, 1996, Heath & Luff, 2000, Goodwin & Goodwin, 1996) ? showing the intricacy between situated action, the use of technologies, and the coordination at distance with other participants (often in a technologically mediated way). Last but not least, in the last couple of decades, there has been a enormous interest in multimodal interaction, generating important studies that include not only language but also gesture, gaze, bodily postures and movements, and manipulations of objects (Goodwin, 2000; Streeck, Goodwin, LeBaron, 2011) ? thus significantly enlarging our view of situated action. All of these approaches are crucially based on video recordings of situated activities in their ordinary settings.

These insights are fundamental for the understanding and in-depth description of multi-activity. The workshop aims at developing analytical tools and conceptual thoughts based on empirical investigations of video documented multi-activity settings. The focus on multi-activity in interaction includes, but is not limited to, the coordination of multi-activity in workplace environments, with a special attention to multiple activities managed by human agents and/or robots in computer-mediated interactions; overlapping activities in real space and in telecommunication space, between co-present and remote participants; interactions in face to face formations combined with the use of teleconferencing system and Skype; interactions in which the multimodal resources are concurrently and complementarily mobilized in spoken interactions, sign language conversations, and technologically mediated work.

Thus, the workshop will offer an interactional, ethnomethodological, conversation analytic perspective on multi-activity, with a special focus on work environments and on the use of computer technologies. Participants will propose in-depth analyses of video data documenting actual professional situations. The workshop will take the form of paper presentations, data sessions and methodological/theoretical discussions.

Its aim is to offer a reconsideration of theoretical frameworks for multimodality and multi-activity and to contribute to interdisciplinary attempts to understand and develop our understanding of situated action, including human communication in artificial intelligence, human-computer interaction, humanoid robotics, human-agent interaction at distance.

The workshop will produce significant analytical results contributing to our understanding of how complex activities are organized together in situ and in real time. This understanding is crucial for many fields; more specifically, in the field of technologies and informatics, these findings can significantly contribute to a) a better knowledge of users’ practices and b) an integration of this knowledge into user-oriented design of interfaces. Concerning the first point, we know that computer users are most often engaged in more than one task while using their computer. They might be involved in a side conversation or activity; they might also be involved in another activity on the screen (like chatting, having a skype conversation, doing emails and navigating on the internet). This multi-activity affects the way in which they use technologies, computers, and softwares. Concerning the second point, and building upon the first, this knowledge of multi-activities might contribute to design specific interfaces, programs, and the ergonomy of the screen in such a way that it supports these dynamics. Currently, all tasks supported by the technologies are conceived as if an individual user would be using them in an exclusive way – which is not the case: users use them in interaction, in complex ecologies of action and within a multiplicity of other parallel actions. Thus, the workshop will improve our knowledge of these very mundane but very configuring practices, producing innovative insights for user-based technology design.