• View

Modeling and Analysis of Instructional Process (MANIP)

Description

While teaching, teachers maintain their attention on a multitude of signals from students to infer how their lesson is going on: smiles, gaze, postures, head nodding, etc. Teachers also emit some of these signals for communication or supervision purposes. Classically, researchers capture these signals through direct observation to infer the quality of the teacher-student relationships. Teachers and supervisors commonly use these signals to draw a reliable idea of teaching quality to scaffold reflexivity or training. Like any knowledge-focused profession, teaching is complex to investigate as it involves layers of events not entirely and immediately visible to observers: relationships, student behavior, mental processes, attention, etc. So far, these (partly) hidden layers are investigated through self-report based methods, direct observation or human–computer interactions with weak reliability.

According to an ecological psychology-inspired framework, attendees (both teachers and learners) are seen as coupled in a specific classroom, the Context-Aware Classroom, (CAC, a.k.a., smart classroom), i.e. a technology-enhanced classroom that proposes resources and instrumentation to both support and understand learning. In a CAC, teachers and students are approaching a variety of resources, be they human or material, enabling different actions and interactions with multiple participants. Materials and humans are resources people orient their attention to, in order to co-construct the meaning to be taught, in sort of multiple perception–action loops: attention directed toward resources shape perception and action which, in turn, can help to transform the resources, making goals and knowledge emerge. As social learning episodes have a multimodal dimension, they are triggered and maintained by gaze and postural signals (shared attention processes, deictics), other human communication modalities (speech, emotions, etc.) and further factors. This justifies resorting to multiple types of data appearing during their progress (gazes’ scan path, classroom noise volume, classroom ambiance, etc.) as they are now perceivable using ubiquitous computing. CACs have also recently become a focus of interest given the current context of the Covid-19 pandemic. Indeed, fostered by containment and health-protocol, distant and different levels of hybrid teaching have been set up in emergency to ensure pedagogical continuity. But this shift in the university paradigm has been done without preliminary studies permitting to ensure its effectiveness. CACs can thus be used as tools sustaining research on teaching and learning analytics in different learning situations from face-to-face to distant (the teacher is in the smart classroom, all students attend remotely) with several possible hybridization degrees.

Over the past two decades, multimodal signal processing and machine learning research were seldom done in real-world classrooms while in education, research on CACs increased. On the one hand, little theoretical research on the use of these classrooms has been undertaken: they were considered as purely technology-driven solutionist “show-rooms” supporting actions like students’ presence scanning. So, their attractiveness shifted down, worsened by privacy concerns. On the other hand, computer scientists have poorly studied CACs, like other “in the wild” contexts, even if research on perceptive spaces is an established research field. In many perception tasks, recent advances of deep neural networks can sustain analyses of instructional events. Fine-grained pedagogical events, like gaze, object or person pointing, emotions, could be recorded and automatically labeled, promoting CAC usage towards multimodal analyses. Nevertheless, while deep learning and statistical methods still progress on state-of-the-art corpora, porting models in real conditions remains a research challenge. This challenge comes from the diversity in recorded variable types (from qualitative state processes to time/spatial continuous data), from the large sources of variability (teacher, students, class, time and remote effects and real-world recording conditions) and from the different time scales of underlying features.

The MANIP project-team focuses on computational social sciences for teaching and learning analytics, at the confluence of mathematics and statistics, signal processing, machine learning and social sciences. With an Ethics by design approach, we aim to improve the perception of classroom interactions between humans on the one hand, and the models and theories developed in the social sciences for education on the other hand. This leads to three major research questions:

• Q1: How to model and analyze the different (hidden) layers of genuine classroom events?
• Q2: How would a CAC, equipped with unobtrusive multimodal and multilevel perception capabilities, help gather information on these (hidden) layers?
• Q3: How do privacy and ethical concerns about personal data influence analysis capabilities and performances of a CAC?

The scientific approach developed by the MANIP project-team will be interdisciplinary, integrating the complementary work of several teams. The Multimodal Perception and Sociable Interaction team of the LIG laboratory (M-PSI1) enforces the project-team on machine learning for multimodal perception in pervasive space (computer vision, speech processing, eye-tracking. . . ) and trace analysis. The Statistique pour les Sciences du Vivant et de l’Homme (SVH) team of the LJK laboratory develops statistics (mixed effect models, variable selection...) and trace-based measures of human behavior, with applications to didactics and educational psychology. LaRAC (Laboratoire de recherche sur les apprentissages en contexte) members contribute to the project-team with knowledge and experience in conversational analysis, usability engineering, and instructional events analysis. The expected outcomes include perceptive computational models describing the interactions between teachers and students within a CAC. These models will allow enhancing both teachers’ pedagogical practices and students’ learning. Our scientific approach will follow a “reproducible science” paradigm: a) we will share source codes and models (Open Source); b) after obfuscation/pseudonymization, corpora will be spread to the scientific community (Open Data).

Locally, the project-team will benefit from the Teaching Lab project funded by the Idex Formation call. This project permitted us to build a fully equipped and functional context-aware classroom at the ENSIMAG building. Some members of the project-team are involved on the PIA3 PEGASE project which focuses on primary and secondary schools. MANIP is complementary to PEGASE as it addresses the seldom studied university level. From a broader point of view, MANIP will sustain current starting collaborations with the ICAR (ENS-Lyon) and Techné (Univ. Poitiers) laboratories.