Projects

The Marie Sklodowska-Curie Actions Innovative Training Network: Deictic Communication (DComm) research programme is organised into two substantial themes: Work Package 1 – Understanding Deictic Communication; and Work Package 2 Deictic Communication in Application.  Each has a cluster of individual ESR projects and, in addition, there are three integrated projects (to be led collaboratively by the ESRs) which will involve all the fellows working together to integrate various components and findings from their own research into three key areas:  gender differences, cross-cultural differences and novel hybrid applications.

UNDERSTANDING DEICTIC COMMUNICATION

ESR1 Project Title: Demonstratives across languages (WP1)
Host Institution: Universitat de les Illes Ballears (UIB), Spain
PI: Professor Pedro Guijarro-Fuentes. Key collaborators: Professor Kenny Coventry, UEA; Professor Holger Diessel, FSU.
ESR: Emanuela Todisco

Project DescriptionThis project will consider spatial demonstratives across languages using a combination of linguistic and experimental approaches to spatial demonstratives (Coventry et al., 2008, 2014) Although spatial demonstratives occur in all languages (Dixon, 2003; Diessel, 1999), demonstrative systems differ across languages in how they map onto space. This project will systematically test demonstrative use across languages in order to chart for the first time whether demonstrative systems across languages are governed by the same basic (universal) perceptual distinctions (Coventry et al., 2014; Diessel, 2014). This project will also examine the relationship between spatial and temporal uses of demonstratives, thus testing whether space and time are symmetrical or asymmetrically related in language.

Enquiries about the project may be addressed to: Professor Pedro Guijarro Fuentes, p.guijarro@uib.es

ESR2 Project Title:  Deictic language and deictic gestures in developmental deficits (WP1)
Host Institution: Norwegian University of Science and Technology (NTNU), Norway
PI: Professor Mila Vulchanova/Dr Valentin Vulchanov. Key collaborators: Dr Andrew Bayliss/Dr Martin Doherty, UEA; Professor Holger Diessel, FSU.
ESR: Sara Ramos Cabo

Project Description – This project focuses on deictic communication in neurodevelopmental disorders. It is well know that there are pragmatic deficits in Autistic Spectrum Disorders. For example, problems with some pragmatic language persist even in the presence of preserved structural language at the high end of the autistic spectrum (Tager-Flusberg et al, 2005). It is also well known that there are deficits in visual attention in neurodevelopmental disorders, but with different disorders showing varying deficit profiles (Riby et al, 2008). This project will employ a range of experimental (EEG, eye tracking) and linguistic methods to provide the first detailed examination of (possible) deficits in deictic gesture and deictic language in autism in comparison with other developmental deficits (e.g. Williams Syndrome) (Landau & Hoffman, 2012).

Enquiries about the project may be addressed to: Professor Mila Vulchanova mila.vulchanova@ntnu.no

ESR3 Project Title: Spatial language and joint attention in child development (WP1)
Host Institution: University of East Anglia (UEA), UK
PI: Dr Martin Doherty/Dr Andrew Bayliss. Key collaborators: Professor Holger Diessel, FSU; Professor Angelo Cangelosi, UOP/UOM.
ESR: Patricia González Peña

Project Description – This project will examine spatial language in typical development, and the relationship between the use of ‘demonstratives’, gesture, and theory of mind. Eye tracking will be used with infants and toddlers during simple play, employing tasks that will encourage children to identify objects being referred to and joint attention cues (e.g. checking the gaze of the speaker). This project will be the first to examine the relationship between demonstratives, perspective taking/theory of mind, and joint attention using a range of controlled and ecologically valid metacognitive tasks.

Enquiries about the project may be addressed to: Dr Martin Doherty Martin.Doherty@uea.ac.uk

ESR4 Project Title: Investigating deictic communication in stroke patients with visual neglect (WP1)
Host Institution: University of East Anglia (UEA), UK
PI: Professor Kenny Coventry/Dr Stephanie Rossit. Key collaborators: Headway, UK; Associate Professor Mikkel Wallentin, AU.
ESR: Michela Caldano

Project Description – This project will investigate deictic communication in stroke for the first time. Two thirds of stroke patients exhibit visual neglect (Stone et al, 1993). Visual neglect is the most severe and common visual processing problems observed after stroke (Stroke association) and is characterised by a loss of awareness of the side of space opposite to the side of the stroke. Although patients are not blind to that side of space, they do not perceive or attend to it. This project will for the first time examine deictic communication deficits in stroke patients, and the relationship between deictic communication abilities and visual neglect. Following screening for the presence of visual neglect and other associated neuropsychological deficits, patients will perform a range of deictic communication tasks (including the ‘memory game task’ used successfully to elicit demonstratives under controlled conditions (Coventry et al, 2008, 2014). Voxel-based lesion-symptom analysis (Rossit et al, 2011) will be used to identify which brain regions when damaged are associated with deictic communication deficits. In turn, a rehabilitation trial will establish whether deictic communication enhancement may improve visual neglect.

Enquiries about the project may be addressed to: Professor Kenny Coventry K.Coventry@uea.ac.uk

ESR5 Project Title: Spatial deixis in (diachronic) language development (WP1)
Host Institution: Friedrich-Schiller-Universität (FSU) Jena, Germany
PI: Professor Holger Diessel. Key collaborators: Professor Pedro Guijarro-Fuentes, UIB; Dr Andrew Bayliss/Dr Martin Doherty, UEA; Professor Mila Vulchanova/Dr Valentin Vulchanov, NTNU; Ordnance Survey.
ESR:  Merlijn Breunesse

Project Description – The project will investigate aspects of spatial deixis from synchronic, diachronic, and cross-linguistic perspectives. Of particular importance will be the origins and diachronic developments of spatial deictic expressions. Methodologically, the project will combine qualitative analyses of data from historical texts, etymological dictionaries, and historical grammars with quantitative analyses of (diachronic) corpus data and data from a typological database to be established in the course of the project.

Enquiries about the project may be addressed to: holger.diessel@uni-jena.de

ESR6 Project Title: The neural correlates of spatial demonstratives (WP1).
Host Institution: Aarhus University (AU), Denmark
PI: Associate Professor Mikkel Wallentin. Key collaborators: Professor Kenny Coventry, UEA.
ESR: Roberta Rocca

Project Description – This project will employ functional Magnetic Resonance Imaging (fMRI) to elucidate the mapping between spatial demonstratives and non-linguistic brain regions involved in the perception of space; specifically peripersonal (near space) versus extrapersonal (far) space (Kemmerer, 1999; Làdavas, 2002; Longo & Lourenco, 2006). Spatial demonstratives will be used as auditory stimuli, presented both within a larger linguistic context (e.g. in stories) and in isolation in a standard forced choice response paradigm. These paradigms will allow for investigation of the differences between demonstratives in different contextual and attentional settings. It will be established if there is a close neurological mapping between perceptual space (Kemmerer, 1999; Làdavas, 2002; Longo, & Lourenco, 2006; Lane et al, 2013) and demonstratives.

Enquiries about the project may be addressed to: Associate Professor Mikkel Wallentin
mikkel@cc.au.dk

ESR7 Project Title: Deictic communication in sign languages (WP1).
Host Institution: ISTC Consiglio Nazionale Delie Ricerche (CNR), Italy
PI: Dr Olga Capirci/Dr Cristina Caselli. Key collaborators: QUALISYS; Professor Holger Diessel, FSU.
ESR: Anita Slonimska

Project Description – This project will focus on deictic communication in sign languages – languages where action is privileged, but also provides key grammatical functions. Drawing on previous work on Sign Languages (SL) and Spoken-Vocal Languages (VL), and on new evidence on the development of deictic gesture and words for demonstrative versus person reference, the project will explore typological, modality-specific features affecting deixis and anaphora in SL. Deictic-anaphoric reference is produced in SL via complex manual and non-manual units, which exhibit highly iconic features and are marked by specific eye-gaze patterns which distinguish them from standard signs. Signed data will be analysed also referring to co-speech deictic gestures and to other deictic devices (e.g. demonstratives), in order to analyze bimodal bilingual (spoken/signed language) patterns of both adults and children.

Enquiries about the project may be addressed to: Olga Capirci, Senior Researcher, ISTC – CNR, Rome olgacapi@gmail.com

DEICTIC COMMUNICATION IN APPLICATION

ESR8 Project Title: Developmental robotics architecture for the co-development of demonstratives and gestures in human-robot cooperation (WP2)
Host Institution: Plymouth University, UK; University of Manchester, UK
PI: Professor Angelo Cangelosi/Dr Anthony Morse. Key collaborators: Dr Martin Doherty/Dr Andrew Bayliss, UEA; Telerobot; IIT.
ESR:  Baris Serhan

Project Description – This project will target the important issue of the robot’s understanding of function words, including demonstratives, to go beyond current state-of-the-art learning and understanding of words naming objects and actions in robots. The interaction between robots and humans, as in cooperation on joint object manipulation, requires the robot’s understanding of sentences such as “Pass me that ball” and “Put that yellow block there.” The cognitive architecture will extend the “ERA” architecture developed by Morse et al. (2010) used in HRI experiments on word learning for object and action names, with the addition of a recurrent module for the processing of action sequences and gestures. The extended architecture will focus on the development of robotics architecture for deictic communication, focusing on demonstratives. This work aims to be developmentally inspired, taking what is known about the developmental trajectory of gesture and language to impact upon robot architecture and learning. The extended cognitive architecture will provide the first cognitive robotic model for the understanding of function words and their grounding in sensorimotor strategies and social interaction, and its application to human-robot interaction/cooperation across a range of settings (e.g. robot companions for older adults).

Enquiries about the project may be addressed to: Professor Angelo Cangelosi angelo.cangelosi@manchester.ac.uk

ESR9 Project Title: From single words to compositional language via gestures: Applications in robot language learning (WP2)
Host Institution: Plymouth University, UK; University of Manchester, UK
PI: Professor Angelo Cangelosi. Key collaborators: Dr Olga Capirci/Dr Cristina Caselli, CNR; Telerobot; IIT.
ESR: Gabriella Pizzuto

Project Description – Studies on sign language and language-gesture development have shown that children go through a gesture-word combination stage before they make full transition to two-word and longer sentences (Capirci et al, 1996). This project will follow the language and action integration roadmap proposed in Cangelosi et al. (2010) on the co-development and sharing of compositional representations common to language and action (Pastra & Aloimonos, 2011) for extended language learning systems in human-robot cooperation tasks. The neural control architecture will integrate both action and language learning through integrated sensorimotor/linguistic areas, using interaction scenarios requiring compositional and recursive actions, and the learning of both words and gestures to describe action-object combination related to the task execution. Analyses of the neural controller’s shared action/language layers will shed light on the nature of shared, compositional representation bootstrapping both motor and linguistic capabilities.

Enquiries about the project may be addressed to: Professor Angelo Cangelosi angelo.cangelosi@manchester.ac.uk

ESR10 Project Title: Deictic communication and mobile phones (WP2)
Host Institution: University of Münster, Germany
PI: Dr Cristian Kray. Key collaborators: Professor Christoph Hölscher/Professor Martin Raubal, ETH; 52°North .
ESR: Samuel Navas Medrano

Project Description – This project will design, implement, and evaluate techniques to enable deictic communication between non-collocated human communication partners via mobile phones. The qualities of technologies produced will be compared to collocated deictic communication that is not technology-mediated. In order to achieve the objectives, a participatory design approach will be adopted employing design interaction elicitation techniques with potential users. The findings will be implemented using rapid prototyping techniques and agile software engineering methods, and will subsequently be evaluated through lab-based, controlled user studies as well as field studies. Among the outputs will be the production of open-source software, which will enable others to realize technology-mediated deictic communication across a range of platforms.

Enquiries about the project may be addressed to: Dr Christian Kray c.kray@uni-muenster.de

ESR11 Project Title: Improved motion capture methodology and tools in Linguistic research (WP2)
Host Institution: Qualisys, Sweden
PI: Mr Fredrik Muller. Key collaborators: Dr Olga Capirci/Dr Cristina Caselli, CNR; Dr Stephanie Rossit/Professor Kenny Coventry, UEA; Professor Mila Vulchanova, NTNU.
ESR: Mehmet Aydin Baytaş

Project Description – The goal of this project will be to focus on motion tracking technology development and integration with respect to communication. Many of the above projects tap how language and gesture are coordinated in deictic communication across populations. This project will develop motion-tracking integration for communication settings. This will involve the optimization of marker sets for motion tracking, the coordination of these sets with speech streams, and the associated algorithmic improvements in the motion capture tool chain required.

Enquiries about the project may be addressed to: Fredrik Müller Fredrik.muller@qualisys.se

ESR12: iCub robot hand redesign for gestural and deictic interaction (WP2)
Host Institution:  Telerobot,
PI: Dr Francesco Becchi, Telerobot. Key collaborators: Professor Angelo Cangelosi, UOP/UOM; IIT; Dr Olga Capirci/Dr Cristina Caselli., CNR; Associate Professor Mikkel Wallentin, AU).
ESR: Raed Bsili

Project Description – In order for robots to be able to use deictic communication effectively, as well as a full range of gestures, it is necessary to design a robotic hand that can fulfil the full gestural vocabulary required. To that end, this project will redesign the iCub robot hand, working closely with the other sites using the iCub (UOP) and informed by findings regarding gestural vocabulary investigated elsewhere (e.g. CNR).  Critically, the new hand will afford greater potential for naturalistic human-robot interaction – a goal that will afford transferability to other technologies and situations.

Enquiries about the project may be addressed to: Dr Giovanni Stellin stellin@telerobotlabs.it

ESR13 Project Title: Deictic communication in architectural and urban design (WP2)
Host Institution: Eidgenössische Technische Hochschule (ETH) Switzerland
PI: Professor Christoph Hölscher/Dr Martin Brösamle. Key collaborators: KCAP; Dr Christian Kray, WWU.
ESR: Yesol Park

Project Description – The spatial arrangements in architectural design tasks are too detailed and nuanced to be reliably communicated by verbal descriptions alone. Thus, the dialogue between designers and with clients is based on deictic gestures and verbal references to objects such as plans, 3d models and ad-hoc sketches. This project will characterize and classify the deictic references occurring in different stages of the design process. The project will also distinguish between novice and senior designers with respect to their deictic reference strategies and their ability to adjust such behaviours to different communication partners.

Enquiries about the project may be addressed to: Professor Christoph Hölscher christoph.hoelscher@gess.ethz.ch