Abstract: We are working on collecting a speech-based Grounded Language Learning dataset that combines visual scenarios of a robot doing tasks and natural language utterances. The visual data will be from a simulated environment. The language data will include low-level commands, high level descriptions of the robotic task and contextual explanations of the human intention behind the robot's actions. We intend to collect this data in the form of audio recordings through Amazon Mechanical Turk (AMT).

Abstract: Learning to understand grounded language, which connects natural language to percepts, is a critical research area. Prior work in grounded language acquisition has focused primarily on textual inputs. In this work we demonstrate the feasibility of performing grounded language acquisition on paired visual percepts and raw speech inputs. This will allow interactions in which language about novel tasks and environments is learned from end users, reducing dependence on textual inputs and potentially mitigating the effects of demographic bias found in widely available speech recognition systems. We leverage recent work in self-supervised speech representation models and show that learned representations of speech can make language grounding systems more inclusive towards specific groups while maintaining or even increasing general performance.

Abstract: Grounded language acquisition is a major area of research combining aspects of natural language processing, computer vision, and signal processing, compounded by domain issues requiring sample efficiency and other deployment constraints. In this work, we present a multimodal dataset of RGB+depth objects with spoken as well as textual descriptions. We analyze the differences between the two types of descriptive language and our experiments demonstrate that the different modalities affect learning. This will enable researchers studying the intersection of robotics, NLP, and HCI to better investigate how the multiple modalities of image, depth, text, speech, and transcription interact, as well as how differences in the vernacular of these modalities impact results.

Abstract: We propose a cross-modality manifold alignment procedure that leverages triplet loss to jointly learn consistent, multi-modal embeddings of language-based concepts of real-world items. Our approach learns these embeddings by sampling triples of anchor, positive, and negative data points from RGB-depth images and their natural language descriptions. We show that our approach can benefit from, but does not require, post-processing steps such as Procrustes analysis, in contrast to some of our baselines which require it for reasonable performance. We demonstrate the effectiveness of our approach on two datasets commonly used to develop robotic-based grounded language learning systems, where our approach outperforms four baselines, including a state-of-the-art approach, across five evaluation metrics.

Abstract: Pre-training language representation models on large-scale corpora has led to state-of-the-art performance in multiple NLP tasks. These models are particularly efficient at transferring the knowledge acquired from their pre-training objectives to other tasks. In this paper, we apply one of these models in BERT, to the context of event modeling. We show that pre-training BERT with its Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) objectives on a large amount of structured event documents can lead to state-of-the-art performance in event modeling tasks.

Abstract: The complex nature of emotions makes it also one of the hardest text classification tasks. We present a comparative study of state of the art language representation models XLNet and Bert in sentiment analysis, specifically multi-label classification of tweets among 6 basic emotions (anger, disgust, fear, joy, sadness and surprise).