Faculty of Science and Engineering, Soka University, Tokyo, Japan
Service robots need to identify users in order to provide personalized services based on prior interactions and preferences. In real environments, however, only a small amount of learning data may be available while appearance, environmental noise, and utterance content change over time.
We propose a multimodal person concept model that integrates visual, acoustic, and linguistic features extracted with CLIP and CLAP. A nonparametric Bayesian generative model based on the Chinese restaurant process forms person concepts without requiring person labels or the number of users in advance. The model was evaluated with six participants under eight combinations of visual, acoustic, and linguistic variation, followed by analyses of feature dimensionality, modality availability, and variation strength, and a qualitative demonstration on a Toyota Human Support Robot (HSR).
Identification accuracy with all modalities under the stable observation condition.
Accuracy remained above chance even when visual, acoustic, and linguistic observations all varied.
With four-dimensional PCA features, the model recovered the six participants in 6 of 10 trials with training ARI 0.961.
Formation-time ablation recovered six concepts in every trial with training ARI 1.00.
A qualitative proof of concept using a Toyota Human Support Robot.
Scope: this is a qualitative demonstration, not a quantitative on-robot evaluation. The online demo uses the original 512-dimensional features without PCA and can therefore over-segment the observations.
The study was conducted as an undergraduate research project under the supervision of the authors' academic advisor. According to guidance provided by the Human Research Ethics Committee of Soka University, ethics review was not required for this study under the university's procedures for undergraduate research. All participants gave informed consent to the recording and to the use of the recordings for research.
The facial images reproduced in the dataset table in the paper and the person shown in Fig.1 and Fig. 4 are of the first author only. The faces of any other participants appearing in the demonstration video are obscured by mosaic processing. Example utterances were selected so that they contain no information identifying a participant.