Multimodal Person Concept Model for Person Identification under Varying Observation Conditions

Mizuho Okitani  ·  Yoshinobu Hagiwara

Faculty of Science and Engineering, Soka University, Tokyo, Japan

Abstract

Service robots need to identify users in order to provide personalized services based on prior interactions and preferences. In real environments, however, only a small amount of learning data may be available while appearance, environmental noise, and utterance content change over time.

We propose a multimodal person concept model that integrates visual, acoustic, and linguistic features extracted with CLIP and CLAP. A nonparametric Bayesian generative model based on the Chinese restaurant process forms person concepts without requiring person labels or the number of users in advance. The model was evaluated with six participants under eight combinations of visual, acoustic, and linguistic variation, followed by analyses of feature dimensionality, modality availability, and variation strength, and a qualitative demonstration on a Toyota Human Support Robot (HSR).

Paper in brief

  • Learns person concepts from limited multimodal observations without person labels.
  • Infers the number of person concepts rather than fixing it in advance.
  • Tests robustness to appearance changes, acoustic noise, and linguistic variation.
  • Demonstrates access to a learned concept from a single visual or acoustic observation and recall of associated linguistic information on the HSR.

Method overview

Observation acquisition → feature extraction → person concept formation → identification.

Overview of the proposed multimodal person concept method
Fig. 1. Overview of the proposed method.

Key results

0.98

Stable condition

Identification accuracy with all modalities under the stable observation condition.

0.76

All modalities varied

Accuracy remained above chance even when visual, acoustic, and linguistic observations all varied.

6/10 trials

Recovered six concepts

With four-dimensional PCA features, the model recovered the six participants in 6 of 10 trials with training ARI 0.961.

10/10 trials

Visual + Acoustic

Formation-time ablation recovered six concepts in every trial with training ARI 1.00.

Concept formation and identification are not the same objective. Without dimensionality reduction, the model did not recover six concepts in any trial and training ARI fell to 0.301, yet identification accuracy under all-modality variation remained 0.68. This suggests that exact recovery of the number of concepts is not always necessary for person identification, although it remains important when evaluating the quality of the latent concept structure.

Demonstration on a domestic service robot

A qualitative proof of concept using a Toyota Human Support Robot.

Demo video supplied with this project page. The faces of participants other than the first author are obscured by mosaic processing. The HSR acquires multimodal observations, accesses a person concept from a single visual or acoustic observation, retrieves associated linguistic information, and generates a person-dependent response.
Proof-of-concept HSR demonstration
Fig. 4. Proof-of-concept demonstration on the HSR. Demo 1 (top) uses only a visual observation from the HSR camera, and Demo 2 (bottom) only an acoustic observation. In each case, the observation is passed to the person concept model, the concept that maximizes the log-likelihood of the available modality is selected, and the linguistic information stored with the selected concept is recalled to generate the robot response. Both modalities accessed the same concept (Concept 1, highlighted), from which the stored utterance "I like Japanese food." was retrieved. The person shown in the figure is the first author.

What the robot demo shows

  • The HSR collects visual, acoustic, and linguistic observations during dialogue.
  • CLIP and CLAP features are used to form person concepts.
  • Demo 1 accesses a concept using only the current visual observation.
  • Demo 2 accesses a concept using only an acoustic observation.
  • After concept selection, linguistic information associated with that concept is recalled and used in the robot response.

Scope: this is a qualitative demonstration, not a quantitative on-robot evaluation. The online demo uses the original 512-dimensional features without PCA and can therefore over-segment the observations.

Ethical considerations

The study was conducted as an undergraduate research project under the supervision of the authors' academic advisor. According to guidance provided by the Human Research Ethics Committee of Soka University, ethics review was not required for this study under the university's procedures for undergraduate research. All participants gave informed consent to the recording and to the use of the recordings for research.

The facial images reproduced in the dataset table in the paper and the person shown in Fig.1 and Fig. 4 are of the first author only. The faces of any other participants appearing in the demonstration video are obscured by mosaic processing. Example utterances were selected so that they contain no information identifying a participant.