Current Research (3)

Semiotics of Human Movements


Artificial intelligence study at YNL originated from mathematical motivation to investigate the fundamental dynamics of intelligence. The modeling of nonlinear dynamics as attractors of humanoid and robot motions was studied by Prof. Masafumi Okada (Tokyo Institute of Technology), Koji Tatani (Mitsubishi Heavy Industries), Prof. Akinori Sekiguchi (Tokyo University of Technology), Dr. Hideki Kadone (Tsukuba University) and many other students.

Statistical modeling of human motion stared by Hideaki Esaki’s MS dissertation using Hidden Markov Model (HMM). Prof. Tetsunari Inamura (National Institute of Informatics) focused on the bidirectional generative and discriminative property of HMM computation and proposed to use HMM for functional modeling of the mirror neurons.  He introduced  a norm into the set of motion symbols described by HMMs and created the spatial representation of motion symbols. The norm was defined by using the Kullback-Leibler divergence.

Prof. Wataru Takano (University of Tokyo) in his Ph.D work developed the theory of symbolic communication inspired by the mirror neurons using HMM. Prof. Takano demonstrated realtime symbolic communication between a humanoid robot and a human at Aichi EXPO in 2005, where UT-mu2:magnam (a 60cm-tall humanoid robot) identified the category of motion of human by using the symbol system of HMMs and generated a motion of the responsive category in realtime. The human motion was measured by motion capture. The communication task was to fight like boxing. Prof. Dana Kulic (University of Waterloo, Canada) developed autonomous segmentation of human motions and unsupervised learning and categorization. Prof. Dongheui Lee (Technical University of Munich, Germany) in her Ph.D study worked on category identification and motion generation from the partial information. Then, Prof. Lee started the study of Physical Human-Robot Interaction (pHRI) where an associated motion form the symbol system is modified in time and space in order to meet the physical and contextual requirements in the real world.

The figures show the recent results of study by Prof. Takano. A picture of black background shows “Cristal Ball” which estimates future motion based on the database of the sequence of motion categories and the the measured sequence of motion categories. A white man is the current measured motion and the other men in the left are the estimation of his future motions. As seen in the middle of the figure, many motion segments of daily life are categorized and stored as the motion symbols of HMM with the text data attached with them in the cloud computing system.  A corpus of language is also stored to enhance the text symbol space.  Ikuo Kusajima (a Ph.D candidate) recently included a Japanese corpus of the Aozora Bunko (the library of copyright-expired books). The figures of humans with green bones and red joints in the right and the top show the sentences generated from the motions as the results of statistical computation. The photograph in the left shows the experiments using KINECT. The sentences are generated not only from the human motion, but also from the identified object (a box in this case).

A current research interest at YNL is in deep connection of the motion symbols and the text symbols, namely words and sentences. The observed human motion makes projections onto the motion symbol space and then onto the text symbol space. The real depth of human intelligence does not stop there and makes further projections onto both symbol spaces like echo and reecho. The echo of symbols causes resonance with the other sensory sensation from the real world. Being weakened and intensified some coherent echo may arise. This is our model of the symbolic intelligence of human.