Multimodal Dialog Management
- We have been developing Multimodal Dialog System in multimodal interactive environment. User interactive appliances (smartphone, smart home control etc.) allow users to communicate with systems not only by fixed interface such as button or controller, but also by multimodal IO devices such as microphones, haptic sensors. In order to develop multimodal dialog system, we have also developed techniques to deal with Referring Expressions (REs) for indicating objects in given environments.
- Multimodal Dialog System
- Natural, easy-to-use system
- High task performance and user preference
- Easy error recovery and handling
- etc.
- Smart Home Control using a Multimodal Dialog System
- Referring Expression Resolution
- Architecture
- Demo Video
- You can download and play the demo video of our multimodal dialog toolkit (D3Toolkit) from here
- You can download and play the demo video of our restaurant finder (Resfinder) spoken dialog system from here
- You can download and play the demo video of our smart TV guide spoken dialog system from here
- Contact
- Sangjun Koo
Multimodal dialog system supports human-computer interaction over multiple different input and/or output modes. Advantages of multimodal dialog system are:
We are developing a simulation GUI tool for the smart home control task. By using a graphic simulator, users can test the system which can perform the functions related to the smart home control such as “turning on the light”, “closing the curtain”, “locking the door” and so on. Users can interact with computer by using both speech and pen-gesture concurrently.
Referring Expression Resolution is the task that extracts possible REs in sentences and matches each referring expression to corresponding objects. We Introduces I-RE structure to symbolize DAG structure for each modifier and modificant relations.
There are two kinds of inputs in this system. Visible Input is supplied to object matcher and used to construct object pool. Objects are described in an attribute-value form and are prepared for further RE process. When sentence is given to the system, the system makes a trial to generate proper response for user intention meanwhile it also try to resolve REs in sentence. Those two subsystem communicate with each other.
I-RE : I-RE is a Directed Acyclic Graph structure which symbolize dependencies between each modifier and its modificant. These dependencies are directly extracted by dependency parsers and other word feature extractors.
