Free CSS Website Template
Multimodal dialog system supports human-computer interaction over multiple different input and/or output modes. Advantages of multimodal dialog system are:
We are developing a simulation GUI tool for the smart home control task. By using a graphic simulator, users can test the system which can perform the functions related to the smart home control such as “turning on the light”, “closing the curtain”, “locking the door” and so on. Users can interact with computer by using both speech and pen-gesture concurrently.
Referring Expression Resolution is the task that extracts possible REs in sentences and matches each referring expression to corresponding objects. We Introduces I-RE structure to symbolize DAG structure for each modifier and modificant relations.
There are two kinds of inputs in this system. Visible Input is supplied to object matcher and used to construct object pool. Objects are described in an attribute-value form and are prepared for further RE process. When sentence is given to the system, the system makes a trial to generate proper response for user intention meanwhile it also try to resolve REs in sentence. Those two subsystem communicate with each other.
I-RE : I-RE is a Directed Acyclic Graph structure which symbolize dependencies between each modifier and its modificant. These dependencies are directly extracted by dependency parsers and other word feature extractors.