Waseda University ICT and Robotics

Top Global University Project: Waseda Goes Global -A Plan to Build a Worldwide Academic Network
that is Open, Dynamic and Diverse

Waseda University


Professor Yiannis Aloimonos

The University of Maryland, USA

MIND DESIGN: The theory of action grammars

Actions (what an intelligent autonomous system does) are the fundamental building blocks of the mind of the system. Actions however reside in different spaces, the visual space (actions seen), the auditory space (actions heard), the sensori-motor space (actions performed) and the language space (actions talked about). To achieve intelligence we must be able to map the different spaces to each other. The grammatical structure of these spaces point to a new approach. Taking advantage of the syntax of action, we model action as a formal system, namely as a program in a special language named AL (Action Language). We focus on manipulation actions and we conceive such mappings as compilers, interpreters or translators that take action descriptions in one space and turn them into the equivalent description in another space.

In this talk, I will outline a system called VALC (VALC: Visual AL Compiler) that will automatically translate visual observations of complex manipulation actions to AL programs. I will also outline MALC (MALC: Motor AL Compiler) that will automatically translate programs in AL to a motor execution plan for any robot whose specification is provided. MALC will also allow new ways of approaching reinforcement learning and provide a universal language for cognitive robot programming. Furthermore, I will introduce VALD (Visual AL Debugger), an Augmented Reality system, which will guide a human through the course of an AL program execution, providing visual instructions and feedback upon program violations. Finally, I will briefly describe VALT (Video Action Language Translator), a multimedia system that will produce semantic descriptions (and descriptions in English) of videos containing manipulation actions.


Yiannis Aloimonos is Professor of Computational Vision and Intelligence at the Department of Computer Science, University of Maryland, College Park, and the Director of the Computer Vision Laboratory at the Institute for Advanced Computer Studies (UMIACS). He is also affiliated with the Institute for Systems Research and the Neural and Cognitive Science Program. He was born in Sparta, Greece and studied Mathematics in Athens and Computer Science at the University of Rochester, NY (PhD 1990). He is interested in Active Perception and the modeling of vision as an active, dynamic process for real time robotic systems. For the past five years he has been working on bridging signals and symbols, specifically on the relationship of vision to reasoning, action and language.