MIND DESIGN: The theory of action grammars

Actions (what an intelligent autonomous system does) are the fundamental building blocks of the mind of the system. Actions however reside in different spaces, the visual space (actions seen), the auditory space (actions heard), the sensori-motor space (actions performed) and the language space (actions talked about). To achieve intelligence we must be able to map the different spaces to each other. The grammatical structure of these spaces point to a new approach. Taking advantage of the syntax of action, we model action as a formal system, namely as a program in a special language named AL (Action Language). We focus on manipulation actions and we conceive such mappings as compilers, interpreters or translators that take action descriptions in one space and turn them into the equivalent description in another space.

In this talk, I will outline a system called VALC (VALC: Visual AL Compiler) that will automatically translate visual observations of complex manipulation actions to AL programs. I will also outline MALC (MALC: Motor AL Compiler) that will automatically translate programs in AL to a motor execution plan for any robot whose specification is provided. MALC will also allow new ways of approaching reinforcement learning and provide a universal language for cognitive robot programming. Furthermore, I will introduce VALD (Visual AL Debugger), an Augmented Reality system, which will guide a human through the course of an AL program execution, providing visual instructions and feedback upon program violations. Finally, I will briefly describe VALT (Video Action Language Translator), a multimedia system that will produce semantic descriptions (and descriptions in English) of videos containing manipulation actions.