Cooking is a big part of our daily life. It requires skills that take years of training and practice to master. Preparing a fine meal also takes time. Many of us spend several hours a day preparing meals for ourselves and our families. But it is worth the effort since the quality of the meals directly affects our physical and mental health.
If robots could reliably cook high-quality meals in home kitchens, it would dramatically improve the quality of our life. People who own a cooking robot will enjoy meals that are as good as the dishes from a Michelin 3-star restaurant at home without hiring an expensive chef or spending years in training.
To achieve this, RPAL has analyzed and identified several technical challenges in building a robotic cooking system and developed a roadmap to solve those challenges. The figure below illustrated the roadmap.
The roadmap has the following main components:
- Learning and representing cooking knowledge for robots. It includes cooking knowledge extraction from recipes, pictures, and videos, knowledge graph representation, motion embedding, and graph search retrieval.
- Learning and developing skills for robots to use kitchen tools and utensils. It includes task-oriented grasp planning and self-supervised manipulation skill learning and generalization.
- Observing and understanding ingredient state change in the cooking process. It includes their shapes, sizes, texture, color, firmness, viscosity, etc.
Learning and representing cooking knowledge for robots. Robots need to gain an understanding of cooking. Our approach is to develop a learning approach that extracts cooking knowledge from recipes and cooking instructional videos. The obtained knowledge is then represented in a functional object-oriented network (FOON). We develop new tools for video understanding, knowledge representation, knowledge embedding, knowledge retrieval in the areas of deep learning, computer vision, and robotics. For more details, please read more here.
The following figure shows the current cooking knowledge FOON learned from 100 cooking instructional videos.
A zoomed-in view of the FOON is in the following figure.
Learning and developing skills for robots to use kitchen tools and utensils. For a cooking task, a robot would need to grasp utensils and manipulate it properly. For example, to a perfect dough, a robot would need to grasp and hold a cup of flour and pour it into a bowl, and grasp a cup of water and pour it into the bowl as well. Then the robot would grasp a whisk to mix them well. Grasping the cups and whisk requires grasp planning so that they are properly grasped and do not fall off during the task. Pouring water and flour should be well planned and controlled for the correction portion and ratio. Mixing motion should dynamically adapt to the state of the mixture and lumps. For grasping utensils and tools, our approach is to develop a task-oriented grasp planning that takes the desired force and torque in the manipulation task into consideration. For more details, please read more here. For manipulation, our approach is to develop a self-supervised learning and generalization approach that learns from human demonstrations and the actual outcomes. Then it generalizes the skill to a variety of conditions through self-supervised practicing. For more details, please read more here.
The following figures show our grasp planning results that learn and mimic human thumb placements, which indicates the task requirements.
The following figure shows an example recording frame of a human cooking manipulation demonstration.
The left of the following figures shows a robot performs accurate water pouring using a bottle that is not in the learning set. The middle figure shows the cups used in training to learn pouring skills. The right figure shows the cups used in the evaluation of the learned pouring skill. The evaluation results show the robot pouring skill is on par with the human pouring expertise in terms of accuracy and speed.
Observing and understanding ingredient state change in the cooking process. The states of the ingredients are critical throughout the cooking process. When the ingredients are raw, their shapes, sizes, texture, color, firmness will decide if they need to be trimmed, washed, or even thrown away. Cooking without knowing their initial states is prone to food poisoning. When processing the ingredient, their shapes, size, texture, color, firmness should be continuously watched and used to judge the readiness of the cooking processes and the necessity of an adjustment. For example, in mixing, if a wrong ratio is detected during mixing, more flour or water should be added.
The following figures show ingredients in their different states.
The following tags links to our work and progress in building a robotic cooking system.
Robotic Cooking Setup --
Cooking Knowledge for Robot --
Grasping Utensils for Cooking --
Cooking Manipulation Skills --
Perception for Cooking.