Daily Interactive Manipulation (DIM) Dataset


Robots that succeed in factories stumble to complete the simplest daily task humans take for granted, for the change of environment makes the task exceedingly difficult. Aiming to teach robots perform daily interactive manipulation in a changing environment using human demonstrations, we collected our own data of interactive manipulation. The dataset focuses on position, orientation, force, and torque of objects manipulated in daily tasks. The dataset includes 1,603 trials of 32 types of daily motions and 1,751 trials of pouring alone, as well as helper code. We present our dataset to facilitate the research on task-oriented interactive manipulation.

We present a dataset of daily interactive manipulation. Specifically, we record daily performed fine motion in which an object is manipulated to interact with another object. We refer to the person who executes the motion as subject, the manipulated object as tool, and the interactive object as object. We focus on recording the motion of the tool. In some cases, we also record the motion of the object.

The dataset consists of two parts. The first part contains 1,603 trials that cover 32 types of motions. We choose fine motions that people commonly perform in daily life which involve interaction with a variety of objects. Different subsets of the motions are found in multiple different motion-related datasets. The motions we collect include those that are most frequently executed in cooking scenarios except that we do not include pick-and-place because it barely involve change of orientation.

The second part contains the pouring motion alone. We collect it to help with motion generalization to different environments. We chose pouring because 1) pouring is found to be the second frequently executed motion in cooking, right after pick-and-place and 2) we can vary the environment setup of the pouring motion easily by switching different materials, cups, and containers. The pouring data contains 1,751 trials of pouring 3 materials from 6 cups into 10 containers. We collect the two parts of the data using the same system.

The dataset provides position and orientation (PO) with 100% coverage, and force and torque (FT) with 100% coverage, and vision with 50% coverage. The less-than-perfect coverage of vision results from filming restrictions.

To cite this dataset:

Huang, Y. and Sun, Y. (2019), A Dataset of Daily Interactive Manipulation, International Journal of Robotics Research (IJRR), 38(8): 879-886. [pdf]

  title={A dataset of daily interactive manipulation},
  author={Huang, Yongqiang and Sun, Yu},
  journal={The International Journal of Robotics Research},
  volume = 38,
  number = 8,
  publisher={SAGE Publications Sage UK: London, England}

Daily Manipulation Data

Helper Code
Motion Code Motion Name Total PO FT Vision Download
m2 stir with spatula 25 25 25 25 m2
m3 sprinkle, shake pepper 40 40 40 40 m3
m4 spread/oil 25 25 25 25 m4
m6 vertical cut 25 25 25 25 m6
m7 use spoon to pick up 98 98 98 35 m7
m8 pizza wheel 25 25 25 25 m8
m10 use black brush 25 25 25 25 m10
m11 spear object using fork 30 30 30 30 m11
m12 stir water using spoon 25 25 25 25 m12
m13 fasten screw with screwdriver 40 40 40 40 m13
m14 loosen screw with screwdriver 35 35 35 m14
m15 unlock lock with key 165 165 165 75 m15
m16 fasten nut with wrench 40 40 40 15 m16
m17 use paint brush to dip and spread 25 25 25 25 m17
m18 use hammer to hammer in nail 25 25 25 25 m18
m19 brush teeth 50 50 50 25 m19
m20 use file to file wooden thing 125 125 125 25 m20
m21 comb hair 25 25 25 25 m21
m22 scrape substance from surface 25 25 25 25 m22
m23 peel cucumber/potato 30 30 30 30 m23
m24 slice cucumber 25 25 25 25 m24
m25 flip bread 124 124 124 74 m25
m26 use spoon to scoop and pour 25 25 25 25 m26
m27 shave object 30 30 30 30 m27
m28 use roller to roll out dough 30 30 30 30 m28
m30 loosen nut with wrench 46 46 46 m30
m31 scoop and pour with measuring spoon/cup 30 30 30 30 m31
m32 insert peg into pegboard 140 140 140 m32
m33 brush powder accross grey tray 80 80 80 m33
m34 insert straw through to-go cup lid 25 25 25 m34
m35 m34 with eyes closed 25 25 25 25 m35
m36 m31 without pour 120 120 120 m36

Pouring Data

Pouring Helper Code
Data Pouring Water
Data Pouring Beans
Data Pouring Ice