A Dataset of Daily Interactive Manipulation


Robots that succeed in factories stumble to complete the simplest daily task humans take for granted, for the change of environment makes the task exceedingly difficult. Aiming to teach robot perform daily manipulative tasks in a changing environment using human demonstrations, we collected our own data of manipulative motions. The dataset focuses on force, torque, position and orientation of objects manipulated in daily tasks. It includes more than 1,400 trials of 33 types of daily motions and more than 1,500 trials of pouring alone, as well as helper code. We present our dataset to facilitate the research on task-oriented interactive manipulation.

We present a dataset of daily interactive manipulation. Specifically, we record daily performed fine motion in which an object is manipulated to interact with another object. We refer to the person who executes the motion as subject, the manipulated object as tool, and the interactive object as object. We focus on recording the motion of the tool. In some cases, we also record the motion of the object.

The dataset consists of two parts. The first part contains 1,483 trials that cover 32 types of motions. We choose fine motions that people commonly perform in daily life which involve interaction with a variety of objects. Different subsets of the motions are found in multiple different motion-related datasets. The motions we collect include those that are most frequently executed in cooking scenarios except that we do not include pick-and-place because it barely involve change of orientation.

The second part contains the pouring motion alone. We collect it to help with motion generalization to different environments. We chose pouring because 1) pouring is found to be the second frequently executed motion in cooking, right after pick-and-place and 2) we can vary the environment setup of the pouring motion easily by switching different materials, cups, and containers. The pouring data contains 1,596 trials of pouring 3 materials from 6 cups into 10 containers. We collect the two parts of the data using the same system.

The dataset provides position and orientation (PO) with 100% coverage, and force and torque (FT) with 96% coverage, and depth vision with 37% coverage. The less-than-perfect FT coverage is because of the introduction of FT collection system after the PO collection system. The less-than-perfect coverage of depth results from filming restrictions.

To cite this dataset:

Huang, Yongqiang, and Yu Sun. "A Dataset of Daily Interactive Manipulation." arXiv preprint arXiv:1807.00858(2018). [pdf]

  title={A Dataset of Daily Interactive Manipulation},
  author={Huang, Yongqiang and Sun, Yu},
  journal={arXiv preprint arXiv:1807.00858},

Available Downloads
m10: "Use Black Brush", 15 Trials
m11: "Spear Stuff Using Fork", 15 Trials
m13: " Fasten Screw ", 13 Trials
m14: " Loosen Screw ", 35 Trials

Please email yusun@usf.edu to get the full access to the dataset.

Collected Data

Download All
m2 stir with spatula 20 20 15 20
m3 sprinkle, shake pepper 40 40 15 40
m4 spread/oil 17 17 17 17
m6 vertical cut 16 16 16 16
m7 use spoon to pick up 98 98 78 35
m8 pizza wheel 25 25 15 25
m10 use black brush 15 15 15 15
m11 spear stuff using fork 32 32 32 32
m12 stir water using spoon 15 15 15 15
m13 fasten screw with screwdriver 15 15 15 15
m14 loosen screw with screwdriver 35 35 35
m15 unlock lock with key 180 180 180 30
m16 fasten nut with wrench 40 40 40 15
m17 use paint brush to dip and spread 14 14 14 14
m18 use hammer to hammer in nail 15 15 15 15
m19 brush teeth 40 40 40 15
m20 use file to file wooden thing 115 115 115 15
m21 comb hair 15 15 15 15
m22 scrape substance from surface 30 30 30 30
m23 peel cucumber/potato 30 30 30 30
m24 slice cucumber 15 15 15 15
m25 flip bread 125 125 125 15
m26 use spoon to scoop and pour 15 15 15 15
m27 shave object 30 30 30 30
m28 use roller to roll out dough 30 30 30 30
m30 loosen nut with wrench 46 46 46
m31 scoop and pour with measuring spoon/cup 30 30 30 30
m32 insert pef into pegboard 140 140 140
m33 brush powder accross grey tray 80 80 80
m34 insert straw through to-go cup lid 25 25 25
m35 m34 with eyes closed 20 20 20
m36 m31 without pour 120 120 120