Workshop Goals: Dataset and Challenges
This is the second workshop from the JRDB workshop series, tailored to many perceptual problems for an autonomous robot to operate, interact and navigate in human environments. These perception tasks include any 2D or 3D visual scene understating problem as well as any other problems pertinent to human action, intention and social behaviour understanding such as 2D-3D human detection, tracking and forecasting, 2D-3D human body skeleton pose estimation, tracking and forecasting and human social grouping and activity recognition.
Recently, the community has paid increasing attention to the human activity understanding problem due to the availability of several large-scale annotated datasets for this computer vision and robotics task. However, the existing datasets for this problem are often collected from platforms such as Youtube and are only limited to 2D annotations for individual actions and activities annotations. The main focus of our CVPR workshop is the novel problem of social human activity understanding, consisting of three sub-tasks such as individual action detection, social group identification and social activity recognition. We also introduce JRDB-Act, a large-scale, ego-centric and multi-modal dataset.
JRDB dataset contains 67 minutes of the annotated sensory data acquired from the JackRabbot mobile manipulator and includes 54 indoor and outdoor sequences in a university campus environment. The sensory data includes a stereo RGB 360° cylindrical video stream, 3D point clouds from two LiDAR sensors, audio and GPS positions. In addition to the current 2D-3D person detection and tracking, we will release a new set of annotations for this dataset such as 1) individual actions, 2) human social group formation, and 3) social activity of each social group. Using these unique annotations, we would launch two new benchmarks and challenges for this workshop. We also have, as invited speakers, world-renowned experts in the field of visual perceptions for understanding human action, intention and social behaviour. Finally, we aim to foster discussion between the attendants to find useful synergies and applications of the solutions of these (or similar) perceptual tasks.
Dataset
The currently available annotations on the JackRobbot dataset and benchmark (JRDB) include:
2D bounding box annotations around all the pedestrians visible in five RGB streams and their cylindrical composition.
3D oriented cuboid annotations around pedestrians in two Velodyne-16 LiDAR streams.
2D-3D associations between bounding boxes and cuboids.
Time consistent trajectories (tracks) for all annotated individuals in both 2D and 3D.
In addition to the above, we have provided a new set of annotations, including:
Individual action labels for all the individuals visible in RGB 360° cylindrical video stream, representing:
Human pose actions (total 11 exclusive categories including a miscellaneous class).
Human-to-human interaction actions (total 3 categories including a miscellaneous class).
Human-to-object interaction actions (total 12 categories including a miscellaneous class).
The action label difficulty level (4 categories) for each attribute.
Social group formation of all the individuals, visible in RGB 360° cylindrical video stream, (ranging from the groups with 1 person to the groups with 29 people)
Social activity labels for each social groups as the accumulation of the majority of individual action labels for each social group
Open Challenge
In addition to the existing four benchmarks and challenges on JRDB, i.e. 2D-3D person detection and tracking challenges, in this workshop, we organise two new challenges using our new annotations:
Human social group detection
Individual action detection and Social activity recognition
The winners of each challenge have an opportunity to present their work as a spotlight (5 minutes) and poster presentation during the workshop.
