Perception Package

by Unity Computer Vision

Perception production version unity 2021.3 unity 2022.1

Join our Mailing List! Sign up to stay up-to-date on our latest feature release, community events, blog posts, and more!

The Perception package provides a toolkit for generating large-scale synthetic datasets for computer vision training and validation. ## Getting Started with the Perception Package **[🌟 Perception Synthetic Data Tutorial 🌟](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/TUTORIAL.md)** Ideal for those new to either Unity, the Perception package, or synthetic data in general. Detailed instructions covering all the important steps: installing the Unity Editor, creating your first synthetic data generation project, adding domain randomization, visualizing, and analyzing your generated datasets. **No prior experience required.** **[Quick Start Guide](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/SetupSteps.md)** Instructions for adding the Perception package to your existing project. Recommended for users already familiar with Unity and the Perception package. If this is the first time exploring our toolset, we highly recommend briefly going through the **[Perception Synthetic Data Tutorial](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/TUTORIAL.md)** below! **[Human Pose Labeling and Randomization Tutorial](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/HumanPose/TUTORIAL.md)** Step by step instructions for using the keypoint, pose, and animation randomization tools included in the Perception package. We recommend that you complete the Perception Tutorial above before diving into this guide! **[Analyzing Datasets with Pysolotools](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools.md)** Introduction to Unity's [pysolotools](https://github.com/Unity-Technologies/pysolotools) - a python package for parsing, converting, and analyzing a SOLO dataset. Step by step instructions on how to parse a SOLO dataset and look at the dataset statistics in a [jupyter notebook](https://jupyter.org/). **[Visualizing a Dataset with Voxel51 Viewer](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/pysolotools-fiftyone.md)** We will walk through using [Voxel51](https://voxel51.com/) with our [custom SOLO importer](https://github.com/Unity-Technologies/pysolotools-fiftyone) to visualize a dataset. **[Converting to COCO](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Tutorial/convert_to_coco.md)** We will use [pysolotools](https://github.com/Unity-Technologies/pysolotools) to convert a SOLO dataset into the [COCO](https://cocodataset.org/#home) format. ## Frequently Asked Questions [Check out our FAQ](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/FAQ/FAQ.md) for a list of common questions, tips, tricks, and sample code for common code patterns. ## Reference Documentation In-depth documentation on specific components of the package. For the full set of feature-based documentation, take a look at the [Features](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Features/index.md) page. | Feature | Description | |-----------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Labeling](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/GroundTruthLabeling.md) | A component that marks a GameObject and its descendants with a set of labels | | [Label Config](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/GroundTruthLabeling.md#label-config) | An asset that defines a taxonomy of labels for ground truth generation | | [Perception Camera](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/PerceptionCamera.md) | Captures RGB images and ground truth from a [Camera](https://docs.unity3d.com/Manual/class-Camera.html). | | [Dataset Capture](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/DatasetCapture.md) | Ensures sensors are triggered at proper rates and accepts data for the JSON dataset. | | [Accumulation / Path Tracing](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Accumulation.md) | Accumulation is a feature that can be used for rendering techniques that require frames to be accumulated, such as path tracing. | | [Randomization](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Randomization/index.md) | The Randomization tool set lets you integrate domain randomization principles into your simulation. | | [Output Endpoint and Dataset Schema](https://github.com/Unity-Technologies/com.unity.perception/blob/main/com.unity.perception/Documentation~/Outputs.md) | An endpoint is responsible for delivering the generated data to the user. Currently supported output endpoints are: Solo endpoint (default), Perception endpoint, and No Output.| ## Datasets and Sample Projects ### Synthetic Homes

Head over to the [Synthetic Homes repository](https://github.com/Unity-Technologies/SyntheticHomes) for a 100K image dataset of annotated synthetic home interiors for the purpose of training computer vision models, as well as a configurable Unity based stand alone application for generating such datasets. ### Synthetic Humans

The [Synthetic Humans](https://github.com/Unity-Technologies/com.unity.cv.synthetichumans) package gives you the ability to procedurally generate and realistically place diverse groups of synthetic humans in your Unity Computer Vision projects. ### People Sans People

[PeopleSansPeople](https://unity-technologies.github.io/PeopleSansPeople/) is a human-centric privacy-preserving synthetic data generator with highly parametrized domain randomization. This dataset generator contains simulation-ready 3D human assets, a parameterized lighting and camera system, and generates 2D and 3D bounding box, instance and semantic segmentation, and COCO pose labels. ### SynthDet

[**SynthDet**](https://github.com/Unity-Technologies/SynthDet) is an end-to-end solution for training a 2D object detection model using synthetic data. ### Robotics Object Pose Estimation Demo

The [**Robotics Object Pose Estimation**](https://github.com/Unity-Technologies/Robotics-Object-Pose-Estimation) project demonstrates pick-and-place with a robot arm in Unity. It includes using ROS with Unity, importing URDF models, collecting labeled training data using the Perception package, and training and deploying a deep learning model. ## Community and Support Perception is an open-source project and we encourage and welcome contributions. If you wish to contribute, be sure to review our [contribution guidelines](https://github.com/Unity-Technologies/com.unity.perception/blob/main/CONTRIBUTING.md) and [code of conduct](https://github.com/Unity-Technologies/com.unity.perception/blob/main/CODE_OF_CONDUCT.md). For setup problems or discussions about leveraging the Perception package in your project, please create a new thread on the **[Unity Computer Vision forums](https://forum.unity.com/forums/computer-vision.626/)** with details such as operating system, reproduction steps, etc. If you run into any other problems with the Perception package or have a specific feature request, please open a **[GitHub Issue](https://github.com/Unity-Technologies/com.unity.perception/issues)**. For any other questions or feedback, connect directly with the Computer Vision team at [computer-vision@unity3d.com](https://github.com/Unity-Technologies/com.unity.perception/raw/main/mailto:computer-vision@unity3d.com). ## License & Citation The Perception package is licensed under [Apache License Version 2.0](https://github.com/Unity-Technologies/com.unity.perception/blob/main/LICENSE.md). If you find this package useful, consider citing it using: ``` @misc{unity-perception2022, title={Unity {P}erception Package}, author={{Unity Technologies}}, howpublished={\url{https://github.com/Unity-Technologies/com.unity.perception}}, year={2020} } ```