Setup

Requirements

• NVidia GPU with matching CUDA installed (we used CUDA 11 for development)
• ZED SDK for corresponding CUDA version https://www.stereolabs.com/developers/release/ (both 3.7.3 and 3.7.7 have been used for development)
• Unity (we used 2021.3.2f1 for development)
• ZED 2i Stereo camera
• Projector
• The StreetLight Unity Package found in releases
  • This package includes the necessary parts from the ZED Plugin for Unity, which can be found in it's entirety at https://github.com/stereolabs/zed-unity/releases/
• Visual Studio for development (used 2022 for development)

Notice that the ZED SDK supports recording .svo files from the camera and the ZED Unity plugin supports replaying them. This can be especially helpful during development.

The projector should be mounted high and projecting on the ground. The camera however, should not film from above, but from the side, since human body detection does not work well from above.

Calibration and test

After installing the requirements and seting up the equipment, one should open the "Calibration" scene in the "Calibration" folder, make sure the the input type in the ZED_Rig_Mono under PersonManager is set to USB and build the project for the required target platform (Windows or Linux). Notice that the target system needs to have the ZED SDK installed as well. Starting the scene on the target system (e.g. a Zotac Box) with the projector and camera attached should display a circle on the ground. This circle will move slowly mutliple rounds until it stops and calibration is completed. A human should stand in the center of the circle and walk with it during calibration. This will generate a caliubration .csv file in the subfolder "CalibrationRecordings" of the program. Use the calibration file as input for the homography calculator, which can be found at https://git.tk.informatik.tu-darmstadt.de/SmartStreetLight/HomographyCalculator/releases This program uses RANSAC to compute a homography for the given calibration points.

The calibration can then be tested using the "CalibrationTest" scene. After building the scene, in the StreamingAssets folder, there will be a Configuration.json file where the path to the final homography file must be set. After that, the position of a person walking in frame will be shown on the ground.

Example application

A sample application for the framework can be found in the folder Pong. It implements a simple Pong game for two players, which will log the events that occured during playing to a text file.

Usage of the framework

The Unity package contains the prefab PersonManager, which can be used for your own applications. You can access the PersonManager in your own MonoBehavior using FindObjectOfType<PersonManager>, which will return the PersonManager instance. This class contains an ObservableCollection of Person-Objects that are currently present on the screen. You can subscribe to the events of this collection or access it in your Update method. You can also use the PersonAppeared and PersonDisappeared events to handle a person appearing or disappearing from the screen.

The event DetectionReady occurs when the ZEDCamera is ready to detect persons. After startup, starting the object detection module may take a few seconds.

The Person class implements the INotifyPropertyChanged interface that will occur when the position of a person changes. Each person gets assigned an integer Id that never changes. When a person disappears and reappears again, they will be recognized as a new person and a new Person object is created. Thus, the Id of a Person object never changes.

Both the events from the PersonManager as well as the events from the Person class can be used for semantically logging meaningful events, which is different for every application to be developed.

The current position in Unity coordinates can be retrieved using the GetUnityPosition() method on a Person instance. Notice that the homography file must be configured for this method to work.