Art installation with TRAjectory Prediction
| trap | ||
| .gitignore | ||
| .python-version | ||
| build_opencv_with_gstreamer.sh | ||
| custom_bytetrack.yaml | ||
| poetry.lock | ||
| pyproject.toml | ||
| README.md | ||
| test_custom_rnn.ipynb | ||
| test_homography.ipynb | ||
| test_model.ipynb | ||
| test_tracker.ipynb | ||
| test_tracking_data.ipynb | ||
| test_training_data.ipynb | ||
Trajectory Prediction Video installation
Install
- Run
bash build_opencv_with_gstreamer.shto build opencv with gstreamer support - Use pyenv + poetry to install
How to
See also the sibling repo traptools for camera calibration and homography tools that are needed for this repo.
These are roughly the steps to go from datagathering to training
- Make sure to have some recordings with a fixed camera. [UPDATE: not needed anymore, except for calibration & homography footage]
- Recording can be done with
ffmpeg -rtsp_transport udp -i rtsp://USER:PASS@IP:554/Streaming/Channels/1.mp4 hof2-cam-$(date "+%Y%m%d-%H%M").mp4
- Recording can be done with
- Follow the steps in the auxilary traptools repository to obtain (1) camera matrix, lens distortion, image dimensions, and (2+3) homography
- Run the tracker, e.g.
poetry run tracker --detector ultralytics --homography ../DATASETS/NAME/homography.json --video-src ../DATASETS/NAME/*.mp4 --calibration ../DATASETS/NAME/calibration.json --save-for-training EXPERIMENTS/raw/NAME/- Note: You can run this right of the camera stream:
poetry run tracker --eval_device cuda:0 --detector ultralytics --video-src rtsp://USER:PW@ADDRESS/STREAM --homography ../DATASETS/NAME/homography.json --calibration ../DATASETS/NAME/calibration.json --save-for-training EXPERIMENTS/raw/NAME/, each recording adding a new file to therawfolder.
- Note: You can run this right of the camera stream:
- Parse tracker data to Trajectron format:
poetry run process_data --src-dir EXPERIMENTS/raw/NAME --dst-dir EXPERIMENTS/trajectron-data/ --name NAMEOptionally, smooth tracks:--smooth-tracks - Train Trajectron model
poetry run trajectron_train --eval_every 10 --vis_every 1 --train_data_dict NAME_train.pkl --eval_data_dict NAME_val.pkl --offline_scene_graph no --preprocess_workers 8 --log_dir EXPERIMENTS/models --log_tag _NAME --train_epochs 100 --conf EXPERIMENTS/config.json --batch_size 256 --data_dir EXPERIMENTS/trajectron-data - The run!
- On a video file (you can use a wildcard)
DISPLAY=:1 poetry run trapserv --remote-log-addr 100.69.123.91 --eval_device cuda:0 --detector ultralytics --homography ../DATASETS/NAME/homography.json --eval_data_dict EXPERIMENTS/trajectron-data/hof2s-m_test.pkl --video-src ../DATASETS/NAME/*.mp4 --model_dir EXPERIMENTS/models/models_DATE_NAME/--smooth-predictions --smooth-tracks --num-samples 3 --render-window --calibration ../DATASETS/NAME/calibration.json(the DISPLAY environment variable is used here to running over SSH connection and display on local monitor) - or on the RTSP stream. Which uses gstreamer to substantially reduce latency compared to the default ffmpeg bindings in OpenCV.
- To just have a single trajectory pulled from distribution use
--full-dist. Also try--z_mode.
- On a video file (you can use a wildcard)