RTPSeg Challenge

Participants are required to develop a road-scene vehicle-borne LiDAR point cloud semantic segmentation algorithm based on multi-modal data from RTPSeg (LiDAR point clouds, RGB images, and thermal infrared images). The algorithm shall address the following key technical challenges: (1) **Complex Illumination Conditions**: In degraded RGB scenarios such as nighttime and backlight, how to effectively fuse thermal radiation information from thermal infrared images to achieve accurate 3D semantic segmentation of road targets. (2) **Multi-modal Heterogeneous Data Fusion**: How to handle cross-modal fusion challenges including sensor projection errors, inconsistent fields of view (FoV) among multiple sensors, mismatched spatial resolutions, and pixel‑level spatial misalignment. (3) **Modality Gap Bridging and Complementary Exploitation**: How to overcome large modality gaps and fully leverage the texture/colour information from RGB images, thermal radiation information from thermal infrared images, and 3D geometric information from LiDAR point clouds. **Semantic Categories** The algorithm focuses on 18 semantic classes in urban and rural road scenes: building, car, carriageway, high vegetation, low vegetation, truck, sidewalk, fence, pole, bus, traffic sign, motorcycle, motorcyclist, traffic barrier, pedestrian, traffic light, bicycle, cyclist. --- **Dataset Information** - **Dataset Name**: RTPSeg – A multi‑modality dataset for LiDAR point cloud semantic segmentation assisted with RGB‑thermal images in autonomous driving. - **Source**: The dataset was jointly collected and constructed by the Information Engineering University and Sun Yat‑sen University, and has been published in the *ISPRS Journal of Photogrammetry and Remote Sensing* (2026). - **Acquisition Platform**: A custom‑designed multi‑sensor integrated platform was used, equipped with: - RGB camera (ZED, 1920×1080, 30 Hz), - Thermal infrared camera (Hinet‑1280, 1280×1024, 30 Hz, thermal sensitivity <0.05°C), - LiDAR (Ouster‑OS1, 128‑channel, 360° horizontal FoV, range 120 m, 10 Hz, accuracy <3 cm), - IMU (ICM‑20948, 9‑axis, 100 Hz). Data acquisition was synchronised with high‑frequency IMU data using FAST‑LIO2 for motion compensation and time synchronisation. Sensor extrinsic parameters were calibrated via the Levenberg‑Marquardt method to establish 3D‑to‑2D projection correspondences. - **Coverage**: Collected in Shenzhen, China, with a total route of approximately 22 km, covering both urban and rural areas under daytime and nighttime illumination conditions. - **Scale**: 103 data sequences in total, each containing 30 consecutive frames, amounting to over 3,000 synchronised multi‑modal frames, with more than 248 million point‑wise semantic annotations. - **Annotation**: 18 autonomous driving semantic categories (as listed above), with point‑wise labelling. - **Data Split**: A mixed‑scene scheme is adopted: - Training set: 94 sequences (sequences 1–14, 15–58, 63–98) - Test set: 9 sequences (sequences 59–62, 99–103) The training‑to‑test sample ratio is approximately 10:1. Sequences are split at the sequence level; all frames within the same sequence share the same illumination condition and collection scene. - **Download**: Official GitHub repository: https://github.com/sssssyf/RTPSeg - **Data License and Citation Requirement**: The dataset is provided exclusively for this competition; any other commercial or research use without permission is prohibited. When citing the dataset in competition reports or subsequent publications, please reference the following paper: > Yifan Sun, Chenguang Dai, Wenke Li, Xinpu Liu, Yongqi Sun, Ye Zhang, Weijun Guan, Yongsheng Zhang, Yulan Guo, Hanyun Wang. "RTPSeg: A multi-modality dataset for LiDAR point cloud semantic segmentation assisted with RGB-thermal images in autonomous driving." *ISPRS Journal of Photogrammetry and Remote Sensing*, 233, pp. 25-38, 2026.