RGB describes appearance. Depth describes distance according to the sensor’s coordinate system and unit convention. Combining them requires attention to calibration, alignment, missing measurements, and scale.
Before modeling
- Record the sensor, depth units, and calibration used for each collection.
- Verify alignment between the color image, depth image, and any segmentation masks.
- Keep track of invalid depth values instead of treating them as valid zero-distance observations.
- Apply spatial transformations consistently across corresponding channels and masks.
Evaluating a benefit from depth
Compare RGB and RGB-D models using the same data partitions and training budget. Check results by occlusion severity or other relevant conditions. An improvement in aggregate performance alone does not explain when depth contributes.
Try it
Inspect a color image, a depth image, and a validity mask side by side. Look for boundary misalignment and regions where the sensor did not return a usable measurement.