Video Segmentation & Understanding
of Automotive Scenes

One pipeline, three tasks, temporal memory

Models that look at one frame at a time tend to flicker, where the same pixel jumps between labels from one moment to the next. This project treats a driving clip as a sequence instead. It carries a learned geometry state forward through the clip so segmentation and depth stay steady over time, and a separate filter works out how the camera itself moved.

How the system fits together

A GeometryFilter carries a ConvGRU memory at the network bottleneck (2048 channels, 32×64 spatial) to produce temporally-consistent segmentation and depth. Its embeddings feed an EgoMotionFilter that regresses the camera pose between frames.

Temporal consistency in action

The biggest win is stability over time. The naïve frame-by-frame baseline flickers, while adding the ConvGRU geometry filter, and then fine-tuning it, gives noticeably steadier segmentation across the whole clip.

Tip: click any clip to replay it from the start.

Segmentation, sequence 0

Naïve baselineper-frame, no temporal memory

Fine-tunedtemporal model, fine-tuned

End-to-endfull temporal pipeline

Segmentation, sequence 1

Naïve baselineper-frame, no temporal memory

Fine-tunedtemporal model, fine-tuned

End-to-endfull temporal pipeline

Segmentation, sequence 2

Naïve baselineper-frame, no temporal memory

Fine-tunedtemporal model, fine-tuned

End-to-endfull temporal pipeline

Segmentation, sequence 3

Naïve baselineper-frame, no temporal memory

Fine-tunedtemporal model, fine-tuned

End-to-endfull temporal pipeline

Segmentation, sequence 4

Naïve baselineper-frame, no temporal memory

Fine-tunedtemporal model, fine-tuned

End-to-endfull temporal pipeline

Depth target maps

These show the depth that the model was trained to recover, with closer surfaces shown lighter and far-away ones darker. The model's own depth head turned out to be the weakest part of the system, so we show the target maps here and let the point clouds below carry the depth story.

3D point-cloud reconstruction

Depth lifted into 3D and coloured by predicted semantic class, then placed in the scene using the estimated camera poses.

Quantitative evaluation (test set, Town 02)

These are the exact numbers reported by the evaluation in demonstration.ipynb. For reference, the loss terms on the very last evaluation batch read about 0.514 for segmentation, 0.00007 for depth, and 3.01 for camera pose. The camera-pose term is tracked separately and is not part of the average loss above.

Four-stage curriculum

The tasks are trained one stage at a time rather than all at once. A single-frame backbone is trained first, then reused and frozen so the later stages learn on top of stable features. The values below are read straight from the training scripts in the repo.

CARLA-simulated driving sequences

Synchronised RGB, semantic segmentation, depth, and per-frame camera extrinsics captured from the CARLA 0.9.16 simulator. Each raw clip holds 20 frames; the temporal model is trained on 6-frame windows from those clips.

22 semantic classes

Training environment

Dig deeper

The full technical walkthrough, the reference paper this project was built from, and the complete source are all available.