End-to-End Object Detection with Transformers (Carion et al.)

July 6, 2026

anchor-based detectors - each prediction corresponds to a particular spatial cell/anchor

-DETR queries are global

panoptic segmentation - unifies semantic segmentation (classifying every pixel) and instance segmentation (distinguishing individual objects)

-each pixel gets a category and object label
-make N bounding box predictions consisting of class and bounding box (x, y, w, h)
-those with no object prediction are assigned to "no object"

Hungarian matching - finds the lowest-cost assignment of predicted bounding boxes to ground truth bounding boxes

-"only one query/bound for each object"

| Query | Prediction |

| --- | --- |

| Q1 | Dog, good box |

| Q2 | Cat, decent box |

| Q3 | Cat, excellent box |

| Q4 | Car |

-matching determines that Q1 -> dog, Q2 -> no object, Q3 -> cat, Q4 -> car, and calculates/backprops loss accordingly
-self-attention helps create context for which queries are attending to which objects (de-duplicate)
-better prediction on larger objects, worse prediction on smaller objects

Architecture

1.CNN backbone - lower resolution activation map of the original image
2.Transformer encoder - learned positional embeddings + global context of the image
3.Transformer decoder - cross-attention between encoder outputs and self-attention across N queries, outputs vectors for those queries
4.Feed-forward NN - takes decoder outputs and predicts class + bounding box