FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation (He et al.)

June 15, 2026

tldr: bidirectional full fusion between RGB appearance and depth (point-cloud) for better pointwise feature extraction + 6DOF pose estimation from RGB-D images

-combine appearance (RGB) + geometry (depth) from RGBD images for representation learning + output representation selection
-predict 3D vector offset between each point on a surface and a given keypoint to predict pose

feature fusion - combine learned features/representations from different modalities to create a unified representation

-data-level/early fusion - stack depth as a 4th channel and feed RGB-D into one CNN
-feature-level fusion - extract features from each modality separately, then combine before making a prediction
-decision-level/late fusion - each modality makes a prediction, combine predictions

DenseFusion - point cloud network (PCN) extracts depth features, CNN extracts features from RGB

-the two feature sets are combined into a joint representation that single network uses to predict pose

convolution - provide a mechanism to convert spatial features into semantic features

-filter - a learned feature, represented as a weight matrix (i.e. edge, corner, etc.)
-receptive field - portion of the image visible at each layer of convolutions
-pooling - after convolutions, downsample using a certain heuristic (i.e. max pooling) over a fixed-size square of pixels
-images are padded on the edges so that each pixel is a center point for a convolution

Process:

1.Kernel (AxB pixels) sweeps over the original image.
2.For each position, calculate the weighted sum of the filter with the pixels
1.Pixels match the filter ←→ the output should be high
2.Pixels don’t match ←→ the output should be low
3.Perform pooling (downsampling occurs)
4.Layers of semantics get stacked on each other after subsequent convolutions
1.Layer 1 —\> 3x3 window, finds primitive patters
2.Layer 2 —\> 3x3 window of output maps from Layer 1
1.learn slightly more complex compositions/patterns then layer 1
3.Layer 3 —\> 3x3 window of output maps from Layer 2

… semantics/understanding of filters keeps increasing as layers get deeper

5.Each layer’s receptive field is larger, and is therefore able to detect larger/more abstract concepts
1.Filters are increased at each layer of convolution to accommodate

CNN Encoder - decreasing spatial resolution, increased semantic resolution (convolutions)

CNN Decoder - takes compressed, semantically-rich representation and upsamples toward resolution

6D pose estimation - predict a transformation matrix from the object coordinate system to the camera coordinate system

Method

1.Given RGBD input, a CNN extracts appearance features and a point cloud network extracts geometric features
2.Pixel-to-point and point-to-pixel modules are used to communicate between processes throughout feature extraction
3.The extracted point-wise features are fed into a segmentation and keypoint detection module
4.Least-squares fitting is applied to recover the 6D pose from the keypoints

pixel-to-point fusion - share appearance information extracted from CNN to the PCN

-given a pixel location, it’s depth, and the camera intrinsics, we can calculate it’s location in 3D space for better fusion
pixel position(u, v)+depth+camera intrinsics(cx, cy, fx, fy)→(x,y,z)\text{pixel position(u, v)} + \text{depth} + \text{camera intrinsics(cx, cy, fx, fy)} \rightarrow (x, y, z)

point-to-pixel fusion - similar as above

-this pipeline gives dense appearance embeddings from the CNN, and dense appearance embeddings from the PCN
-find the correspondence by projecting each point to the image plane with the camera intrinsic matrix
-obtain pairs of appearance and geometry features and concatenate them together to form the extracted dense RGBD feature