Maximizing FOV

In this tutorial we will look at how you can use the full FOV of the image sensor.

When you are using preview output from ColorCamera, DepthAI will crop the frames by default to get the desired aspect ratio. preview stream derives from the video stream, which is cropped (16:9 aspect ratio, max 4k resolution) from the isp stream, which has the full FOV.

To get the full FOV of a sensor you need to use its max resolution (or 1/N of it, if supported). So for OV9282 (800P) you can use either 800P or 400P resolution to get full FOV. Meanwhile, for IMX378, you need to set full 12MP resolution (as there’s no eg. 6MP support). You can use ColorCamera’s’ .setIspScale() to downscale the 12MP if you don’t need such large frames.

A challenge occurs when your NN model expects a different aspect ratio (eg. 1:1) compared to the sensors aspect ratio (eg. 4:3), and we want to run NN inference on the full FOV of the sensor. Let’s say we have a MobileNet-SSD that requires 300x300 frames (1:1 aspect ratio) - we have a few options:

1. Stretch the ISP frame to 1:1 aspect ratio of the NN

2. Apply letterboxing to the ISP frame to get 1:1 aspect ratio frame

3. Crop the ISP frame to 1:1 aspect ratio and lose some FOV

The image above is the isp output from the ColorCamera (12MP resolution from IMX378). If you aren’t downscaling ISP, the video output is cropped to 4k (max 3840x2160 due to the limitation of the video output) as represented by the blue rectangle. The Yellow rectangle represents a cropped preview output when the preview size is set to a 1:1 aspect ratio (eg. when using a 300x300 preview size for the MobileNet-SSD NN model) because the preview output is derived from the video output (see ColorCamera for more information).

Change aspect ratio

Pros: Preserves full FOV. Cons: Due to stretched frames, NNs accuracy might decrease.

Changing aspect ratio (stretching) can be used Use camRgb.setPreviewKeepAspectRatio(False). This means the aspect ratio will not be preserved and the image will be “stretched”. This might be problematic for some off-the-shelf NN models, so some fine-tuning might be required. Usage example here.

Letterboxing

Pros: Preserves full FOV. Cons: Smaller “frame” means less features might decrease NN accuracy.

Letterboxing approach will apply “black bars” above and below the image to the full FOV (isp) frames, so the aspect ratio will be preserved. You can achieve this by using ImageManip with manip.setResizeThumbnail(x,y) (for Mobilenet x=300,y=300). The downside of using this method is that your actual image will be smaller, so some features might not be preserved, which can mean the NN accuracy could decrease. Usage example here.

Cropping

Pros: No NN accuracy decrease. Cons: Frame is cropped, so it’s not full FOV.

Cropping the full FOV (isp) frames to match the NN aspect ratio can be used to get the best NN accuracy, but this decreases FOV. Usage example here.

Got questions?

Head over to Discussion Forum for technical support or any other questions you might have.