In this tutorial we will look at how you can use the full FOV of the image sensor.
When you are using
preview output from ColorCamera, DepthAI will crop the
frames by default to get the desired aspect ratio.
preview stream derives from the
video stream, which is cropped
(16:9 aspect ratio, max 4k resolution) from the
isp stream, which has the full FOV.
To get the full FOV of a sensor you need to use its max resolution (or 1/N of it, if supported).
So for OV9282 (800P)
you can use either 800P or 400P resolution to get full FOV. Meanwhile, for IMX378,
you need to set full 12MP resolution (as there’s no eg. 6MP support). You can use ColorCamera’s’
to downscale the 12MP if you don’t need such large frames.
A challenge occurs when your NN model expects a different aspect ratio (eg. 1:1) compared to the sensors aspect ratio (eg. 4:3), and we want to run NN inference on the full FOV of the sensor. Let’s say we have a MobileNet-SSD that requires 300x300 frames (1:1 aspect ratio) - we have a few options:
Stretch the ISP frame to 1:1 aspect ratio of the NN
Apply letterboxing to the ISP frame to get 1:1 aspect ratio frame
Crop the ISP frame to 1:1 aspect ratio and lose some FOV
The image above is the
isp output from the ColorCamera (12MP resolution from IMX378). If you aren’t downscaling ISP,
video output is cropped to 4k (max 3840x2160 due to the limitation of the
video output) as represented by
the blue rectangle. The Yellow rectangle represents a cropped
preview output when the preview size is set to a 1:1 aspect
ratio (eg. when using a 300x300 preview size for the MobileNet-SSD NN model) because the
preview output is derived from
video output (see ColorCamera for more information).
Change aspect ratio¶
Pros: Preserves full FOV. Cons: Due to stretched frames, NNs accuracy might decrease.
Changing aspect ratio (stretching) can be used Use
camRgb.setPreviewKeepAspectRatio(False). This means the aspect
ratio will not be preserved and the image will be “stretched”. This might be problematic for some off-the-shelf NN models, so some fine-tuning might be required.
Usage example here.
Pros: Preserves full FOV. Cons: Smaller “frame” means less features might decrease NN accuracy.
Letterboxing approach will apply “black bars” above and below
the image to the full FOV (isp) frames, so the aspect ratio will be preserved. You can
achieve this by using ImageManip with
manip.setResizeThumbnail(x,y) (for Mobilenet
The downside of using this method is that your actual image will be smaller, so some features might not be preserved,
which can mean the NN accuracy could decrease.
Usage example here.
Pros: No NN accuracy decrease. Cons: Frame is cropped, so it’s not full FOV.
Cropping the full FOV (isp) frames to match the NN aspect ratio can be used to get the best NN accuracy, but this decreases FOV. Usage example here.