NNComponent

NNComponent abstracts sourcing & decoding AI models, creating a DepthAI API node for neural inferencing, object tracking, and MultiStage pipelines setup. It also supports Roboflow integration.

DepthAI API nodes

For neural inference, NNComponent will use DepthAI API node:

• If we are using MobileNet-SSD based AI model, this component will create MobileNetDetectionNetwork (or MobileNetSpatialDetectionNetwork if spatial argument is set).

• If we are using YOLO based AI model, this component will create YoloDetectionNetwork (or YoloSpatialDetectionNetwork if spatial argument is set).

• If it’s none of the above, component will create NeuralNetwork node.

If tracker argument is set and we have YOLO/MobileNet-SSD based model, this component will also create ObjectTracker node, and connect the two nodes togeter.

Usage

from depthai_sdk import OakCamera, ResizeMode

with OakCamera(recording='cars-tracking-above-01') as oak:
    color = oak.create_camera('color')
    nn = oak.create_nn('vehicle-detection-0202', color, tracker=True)
    nn.config_nn(resize_mode='stretch')

    oak.visualize([nn.out.tracker, nn.out.passthrough], fps=True)
    oak.start(blocking=True)

Component outputs

• main - Default output. Streams NN results and high-res frames that were downscaled and used for inferencing. Produces DetectionPacket or TwoStagePacket (if it’s 2. stage NNComponent).

• passthrough - Default output. Streams NN results and passthrough frames (frames used for inferencing). Produces DetectionPacket or TwoStagePacket (if it’s 2. stage NNComponent).

• spatials - Streams depth and bounding box mappings (SpatialDetectionNework.boundingBoxMapping). Produces SpatialBbMappingPacket.

• twostage_crops - Streams 2. stage cropped frames to the host. Produces FramePacket.

• tracker - Streams ObjectTracker’s tracklets and high-res frames that were downscaled and used for inferencing. Produces TrackerPacket.

• nn_data - Streams NN raw output. Produces NNDataPacket.

Decoding outputs

NNComponent allows user to define their own decoding functions. There is a set of standardized outputs:

• Detections

• SemanticSegmentation

• ImgLandmarks

• InstanceSegmentation

Note

This feature is still in development and is not guaranteed to work correctly in all cases.

Example usage:

import numpy as np
from depthai import NNData

from depthai_sdk import OakCamera
from depthai_sdk.classes import Detections

def decode(nn_data: NNData):
    layer = nn_data.getFirstLayerFp16()
    results = np.array(layer).reshape((1, 1, -1, 7))
    dets = Detections(nn_data)

    for result in results[0][0]:
        if result[2] > 0.5:
            dets.add(result[1], result[2], result[3:])

    return dets


def callback(packet: DetectionPacket, visualizer: Visualizer):
    detections: Detections = packet.img_detections
    ...


with OakCamera() as oak:
    color = oak.create_camera('color')

    nn = oak.create_nn(..., color, decode_fn=decode)

    oak.visualize(nn, callback=callback)
    oak.start(blocking=True)

Reference