Custom Pipelines
IP Camera streams can be configured with their own custom GStreamer pipelines, allowing for rich configuration of how the stream is processed. This section will not explain the intricacies of GStreamer pipelines as the official website provides excellent documentation on how these work. Instead, included are a few pipeline examples.
BrainFrame does quite a bit of work in the background to ensure that many different IP camera types are supported seamlessly. When using custom pipelines, more intimate knowledge of the IP camera stream is required compared to using BrainFrame normally.
Note that all custom pipelines:
- Must include a
{url}
template field. This is where the specified IP camera URL will be inserted into the pipeline. - Must have an
appsink
element named "main_sink". This is where frames will be extracted from the pipeline for processing. - May optionally include an element named "buffer_src". This is required for
frame skipping to work with custom
pipelines. This name should be given to an element in the pipeline
that sections off frame data from the network before decoding, like
rtph264depay
.
To specify a custom pipeline, check the "Advanced Options" checkbox in the stream creation window and enter your pipeline into the "Pipeline" textbox.
Example Pipelines¶
Cropping the Video Stream¶
For composite video streams or for scenes that contain uninteresting sections, one may want to crop the video stream before processing. Here is an example of a custom pipeline to accomplish this for an H264 RTSP stream:
rtspsrc location="{url}" ! rtph264depay name="buffer_src" ! decodebin ! videocrop top=x left=x right=x bottom=x ! videoconvert ! video/x-raw,format=(string)BGR ! appsink name="main_sink"
This pipeline uses the videocrop element to crop the video by some configurable value. The "x" values should be replaced with the amount in pixels to crop from each side of the frame.
Lower Latency Streaming¶
By default, BrainFrame will “buffer” frames in order to ensure a more stable streaming experience. In order to prevent that, try using the pipeline below:
rtspsrc location="{url}" latency=X ! rtph264depay name="buffer_src" ! decodebin ! videoconvert ! video/x-raw,format=(string)BGR ! appsink name="main_sink"
Replace the "X" in latency=X with 0 for no buffering at all. The unit X is in milliseconds.
Rotating the Video Stream¶
rtspsrc location="{url}" ! rtph264depay ! avdec_h264 ! videoconvert ! videoflip video-direction=x ! videoconvert ! video/x-raw,format=(string)BGR ! appsink name="main_sink"
The "x" value should be the number of degrees to rotate. Try numbers such as 0, 90, 180, and so on.
Hardware Decoding with Multiple Nvidia GPUs¶
BrainFrame automatically detects when an Nvidia GPU is available and attempts to do hardware video decoding on it. Currently, video decoding is only done on the first available device. This means that if your machine has multiple Nvidia GPUs installed, only one of them will be utilized.
GStreamer dynamically creates decoder elements that allow you to choose which Nvidia GPU the work will be done on. Using H.264 as our example format:
nvh264dec
uses device 0nvh264device1dec
uses device 1nvh264device2dec
uses device 2- ... and so on
By using different decoder elements for each stream's custom pipeline, you can
distribute decoding work across multiple GPUs. For example, if you had fifteen
video streams and three GPUs, you might consider having the first five use
nvh264dec
, the next five use nvh264device1dec
, and the final five use
nvh264device2dec
.
The device IDs referenced here are CUDA device IDs. By default, CUDA orders devices from fastest to slowest, device 0 being the fastest. It is possible to change the way CUDA orders devices via an environment variable. See the official documentation for details.
Here is an example pipeline that uses device 1 to decode an H.264 RTSP stream:
rtspsrc location="{url}" ! rtph264depay name="buffer_src" ! h264parse ! nvh264device1dec ! glcolorconvert ! video/x-raw(memory:GLMemory),format=(string)BGR ! gldownload ! video/x-raw,format=(string)BGR ! appsink name="main_sink"
Warning
Current releases of the BrainFrame Client do not support using NVCODEC hardware decoding. Using these custom pipelines will cause streaming errors in the client as a result. We only recommend these pipelines for advanced use cases.