Using the Video Capture Connector and Adapter
Using the Video Capture Connector
Overview
The Video Capture connector enables SAS Event Stream Processing (ESP) to stream video frames into or out of an ESP model. The connector uses third-party libraries from FFmpeg to read and decode the input, supporting publish and subscribe operations. As a publisher, the Video Capture connector reads video from an external source and publishes each extracted image frame into a Source window as a unique event that contains a binary blob field.
As a subscriber, the Video Capture connector subscribes to a window and reads output events that contain image data. The output can represent a real-time streaming protocol (RTSP) stream, a video file, or another stream type that is supported by FFmpeg.
The following parameters enable you to set values in the AVOption data structure. FFmpeg uses this data structure to set parameters that control audio-video codecs, formats, and filters.
inputformatavoptions
The AVOption data structure enables you to customize the behavior of FFmpeg during encoding, decoding, and processing of multimedia streams. For more information about the parameters that you can set, see FFmpeg documentation.
Source Window Schema Requirements
The Source window schema must contain at least
two fields: a key field of type int64 for a unique ID and one
of type blob for a video frame. The video frame is encoded in
JPEG format, TIFF format, SAS wide format (uncompressed), or PNG format.
<schema>
<fields>
<field name="ID" type="int64" key="true"/>
<field name="image" type="blob"/>
</fields>
</schema>
In addition to these required fields, the
Source window schema can contain a third field of type stamp.
When specified, this field contains the time that the frame is received.
<schema>
<fields>
<field name="ID" type="int64" key="true"/>
<field name="image" type="blob"/>
<field name="image_time" type="stamp"/>
</fields>
</schema>
The schema can contain a different third field
of type string that, when specified, contains the string
configured in the cameraid configuration parameter.
<schema>
<fields>
<field name="ID" type="int64" key="true"/>
<field name="image" type="blob"/>
<field name="cameraid" type="string"/>
</fields>
</schema>
The schema can contain both the time that the
frame is received and the string configured in the cameraid
configuration parameter. When both are specified, the time must precede the
cameraid.
<schema>
<fields>
<field name="ID" type="int64" key="true"/>
<field name="image" type="blob"/>
<field name="image_time" type="stamp"/>
<field name="cameraid" type="string"/>
</fields>
</schema>
If the schema contains both the time that the
frame is received and the string configured in the cameraid
configuration parameter, it can contain an additional field of type
string or rstring. When specified,
this field contains a JSON string containing metadata coming from FFmpeg related to
the
input stream and frame.
<schema>
<fields>
<field name="ID" type="int64" key="true"/>
<field name="image" type="blob"/>
<field name="image_time" type="stamp"/>
<field name="cameraid" type="string"/>
<field name="metadata" type="string"/>
</fields>
</schema>
Subscriber Usage
|
Parameter |
Description |
|---|---|
|
|
Specifies the output file name or stream name. |
|
|
Specifies the output
format. Supported formats include
|
|
|
Specifies the field from the output event that contains the image to output. |
|
Parameter |
Description |
|---|---|
|
|
Specifies the denominator of the time base for the output. |
|
|
Specifies the output
codec. The default value depends on
|
|
|
Specifies the output bit
rate in megabits per second. The default value is
|
|
|
Specifies any additional
FFmpeg AVoption as a comma-separated list (for example,
Note: String values must be enclosed in double quotation marks. |
|
|
Specifies the horizontal size of published images. The default value is the original size. |
|
|
Specifies the vertical size of published images. The default value is the original size. |
|
|
Specifies the subscribed
window field that contains the value to compare to the ID that is
configured in |
|
|
Specifies the camera ID to
compare to the value in the event field that is specified by
|
|
|
Specifies the number of
times the connector attempts to reconnect. A value of
|
|
|
Specifies the delay
between retry attempts in milliseconds. When the value is
|
|
|
Specifies the group of
pictures (GOP) value as an integer. The default value is
|
|
|
Specifies the FFmpeg
filters to apply to received frames as a string. Valid filters
include Note: If you specify the
|
|
|
Specifies whether to restart the connector when it stops due to an error. The connector restarts only if the error occurs after the connector reaches the running state. |
Publisher Usage
|
Parameter |
Description |
|---|---|
|
|
Specifies publisher
connector. The value must be set to
|
|
|
Specifies a real-time
streaming protocol URL to read a stream from or the file system path
to read a file from (for example,
|
|
|
Specifies the output
format of published images. Valid values include
|
|
Parameter |
Description |
|---|---|
|
|
Sets the event block type
to transactional. The default event block type is
|
|
|
Specifies the number of
events to include in a published event block. Specify an integer
value. The default value is
|
|
|
Specifies the name of
the section in
/opt/sas/viya/config/etc/SASEventStreamProcessingEngine/default/connectors.config
to parse for configuration parameters. Specify the value as
|
|
|
Specifies to build events
with |
|
|
Specifies the input video frame rate. The default value is the rate that is extracted from the input stream. |
|
|
Specifies the publish rate
in frames per second. Specify |
|
|
Specifies the order of
color layers for published images. Valid values are
|
|
|
Specifies the horizontal size of published images. The default value is the original size. |
|
|
Specifies the vertical size of published images. The default value is the original size. |
|
|
Specifies the desired
image quality for |
|
|
Specifies the number of
times to repeat reading and processing the file received from the
path specified by the |
|
|
Specifies the value of the
FFmpeg AVoption parameter |
|
|
Specifies any additional
FFmpeg AVoption as a comma separated list (for example,
Note: String values must be enclosed in double quotation marks. |
|
|
Specifies an arbitrary string that is copied into the corresponding string field in the Source window. This value can be used by the model to identify the source camera. |
|
|
Specifies whether to include additional metadata in the source window field containing the FFmpeg metadata. This value is ignored if that field is not present in the source window schema. |
|
|
Specifies the number of
times the connector attempts to reconnect. A value of
|
|
|
Specifies the delay
between retry attempts in milliseconds. When the value is
|
|
|
Specifies the FFmpeg
filters to apply to received frames as a string. Valid values
include Note: If you specify the
|
|
|
Specifies whether to restart the connector when it stops due to an error. The connector restarts only if the error occurs after the connector reaches the running state. |
|
|
Specifies the method that
is used for sampling the published events. This parameter is
effective only for stream-type input, such as RTSP streams. Valid
values include |
Using the Video Capture Adapter
Overview
The Video Capture adapter reads input video streams and extracts each image frame. Components that use the adapter can process the frame as a binary blob field. The adapter uses third-party libraries from FFmpeg to read and decode the input, supporting publish and subscribe operations. The information in Source Window Schema Requirements also applies to the Video Capture adapter.
As a publisher, the Video Capture adapter reads video from an external source and publishes each extracted image frame into a Source window as a unique event that contains a binary blob field. As a subscriber, the Video Capture adapter subscribes to a window and reads output events that contain image data. The output can represent a real-time streaming protocol (RTSP) stream, a video file, or another stream type that is supported by FFmpeg.
The following parameters enable you to set values in the AVOption data structure. FFmpeg uses this data structure to set parameters that control audio-video codecs, formats, and filters.
inputformatavoptions
The AVOption data structure enables you to customize the behavior of FFmpeg during encoding, decoding, and processing of multimedia streams. For more information, see FFmpeg documentation.
Subscriber Usage
|
Parameter |
Description |
|---|---|
|
|
Specifies a subscriber adapter. |
|
|
Specifies the standard URL in the form dfESP://host :pubsubport/ project/continuousquery /window. Users append the following
strings for subscribers: Note: Users append
|
|
|
Specifies the real-time
streaming protocol URL to write the stream to or specifies the path
to write an output file to (for example,
|
|
|
Specifies the subscribed window field that contains the image data. |
|
Parameter |
Description |
|---|---|
|
|
When
|
|
|
Specifies the publish/subscribe transport configuration file. For
For
Note: No transport configuration file is required for native transport. |
|
|
Specifies the camera ID to
compare to the value in the event field that is specified by
|
|
|
Specifies the subscribed
window field that contains the value to compare to the ID that is
configured in |
|
|
Specifies the horizontal size of produced images. The default value is the original size. |
|
|
Specifies the vertical size of produced images. The default value is the original size. |
|
|
Specifies any additional
FFmpeg AVoption as a comma-separated list (for example,
Note: String values must be enclosed in double quotation marks. |
|
|
Specifies the number of
times the connector attempts to reconnect. A value of
|
|
|
Specifies the delay
between retry attempts in milliseconds. When the value is
|
|
|
Specifies the time base of
the output stream. The default value is
|
|
|
Specifies the output
format. Supported formats include
|
|
|
Specifies the bit rate of
the output stream. The default value is
|
|
|
Specifies the group of
pictures (GOP) value. The default value is
|
|
|
Specifies the output
codec. The default value depends on
|
|
|
Specifies the FFmpeg
filters to apply to received frames as a string. Valid filters
include Note: If you specify the
|
Publisher Usage
|
Key-Value Pair |
Description |
|---|---|
|
|
Specifies the standard URL in the form dfESP://host:pubsubport/ project/continuousquery /window. |
|
|
Specifies a real-time
streaming protocol URL to read a stream from or specifies the file
system path to read a file from (for example,
|
|
|
Specifies the output format of published images. |
|
Key-Value Pair |
Description |
|---|---|
|
|
Specifies the guaranteed delivery configuration file. |
|
|
Specifies the transport
type. If you specify
|
|
|
Sets the logging level for
the adapter. This is the same range of logging levels that you can
set in the |
|
|
Specifies the log configuration file. |
|
|
Specifies the location of the file in the local file system that contains the OAuth token that is required for authentication by the publish/subscribe server. |
|
|
Specifies the name of
the section in
/opt/sas/viya/config/etc/SASEventStreamProcessingEngine/default/connectors.config
to parse for configuration parameters. Specify the value as
|
|
|
Sets the block size. The
default value is
|
|
|
When
|
|
|
When
|
|
|
When
|
|
|
Specifies the publish/subscribe transport configuration file. For
For
Note: No transport configuration file is required for native transport. |
|
|
When
|
|
|
Specifies the input video frame rate. The default value is the rate that is extracted from the input stream. |
|
|
Specifies the publish rate
in frames per second. Specify |
|
|
Specifies the order of
color layers for published images. Valid values are
|
|
|
Specifies the horizontal size of published images. The default value is the original size. |
|
|
Specifies the vertical size of published images. The default value is the original size. |
|
|
Specifies the desired
image quality for |
|
|
Specifies the number of
times to repeat reading and processing the file received from the
path specified by the |
|
|
Specifies the value of the
FFmpeg AVoption parameter |
|
|
Specifies any additional
FFmpeg AVoption as a comma-separated list (for example,
Note: String values must be enclosed in double quotation marks. |
|
|
Specifies an arbitrary string that is copied into the corresponding string field in the Source window. This value can be used by the model to identify the source camera. |
|
|
When
|
|
|
Specifies the number of
times the connector attempts to reconnect. A value of
|
|
|
Specifies the delay
between retry attempts. When the value is
|
|
|
Specifies the FFmpeg
filters to apply to received frames as a string. Multiple filters
can be used together in a single line as a FFmpeg filtergraph (for
example,
Note: If you specify the
|
Output Examples
Here is an example of an RTSP stream output:
<connector class="videocap" name="video_out" active="true">
<properties>
<property name="type"><![CDATA[sub]]></property>
<property name="snapshot"><![CDATA[false]]></property>
<property name="outputtimebase"><![CDATA[30]]></property>
<property name="imagefieldname"><![CDATA[image]]></property>
<property name="filename"><![CDATA[rtsp://127.0.0.1:8554/stream]]></property>
<property name="outputformat"><![CDATA[rtsp]]></property>
</properties>
</connector>
Here is an example of an MP4 file output:
<connector class="videocap" name="video_out" active="true">
<properties>
<property name="type"><![CDATA[sub]]></property>
<property name="snapshot"><![CDATA[false]]></property>
<property name="outputtimebase"><![CDATA[30]]></property>
<property name="imagefieldname"><![CDATA[image]]></property>
<property name="filename"><![CDATA[output/file.mp4]]></property>
<property name="outputformat"><![CDATA[mp4]]></property>
<property name="outputcodec"><![CDATA[h264]]></property>
<property name="bitrate"><![CDATA[4.5]]></property>
</properties>
</connector>
Here is an example of an HLS output:
<connector class="videocap" name="video_out" active="true">
<properties>
<property name="type"><![CDATA[sub]]></property>
<property name="snapshot"><![CDATA[false]]></property>
<property name="outputtimebase"><![CDATA[30]]></property>
<property name="imagefieldname"><![CDATA[image]]></property>
<property name="filename"><![CDATA[output/stream.m3u8]]></property>
<property name="outputformat"><![CDATA[hls]]></property>
<property name="avoptions"><![CDATA[timeout=2000,strftime=1,hls_init_time=5,hls_time=5,hls_list_size=5,hls_flags="+split_by_time+round_durations",hls_segment_filename="output/clip_%Y%m%d_%H%M%S.ts"]]></property>
</properties>
</connector>
Here is an example of a JPEG output:
<connector class="videocap" name="sub_alert_image">
<properties>
<property name="type"><![CDATA[sub]]></property>
<property name="cameraid"><![CDATA[pub_video_1]]></property>
<property name="cameraidfieldname"><![CDATA[camera_id]]></property>
<property name="outputformat"><![CDATA[image2]]></property>
<property name="bitrate"><![CDATA[4]]></property>
<property name="outputcodec"><![CDATA[mjpeg]]></property>
<property name="snapshot"><![CDATA[false]]></property>
<property name="imagefieldname"><![CDATA[image]]></property>
<property name="filename"><![CDATA[@ESP_PROJECT_OUTPUT@/alert_%06d.jpg]]></property>
</properties>
</connector>