Using the Video Capture Connector and Adapter

Using the Video Capture Connector

Overview

The Video Capture connector enables SAS Event Stream Processing (ESP) to stream video frames into or out of an ESP model. The connector uses third-party libraries from FFmpeg to read and decode the input, supporting publish and subscribe operations. As a publisher, the Video Capture connector reads video from an external source and publishes each extracted image frame into a Source window as a unique event that contains a binary blob field.

As a subscriber, the Video Capture connector subscribes to a window and reads output events that contain image data. The output can represent a real-time streaming protocol (RTSP) stream, a video file, or another stream type that is supported by FFmpeg.

The following parameters enable you to set values in the AVOption data structure. FFmpeg uses this data structure to set parameters that control audio-video codecs, formats, and filters.

  • inputformat
  • avoptions

The AVOption data structure enables you to customize the behavior of FFmpeg during encoding, decoding, and processing of multimedia streams. For more information about the parameters that you can set, see FFmpeg documentation.

Source Window Schema Requirements

The Source window schema must contain at least two fields: a key field of type int64 for a unique ID and one of type blob for a video frame. The video frame is encoded in JPEG format, TIFF format, SAS wide format (uncompressed), or PNG format.

<schema>
   <fields>
      <field name="ID" type="int64" key="true"/>
      <field name="image" type="blob"/>
   </fields>
</schema>

In addition to these required fields, the Source window schema can contain a third field of type stamp. When specified, this field contains the time that the frame is received.

<schema>
   <fields>
      <field name="ID" type="int64" key="true"/>
      <field name="image" type="blob"/>
      <field name="image_time" type="stamp"/>
   </fields>
</schema>

The schema can contain a different third field of type string that, when specified, contains the string configured in the cameraid configuration parameter.

<schema>
   <fields>
      <field name="ID" type="int64" key="true"/>
      <field name="image" type="blob"/>
      <field name="cameraid" type="string"/>
   </fields>
</schema>

The schema can contain both the time that the frame is received and the string configured in the cameraid configuration parameter. When both are specified, the time must precede the cameraid.

<schema>
   <fields>
      <field name="ID" type="int64" key="true"/>
      <field name="image" type="blob"/>
      <field name="image_time" type="stamp"/>
      <field name="cameraid" type="string"/>
   </fields>
</schema>

If the schema contains both the time that the frame is received and the string configured in the cameraid configuration parameter, it can contain an additional field of type string or rstring. When specified, this field contains a JSON string containing metadata coming from FFmpeg related to the input stream and frame.

<schema>
   <fields>
      <field name="ID" type="int64" key="true"/>
      <field name="image" type="blob"/>
      <field name="image_time" type="stamp"/>
      <field name="cameraid" type="string"/>
      <field name="metadata" type="string"/>
   </fields>
</schema>

Subscriber Usage

Required Parameters for Subscriber Video Capture Connector

Parameter

Description

filename

Specifies the output file name or stream name.

outputformat

Specifies the output format. Supported formats include rtsp, mpeg, mp4, and hls. For more information, see FFmpeg formats.

imagefieldname

Specifies the field from the output event that contains the image to output.

Optional Parameters for Subscriber Video Capture Connector

Parameter

Description

outputtimebase

Specifies the denominator of the time base for the output.

outputcodec

Specifies the output codec. The default value depends on outputformat.

bitrate

Specifies the output bit rate in megabits per second. The default value is 4.

avoptions

Specifies any additional FFmpeg AVoption as a comma-separated list (for example, "pixel_format="mjpeg",video_size="640x480",timeout=10000000").

Note: String values must be enclosed in double quotation marks.

resize_x

Specifies the horizontal size of published images. The default value is the original size.

resize_y

Specifies the vertical size of published images. The default value is the original size.

cameraidfieldname

Specifies the subscribed window field that contains the value to compare to the ID that is configured in cameraid. If the values do not match, the frame is suppressed. By default, no frames are suppressed. This parameter is not allowed unless cameraid is also configured.

cameraid

Specifies the camera ID to compare to the value in the event field that is specified by cameraidfieldname. If the values do not match, the frame is suppressed. By default, no frames are suppressed. This parameter is not allowed unless cameraidfieldname is also configured.

maxreconnectretries

Specifies the number of times the connector attempts to reconnect. A value of -1 enables unlimited retries. The default value is 0, which disables retries. This parameter applies only to streaming input or output (for example, real-time streaming protocol) and is ignored when the input or output is a file.

reconnectdelay

Specifies the delay between retry attempts in milliseconds. When the value is 0 or less, the delay increases incrementally from 0 seconds up to a maximum of 256 seconds. The default value is 0. When the connector runs in subscriber mode, events that are generated between retries are discarded.

gopsize

Specifies the group of pictures (GOP) value as an integer. The default value is 12.

filters

Specifies the FFmpeg filters to apply to received frames as a string. Valid filters include crop, rotate, vflip, hflip, and scale. Multiple filters can be used together in a single line as a FFmpeg filtergraph (for example, <property name="filters"><![CDATA[rotate=90*PI/180 , hflip]]></property>). You can reference incoming event field values in a filtergraph by using the FFmpeg metadata definition with the key esp.event.<fieldname>. For example, %{metadata\:esp.event.camera_id} is replaced with the value of camera_id for the corresponding event or frame. For more information, see FFmpeg documentation.

Note: If you specify the resize_x and resize_y parameters, the connector applies resizing after the filtergraph runs. To resize frames before or within the filtergraph, use the scale filter. For more information, see Filtergraph documentation.

restartonerror

Specifies whether to restart the connector when it stops due to an error. The connector restarts only if the error occurs after the connector reaches the running state.

Publisher Usage

Required Parameters for Publisher Video Capture Connector

Parameter

Description

type

Specifies publisher connector. The value must be set to pub.

filename

Specifies a real-time streaming protocol URL to read a stream from or the file system path to read a file from (for example, rtsp://167.21.72.35:554/live/NCAM065.stream or /devs/videoStream/small_bunny_1080p_60fps.mp4).

publishformat

Specifies the output format of published images. Valid values include jpeg, wide, tiff, and png.

Optional Parameters for Publisher Video Capture Connector

Parameter

Description

transactional

Sets the event block type to transactional. The default event block type is normal.

blocksize

Specifies the number of events to include in a published event block. Specify an integer value. The default value is 1.

configfilesection

Specifies the name of the section in /opt/sas/viya/config/etc/SASEventStreamProcessingEngine/default/connectors.config to parse for configuration parameters. Specify the value as [configfilesection].

publishwithupsert

Specifies to build events with opcode = Upsert instead of opcode = Insert.

inputrate

Specifies the input video frame rate. The default value is the rate that is extracted from the input stream.

publishrate

Specifies the publish rate in frames per second. Specify 0 to match the input video frame rate or -1 to publish at the maximum rate. The default value is 0.

publishcolor

Specifies the order of color layers for published images. Valid values are bgr (default) and rgb.

resize_x

Specifies the horizontal size of published images. The default value is the original size.

resize_y

Specifies the vertical size of published images. The default value is the original size.

qscale

Specifies the desired image quality for jpeg or tiff. Valid values are integers from 1 to 31, with 1 specifying the highest quality and largest file size and 31 specifying the lowest. The normal range for jpeg is 2 through 31. The default value is 3.

repeatcount

Specifies the number of times to repeat reading and processing the file received from the path specified by the filename parameter. The default value is 0.

inputformat

Specifies the value of the FFmpeg AVoption parameter input_format, which sets the preferred pixel format (for raw video) or codec name.

avoptions

Specifies any additional FFmpeg AVoption as a comma separated list (for example, "pixel_format="mjpeg",video_size="640x480",timeout=10000000").

Note: String values must be enclosed in double quotation marks.

cameraid

Specifies an arbitrary string that is copied into the corresponding string field in the Source window. This value can be used by the model to identify the source camera.

extendedframeinfo

Specifies whether to include additional metadata in the source window field containing the FFmpeg metadata. This value is ignored if that field is not present in the source window schema.

maxreconnectretries

Specifies the number of times the connector attempts to reconnect. A value of -1 enables unlimited retries. The default value is 0, which disables retries.This parameter applies only to streaming input or output (for example, real-time streaming protocol) and is ignored when the input or output is a file.

reconnectdelay

Specifies the delay between retry attempts in milliseconds. When the value is 0 or less, the delay increases incrementally from 0 seconds up to a maximum of 256 seconds. The default value is 0. When the connector runs in subscriber mode, the events that are generated between retries are discarded.

filters

Specifies the FFmpeg filters to apply to received frames as a string. Valid values include crop, rotate, vflip, hflip, and scale. Multiple filters can be used together in a single line as a FFmpeg filtergraph (for example, <property name="filters"><![CDATA[rotate=90*PI/180 , hflip]]></property>). For more information, see FFmpeg documentation.

Note: If you specify the resize_x and resize_y parameters, the connector applies resizing after the filtergraph runs. To resize frames before or within the filtergraph, use the scale filter. For more information, see Filtergraph documentation.

restartonerror

Specifies whether to restart the connector when it stops due to an error. The connector restarts only if the error occurs after the connector reaches the running state.

frameratemethod

Specifies the method that is used for sampling the published events. This parameter is effective only for stream-type input, such as RTSP streams. Valid values include last, pts, or closest. When last, the connector publishes the latest frame before the tick time. When pts, the connector calculates the frame presentation time stamp (PTS) interval based on the publish period and publishes the first image with a PTS greater than or equal to the calculated PTS. When closest, the connector publishes the closest frame to the tick time. The default value is closest.

Using the Video Capture Adapter

Overview

The Video Capture adapter reads input video streams and extracts each image frame. Components that use the adapter can process the frame as a binary blob field. The adapter uses third-party libraries from FFmpeg to read and decode the input, supporting publish and subscribe operations. The information in Source Window Schema Requirements also applies to the Video Capture adapter.

As a publisher, the Video Capture adapter reads video from an external source and publishes each extracted image frame into a Source window as a unique event that contains a binary blob field. As a subscriber, the Video Capture adapter subscribes to a window and reads output events that contain image data. The output can represent a real-time streaming protocol (RTSP) stream, a video file, or another stream type that is supported by FFmpeg.

The following parameters enable you to set values in the AVOption data structure. FFmpeg uses this data structure to set parameters that control audio-video codecs, formats, and filters.

  • inputformat
  • avoptions

The AVOption data structure enables you to customize the behavior of FFmpeg during encoding, decoding, and processing of multimedia streams. For more information, see FFmpeg documentation.

Subscriber Usage

Required Parameters for Subscriber Video Capture Adapter

Parameter

Description

type=sub

Specifies a subscriber adapter.

url=pubsubURL

Specifies the standard URL in the form dfESP://host :pubsubport/ project/continuousquery /window.

Users append the following strings for subscribers:?snapshot=false. When ?snapshot=true, the subscriber receives a collection of Insert events that are contained in the window at that point in time. The subscriber then receives a stream of events produced from the time of the snapshot onward. Those subsequent events can be Inserts, Updates, or Deletes.

Note: ?snapshot=true is invalid.

Users append ?collapse=true | false or ?rmretdel=true | false for subscribers if needed.

filename=url

Specifies the real-time streaming protocol URL to write the stream to or specifies the path to write an output file to (for example, rtsp://167.21.72.35:554/live/NCAM065.stream or /devs/videoStream/small_bunny_1080p_60fps.mp4).

imagefieldname=field

Specifies the subscribed window field that contains the image data.

Optional Parameters for Subscriber Video Capture Adapter

Parameter

Description

restartonerror= true | false

When true, specifies to restart the adapter if a fatal error is reported.

transportconfigfile=file

Specifies the publish/subscribe transport configuration file.

For rabbitmq, the default is ./rabbitmq.cfg.

For kafka, the default is ./kafka.cfg.

Note: No transport configuration file is required for native transport.

cameraid=id

Specifies the camera ID to compare to the value in the event field that is specified by cameraidfieldname. If the values do not match, the frame is suppressed. By default, no frames are suppressed. This parameter is not allowed unless cameraidfieldname is also configured.

cameraidfieldname=field

Specifies the subscribed window field that contains the value to compare to the ID that is configured in cameraid. If the values do not match, the frame is suppressed. By default, no frames are suppressed. This parameter is not allowed unless cameraid is also configured.

resize_x=size

Specifies the horizontal size of produced images. The default value is the original size.

resize_y=size

Specifies the vertical size of produced images. The default value is the original size.

avoptions=options

Specifies any additional FFmpeg AVoption as a comma-separated list (for example, "pixel_format="mjpeg",video_size="640x480",timeout=10000000").

Note: String values must be enclosed in double quotation marks.

maxreconnectretries=number

Specifies the number of times the connector attempts to reconnect. A value of -1 enables unlimited retries. The default value is 0, which disables retries. This parameter applies only to streaming input or output (for example, real-time streaming protocol) and is ignored when the input or output is a file.

reconnectdelay=milliseconds

Specifies the delay between retry attempts in milliseconds. When the value is 0 or less, the delay increases incrementally from 0 seconds up to a maximum of 256 seconds. The default value is 0. When the connector runs in subscriber mode, the events that are generated between retries are discarded.

outputtimebase=time base

Specifies the time base of the output stream. The default value is 30.

outputformat=format

Specifies the output format. Supported formats include rtsp, mpeg, mp4, and hls. For more information, see FFmpeg formats.

bitrate=mbps

Specifies the bit rate of the output stream. The default value is 4.

gopsize=integer

Specifies the group of pictures (GOP) value. The default value is 12.

outputcodec=codec

Specifies the output codec. The default value depends on outputformat.

filters=<value>

Specifies the FFmpeg filters to apply to received frames as a string. Valid filters include crop, rotate, vflip, hflip, and scale. Multiple filters can be used together in a single line as a FFmpeg filtergraph (for example, <property name="filters"><![CDATA[rotate=90*PI/180 , hflip]]></property>). You can reference incoming event field values in a filtergraph by using the FFmpeg metadata definition with the key esp.event.<fieldname>. For example, %{metadata\:esp.event.camera_id} is replaced with the value of camera_id for the corresponding event or frame. For more information, see FFmpeg documentation.

Note: If you specify the resize_x and resize_y parameters, the connector applies resizing after the filtergraph runs. To resize frames before or within the filtergraph, use the scale filter. For more information, see Filtergraph documentation.

Publisher Usage

Required Keys

Key-Value Pair

Description

url= pubsubURL

Specifies the standard URL in the form dfESP://host:pubsubport/ project/continuousquery /window.

filename= name

Specifies a real-time streaming protocol URL to read a stream from or specifies the file system path to read a file from (for example, rtsp://167.21.72.35:554/live/NCAM065.stream or /devs/videoStream/small_bunny_1080p_60fps.mp4).

publishformat= jpeg | wide | tiff | png

Specifies the output format of published images.

Optional Keys

Key-Value Pair

Description

gdconfig= file

Specifies the guaranteed delivery configuration file.

transport=native | rabbitmq | kafka

Specifies the transport type. If you specify rabbitmq or kafka transports instead of the default native transport, then use the required client configuration files that are specified in the description of the C++ C_dfESPpubsubSetPubsubLib() API call.

loglevel=trace | debug | info | warn | error | fatal | off

Sets the logging level for the adapter. This is the same range of logging levels that you can set in the C_dfESPpubsubInit() publish/subscribe API call and in the engine initialize() call. The default level is warn.

logconfigfile= file

Specifies the log configuration file.

tokenlocation= location

Specifies the location of the file in the local file system that contains the OAuth token that is required for authentication by the publish/subscribe server.

configfilesection=[ section]

Specifies the name of the section in /opt/sas/viya/config/etc/SASEventStreamProcessingEngine/default/connectors.config to parse for configuration parameters. Specify the value as [configfilesection]..

blocksize= size

Sets the block size. The default value is 1.

transactional=true | false

When true, event blocks are transactional. By default, event blocks are normal.

publishwithupsert=true | false

When true, build events with opcode = Upsert instead of Insert.

restartonerror=true | false

When true, specifies to restart the adapter if a fatal error is reported.

transportconfigfile= file

Specifies the publish/subscribe transport configuration file.

For rabbitmq, the default is ./rabbitmq.cfg.

For kafka, the default is ./kafka.cfg.

Note: No transport configuration file is required for native transport.

quiesceproject=true | false

When true, quiesces the project after all events are injected into the Source window.

inputrate=rate

Specifies the input video frame rate. The default value is the rate that is extracted from the input stream.

publishrate=rate

Specifies the publish rate in frames per second. Specify 0 to match the input video frame rate or -1 to publish at the maximum rate. The default value is 0.

publishcolor= bgr | rgb

Specifies the order of color layers for published images. Valid values are bgr (default) and rgb.

resize_x=size

Specifies the horizontal size of published images. The default value is the original size.

resize_y=size

Specifies the vertical size of published images. The default value is the original size.

qscale=scale

Specifies the desired image quality for jpeg or tiff. Valid values are integers from 1 to 31, with 1 specifying the highest quality and largest file size and 31 specifying the lowest. The normal range for jpeg is 2 through 31. The default value is 3.

repeatcount=count

Specifies the number of times to repeat reading and processing the file received from the path specified by the filename parameter. The default value is 0.

inputformat=format

Specifies the value of the FFmpeg AVoption parameter input_format, which sets the preferred pixel format (for raw video) or codec name.

avoptions=options

Specifies any additional FFmpeg AVoption as a comma-separated list (for example, "pixel_format="mjpeg",video_size="640x480",timeout=10000000").

Note: String values must be enclosed in double quotation marks.

cameraid=id

Specifies an arbitrary string that is copied into the corresponding string field in the Source window. This value can be used by the model to identify the source camera.

extendedframeinfo=true | false

When true, specifies to include additional metadata in the source window field containing the FFmpeg metadata. This value is ignored if that field is not present in the source window schema.

maxreconnectretries=number

Specifies the number of times the connector attempts to reconnect. A value of -1 enables unlimited retries. The default value is 0, which disables retries.This parameter applies only to streaming input or output (for example, real-time streaming protocol) and is ignored when the input or output is a file.

reconnectdelay=milliseconds

Specifies the delay between retry attempts. When the value is 0 or less, the delay increases incrementally from 0 seconds up to a maximum of 256 seconds. The default value is 0. When the connector runs in subscriber mode, the events that are generated between retries are discarded.

filters=crop | rotate | vflip | hflip | scale

Specifies the FFmpeg filters to apply to received frames as a string. Multiple filters can be used together in a single line as a FFmpeg filtergraph (for example, <property name="filters"><![CDATA[rotate=90*PI/180 , hflip]]></property>). For more information, see FFmpeg documentation.

Note: If you specify the resize_x and resize_y parameters, the connector applies resizing after the filtergraph runs. To resize frames before or within the filtergraph, use the scale filter. For more information, see Filtergraph documentation.

Output Examples

Here is an example of an RTSP stream output:

<connector class="videocap" name="video_out" active="true">
<properties>
<property name="type"><![CDATA[sub]]></property>
<property name="snapshot"><![CDATA[false]]></property>
<property name="outputtimebase"><![CDATA[30]]></property>
<property name="imagefieldname"><![CDATA[image]]></property>
<property name="filename"><![CDATA[rtsp://127.0.0.1:8554/stream]]></property>
<property name="outputformat"><![CDATA[rtsp]]></property>
</properties>
</connector>

Here is an example of an MP4 file output:

<connector class="videocap" name="video_out" active="true">
<properties>
<property name="type"><![CDATA[sub]]></property>
<property name="snapshot"><![CDATA[false]]></property>
<property name="outputtimebase"><![CDATA[30]]></property>
<property name="imagefieldname"><![CDATA[image]]></property>
<property name="filename"><![CDATA[output/file.mp4]]></property>
<property name="outputformat"><![CDATA[mp4]]></property>
<property name="outputcodec"><![CDATA[h264]]></property>
<property name="bitrate"><![CDATA[4.5]]></property>
</properties>
</connector>

Here is an example of an HLS output:

<connector class="videocap" name="video_out" active="true">
<properties>
<property name="type"><![CDATA[sub]]></property>
<property name="snapshot"><![CDATA[false]]></property>
<property name="outputtimebase"><![CDATA[30]]></property>
<property name="imagefieldname"><![CDATA[image]]></property>
<property name="filename"><![CDATA[output/stream.m3u8]]></property>
<property name="outputformat"><![CDATA[hls]]></property>
<property name="avoptions"><![CDATA[timeout=2000,strftime=1,hls_init_time=5,hls_time=5,hls_list_size=5,hls_flags="+split_by_time+round_durations",hls_segment_filename="output/clip_%Y%m%d_%H%M%S.ts"]]></property>
</properties>
</connector>

Here is an example of a JPEG output:

<connector class="videocap" name="sub_alert_image">
<properties>
<property name="type"><![CDATA[sub]]></property>
<property name="cameraid"><![CDATA[pub_video_1]]></property>
<property name="cameraidfieldname"><![CDATA[camera_id]]></property>
        <property name="outputformat"><![CDATA[image2]]></property>
<property name="bitrate"><![CDATA[4]]></property>
<property name="outputcodec"><![CDATA[mjpeg]]></property>
<property name="snapshot"><![CDATA[false]]></property>
<property name="imagefieldname"><![CDATA[image]]></property>
<property name="filename"><![CDATA[@ESP_PROJECT_OUTPUT@/alert_%06d.jpg]]></property>
</properties>
</connector>
Last updated: October 9, 2026