使用Gstreamer处理RTSP视频流

文章目录

• RTSP视频流处理方法
• 1. Gstreamer整体框架
• • 1.1 Media Applications
  • 1.2 Core Framework
  • 1.3 Plugins
• 2. Gstreamer组件
• • 2.1 Element
  • 2.2 Pad
  • 2.3 Bin和Pipeline
• 3. gstreamer tools
• • 3.1 gst-inspect-1.0
  • 3.2 gst-launch-1.0
• 4. 参考链接

RTSP视频流处理方法

这里使用Gstreamer + OpenCV来处理RTSP视频流，因此对Gstreamer进行调查。

1. Gstreamer整体框架

Gstreamer是一个用于开发流式多媒体应用的开源框架，采用了基于插件（plugin）和管道（pipeline）的体系结构，框架中的所有的功能模块都被实现成可以插拔的组件（component）， 并且能够很方便地安装到任意一个管道上。由于所有插件都通过管道机制进行统一的数据交换，因此很容易利用已有的各种插件“组装”出一个功能完善的多媒体应用程序。
Nvidia为Gstreamer开发了许多plugin，这些plugin能够利用Nvidia硬件进行加速。Nvidia的deepstream就是基于gstreamer开发的。

下图是对基于Gstreamer框架的应用的简单分层：

1.1 Media Applications

最上面一层为应用，比如gstreamer自带的一些工具（gst-launch，gst-inspect等），以及基于gstreamer封装的库（gst-player，gst-rtsp-server，gst-editing-services等)根据不同场景实现的应用。

1.2 Core Framework

中间一层为Core Framework，主要提供：

• 上层应用所需接口
• Plugin的框架
• Pipline的框架
• 数据在各个Element间的传输及处理机制
• 多个媒体流（Streaming）间的同步（比如音视频同步）
• 其他各种所需的工具库

1.3 Plugins

最下层为各种插件，实现具体的数据处理及音视频输出，应用不需要关注插件的细节，会由Core Framework层负责插件的加载及管理。主要分类为：

• Protocols：负责各种协议的处理，file，http，rtsp等。
• Sources：负责数据源的处理，alsa，v4l2，tcp/udp等。
• Formats：负责媒体容器的处理，avi，mp4，ogg等。
• Codecs：负责媒体的编解码，mp3，vorbis等。
• Filters：负责媒体流的处理，converters，mixers，effects等。
• Sinks：负责媒体流输出到指定设备或目的地，alsa，xvideo，tcp/udp等。

2. Gstreamer组件

Gstreamer由许多基础的组件构成。

2.1 Element

Element是Gstreamer中最重要的对象类型之一。一个element实现了一个功能（读取文件，解码，输出等），程序需要创建多个element，并按顺序将其串连起来，构成一个完整的pipeline。

• Source Element 数据源元件　只有输出端，它仅能用来产生供管道消费的数据，而不能对数据做任何处理。一个典型的数据源元件的例子是音频捕获单元，它负责从声卡读取原始的音频数据，然后作为数据源提供给其它模块使用。

• Filter Element 过滤器元件　既有输入端又有输出端，它从输入端获得相应的数据，并在经过特殊处理之后传递给输出端。一个典型的过滤器元件的例子是音频编码单元，它首先从外界获得音频数据，然后根据特定的压缩算法对其进行编码，最后再将编码后的结果提供给其它模块使用。

• Sink Element 接收器元件　只有输入端，它仅具有消费数据的能力，是整条媒体管道的终端。一个典型的接收器元件的例子是音频回放单元，它负责将接收到的数据写到声卡上，通常这也是音频处理过程中的最后一个环节。

2.2 Pad

Pad是一个element的输入/输出接口，分为src pad（生产数据）和sink pad（消费数据）两种。
两个element必须通过pad才能连接起来，pad拥有当前element能处理数据类型的能力（capabilities），会在连接时通过比较src pad和sink pad中所支持的能力，来选择最恰当的数据类型用于传输。如果element不支持，程序会直接退出。
在element通过pad连接成功后，数据会从上一个element的src pad传到下一个element的sink pad然后进行处理。当element支持多种数据处理能力时，我们可以通过Cap来指定数据类型.

例如，下面的命令通过Cap指定了视频的宽高，videotestsrc会根据指定的宽高产生相应数据：

gst-launch-1.0 videotestsrc ! "video/x-raw,width=1280,height=720" ! autovideosink

2.3 Bin和Pipeline

Bin是一个容器，用于管理多个element，改变bin的状态时，bin会自动去修改所包含的element的状态，也会转发所收到的消息。如果没有bin，我们需要依次操作我们所使用的element。通过bin降低了应用的复杂度。
Pipeline继承自bin，为程序提供一个bus用于传输消息，并且对所有子element进行同步。当将pipeline的状态设置为PLAYING时，pipeline会在一个/多个新的线程中通过element处理数据。

下面通过一个文件播放的例子来熟悉上述提及的概念：
测试文件：sintel_trailer-480p.ogv

gst-launch-1.0 filesrc location=sintel_trailer-480p.ogv ! oggdemux name=demux ! queue ! vorbisdec ! autoaudiosink demux. ! queue ! theoradec ! videoconvert ! autovideosink

通过上面的命令播放文件时，会创建如下pipeline：

可以看到这个pipeline由8个element构成，每个element都实现各自的功能：

• filesrc读取文件
• oggdemux解析文件，分别提取audio，video数据
• queue缓存数据
• vorbisdec解码audio
• autoaudiosink自动选择音频设备并输出
• theoradec解码video
• videoconvert转换video数据格式
• autovideosink自动选择显示设备并输出

不同的element拥有不同数量及类型的pad，只有src pad的element被称为source element，只有sink pad的被称为sink element。

element可以同时拥有多个相同的pad，例如oggdemux在解析文件后，会将audio，video通过不同的pad输出。

3. gstreamer tools

Gstreamer自带了gst-inspect-1.0和gst-launch-1.0等其他命令行工具，我们可以使用这些工具完成常见的处理任务。

3.1 gst-inspect-1.0

查看gstreamer的plugin、element的信息。直接将plugin/element作为参数，会列出其详细信息，包括plugin的功能、Pad的输入输出类型、plugin的属性等。
如果不跟任何参数，会列出当前系统gstreamer所能查找到的所有插件。

# gst-inspect-1.0 rtspsrc
Factory Details:
Rank                     none (0)
Long-name                RTSP packet receiver
Klass                    Source/Network
Description              Receive data over the network via RTSP (RFC 2326)
Author                   Wim Taymans <wim@fluendo.com>, Thijs Vermeir <thijs.vermeir@barco.com>, Lutz Mueller <lutz@topfrose.de>
Plugin Details:
Name                     rtsp
Description              transfer data via RTSP
Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstrtsp.so
Version                  1.14.5
License                  LGPL
Source module            gst-plugins-good
Source release date      2019-05-29
Binary package           GStreamer Good Plugins (Ubuntu)
Origin URL               https://launchpad.net/distros/ubuntu/+source/gst-plugins-good1.0
GObject
+----GInitiallyUnowned
+----GstObject
+----GstElement
+----GstBin
+----GstRTSPSrc
Implemented Interfaces:
GstChildProxy
GstURIHandler
Pad Templates:
SRC template: 'stream_%u'
Availability: Sometimes
Capabilities:
application/x-rtp
application/x-rdt
Element has no clocking capabilities.
URI handling capabilities:
Element can act as source.
Supported URI protocols:
rtsp
rtspu
rtspt
rtsph
rtsp-sdp
rtsps
rtspsu
rtspst
rtspsh
Pads:
none
Element Properties:
name                : The name of the object
flags: readable, writable
String. Default: "rtspsrc0"
parent              : The parent of the object
flags: readable, writable
Object of type "GstObject"
async-handling      : The bin will handle Asynchronous state changes
flags: readable, writable
Boolean. Default: false
message-forward     : Forwards all children messages
flags: readable, writable
Boolean. Default: false
location            : Location of the RTSP url to read
flags: readable, writable
String. Default: null
protocols           : Allowed lower transport protocols
flags: readable, writable
Flags "GstRTSPLowerTrans" Default: 0x00000007, "tcp+udp-mcast+udp"
(0x00000000): unknown          - GST_RTSP_LOWER_TRANS_UNKNOWN
(0x00000001): udp              - GST_RTSP_LOWER_TRANS_UDP
(0x00000002): udp-mcast        - GST_RTSP_LOWER_TRANS_UDP_MCAST
(0x00000004): tcp              - GST_RTSP_LOWER_TRANS_TCP
(0x00000010): http             - GST_RTSP_LOWER_TRANS_HTTP
(0x00000020): tls              - GST_RTSP_LOWER_TRANS_TLS
debug               : Dump request and response messages to stdout(DEPRECATED: Printed all RTSP message to gstreamer log as 'log' level)
flags: readable, writable, deprecated
Boolean. Default: false
retry               : Max number of retries when allocating RTP ports.
flags: readable, writable
Unsigned Integer. Range: 0 - 65535 Default: 20
timeout             : Retry TCP transport after UDP timeout microseconds (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 5000000
tcp-timeout         : Fail after timeout microseconds on TCP connections (0 = disabled)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 20000000
latency             : Amount of ms to buffer
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 2000
drop-on-latency     : Tells the jitterbuffer to never exceed the given latency in size
flags: readable, writable
Boolean. Default: false
connection-speed    : Network connection speed in kbps (0 = unknown)
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551 Default: 0
nat-method          : Method to use for traversing firewalls and NAT
flags: readable, writable
Enum "GstRTSPNatMethod" Default: 1, "dummy"
(0): none             - None
(1): dummy            - Send Dummy packets
do-rtcp             : Send RTCP packets, disable for old incompatible server.
flags: readable, writable
Boolean. Default: true
do-rtsp-keep-alive  : Send RTSP keep alive packets, disable for old incompatible server.
flags: readable, writable
Boolean. Default: true
proxy               : Proxy settings for HTTP tunneling. Format: [http://][user:passwd@]host[:port]
flags: readable, writable
String. Default: null
proxy-id            : HTTP proxy URI user id for authentication
flags: readable, writable
String. Default: null
proxy-pw            : HTTP proxy URI user password for authentication
flags: readable, writable
String. Default: null
rtp-blocksize       : RTP package size to suggest to server (0 = disabled)
flags: readable, writable
Unsigned Integer. Range: 0 - 65536 Default: 0
user-id             : RTSP location URI user id for authentication
flags: readable, writable
String. Default: null
user-pw             : RTSP location URI user password for authentication
flags: readable, writable
String. Default: null
buffer-mode         : Control the buffering algorithm in use
flags: readable, writable
Enum "GstRTSPSrcBufferMode" Default: 3, "auto"
(0): none             - Only use RTP timestamps
(1): slave            - Slave receiver to sender clock
(2): buffer           - Do low/high watermark buffering
(3): auto             - Choose mode depending on stream live
(4): synced           - Synchronized sender and receiver clocks
port-range          : Client port range that can be used to receive RTP and RTCP data, eg. 3000-3005 (NULL = no restrictions)
flags: readable, writable
String. Default: null
udp-buffer-size     : Size of the kernel UDP receive buffer in bytes, 0=default
flags: readable, writable
Integer. Range: 0 - 2147483647 Default: 524288
short-header        : Only send the basic RTSP headers for broken encoders
flags: readable, writable
Boolean. Default: false
probation           : Consecutive packet sequence numbers to accept the source
flags: readable, writable
Unsigned Integer. Range: 0 - 4294967295 Default: 2
udp-reconnect       : Reconnect to the server if RTSP connection is closed when doing UDP
flags: readable, writable
Boolean. Default: true
multicast-iface     : The network interface on which to join the multicast group
flags: readable, writable
String. Default: null
ntp-sync            : Synchronize received streams to the NTP clock
flags: readable, writable
Boolean. Default: false
use-pipeline-clock  : Use the pipeline running-time to set the NTP time in the RTCP SR messages(DEPRECATED: Use ntp-time-source property)
flags: readable, writable, deprecated
Boolean. Default: false
sdes                : The SDES items of this session
flags: readable, writable
Boxed pointer of type "GstStructure"
tls-validation-flags: TLS certificate validation flags used to validate the server certificate
flags: readable, writable
Flags "GTlsCertificateFlags" Default: 0x0000007f, "validate-all"
(0x00000001): unknown-ca       - G_TLS_CERTIFICATE_UNKNOWN_CA
(0x00000002): bad-identity     - G_TLS_CERTIFICATE_BAD_IDENTITY
(0x00000004): not-activated    - G_TLS_CERTIFICATE_NOT_ACTIVATED
(0x00000008): expired          - G_TLS_CERTIFICATE_EXPIRED
(0x00000010): revoked          - G_TLS_CERTIFICATE_REVOKED
(0x00000020): insecure         - G_TLS_CERTIFICATE_INSECURE
(0x00000040): generic-error    - G_TLS_CERTIFICATE_GENERIC_ERROR
(0x0000007f): validate-all     - G_TLS_CERTIFICATE_VALIDATE_ALL
tls-database        : TLS database with anchor certificate authorities used to validate the server certificate
flags: readable, writable
Object of type "GTlsDatabase"
tls-interaction     : A GTlsInteraction object to promt the user for password or certificate
flags: readable, writable
Object of type "GTlsInteraction"
do-retransmission   : Ask the server to retransmit lost packets
flags: readable, writable
Boolean. Default: true
ntp-time-source     : NTP time source for RTCP packets
flags: readable, writable
Enum "GstRTSPSrcNtpTimeSource" Default: 0, "ntp"
(0): ntp              - NTP time based on realtime clock
(1): unix             - UNIX time based on realtime clock
(2): running-time     - Running time based on pipeline clock
(3): clock-time       - Pipeline clock time
user-agent          : The User-Agent string to send to the server
flags: readable, writable
String. Default: "GStreamer/1.14.5"
max-rtcp-rtp-time-diff: Maximum amount of time in ms that the RTP time in RTCP SRs is allowed to be ahead (-1 disabled)
flags: readable, writable
Integer. Range: -1 - 2147483647 Default: 1000
rfc7273-sync        : Synchronize received streams to the RFC7273 clock (requires clock and offset to be provided)
flags: readable, writable
Boolean. Default: false
max-ts-offset-adjustment: The maximum number of nanoseconds per frame that time stamp offsets may be adjusted (0 = no limit).
flags: readable, writable
Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0
max-ts-offset       : The maximum absolute value of the time offset in (nanoseconds). Note, if the ntp-sync parameter is set the default value is changed to 0 (no limit)
flags: readable, writable
Integer64. Range: 0 - 9223372036854775807 Default: 3000000000
default-rtsp-version: The RTSP version that should be tried first when negotiating version.
flags: readable, writable
Enum "GstRTSPVersion" Default: 16, "1-0"
(0): invalid          - GST_RTSP_VERSION_INVALID
(16): 1-0              - GST_RTSP_VERSION_1_0
(17): 1-1              - GST_RTSP_VERSION_1_1
(32): 2-0              - GST_RTSP_VERSION_2_0
backchannel         : The type of backchannel to setup. Default is 'none'.
flags: readable, writable
Enum "GstRTSPBackchannel" Default: 0, "none"
(0): none             - No backchannel
(1): onvif            - ONVIF audio backchannel
Element Signals:
"pad-added" :  void user_function (GstElement* object,
GstPad* arg0,
gpointer user_data);
"pad-removed" :  void user_function (GstElement* object,
GstPad* arg0,
gpointer user_data);
"no-more-pads" :  void user_function (GstElement* object,
gpointer user_data);
"handle-request" :  void user_function (GstElement* object,
gpointer arg0,
gpointer arg1,
gpointer user_data);
"on-sdp" :  void user_function (GstElement* object,
GstSDPMessage* arg0,
gpointer user_data);
"select-stream" :  gboolean user_function (GstElement* object,
guint arg0,
GstCaps* arg1,
gpointer user_data);
"new-manager" :  void user_function (GstElement* object,
GstElement* arg0,
gpointer user_data);
"request-rtcp-key" :  GstCaps * user_function (GstElement* object,
guint arg0,
gpointer user_data);
"accept-certificate" :  gboolean user_function (GstElement* object,
GTlsConnection* arg0,
GTlsCertificate* arg1,
GTlsCertificateFlags arg2,
gpointer user_data);
"before-send" :  gboolean user_function (GstElement* object,
GstRTSPMessage* arg0,
gpointer user_data);
Element Actions:
"push-backchannel-buffer" :  GstFlowReturn user_function (GstElement* object,
guint arg0,
GstBuffer* arg1);

3.2 gst-launch-1.0

用于创建及执行一个Pipline，因此通常使用gst-launch先验证相关功能，然后再编写相应应用。
通过上面ogg视频播放的例子，我们已经看到，一个pipeline的多个element之间通过 “!” 分隔，同时可以设置element及Cap的属性。
下面是解析RTSP视频流的pipeline：

gst-launch-1.0 -v rtspsrc location=rtsp://10.201.0.158:8554/vlc ! rtph264depay !  h264parse ! omxh264dec ! nvvidconv ! video/x-raw, width=(int)2048, height=(int)1536, format=(string)BGRx ! videoconvert !  ximagesink sync=false

我们可以通过gst-inspect-1.0工具查看每个plugin的功能和属性，选择合适的插件来构成pipeline。
具体在python实现OpenCV+Gstreamer的方法是：OpenCV提供了cv2.VideoCapture()函数，只需把Gstreamer参数传给该函数即可。
具体代码如下：

def open_cam_rtsp(uri, width, height, latency):
gst_str = ("rtspsrc location={} latency={} ! rtph264depay ! h264parse ! omxh264dec ! "
"nvvidconv ! video/x-raw, width=(int){}, height=(int){}, format=(string)BGRx ! "
"videoconvert ! appsink").format(uri, latency, width, height)
return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)
def open_cam_usb(dev, width, height):
# We want to set width and height here, otherwise we could just do:
#     return cv2.VideoCapture(dev)
gst_str = ("v4l2src device=/dev/video{} ! "
"video/x-raw, width=(int){}, height=(int){}, format=(string)RGB ! "
"videoconvert ! appsink").format(dev, width, height)
return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)
def open_cam_onboard(width, height):
# On versions of L4T previous to L4T 28.1, flip-method=2
# Use Jetson onboard camera
gst_str = ("nvcamerasrc ! "
"video/x-raw(memory:NVMM), width=(int)2592, height=(int)1458, format=(string)I420, framerate=(fraction)30/1 ! "
"nvvidconv ! video/x-raw, width=(int){}, height=(int){}, format=(string)BGRx ! "
"videoconvert ! appsink").format(width, height)
return cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)

4. 参考链接

https://gstreamer.freedesktop.org/documentation/application-development/introduction/gstreamer.html
https://gstreamer.freedesktop.org/documentation/application-development/introduction/basics.html
https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html

版权声明：本文内容由互联网用户自发贡献，该文观点仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 举报，一经查实，本站将立刻删除。

发布者：全栈程序员栈长，转载请注明出处：https://javaforall.cn/180740.html原文链接：https://javaforall.cn

未经允许不得转载：木盒主机 » 使用Gstreamer处理RTSP视频流