This article assumes your Shinobi is installed in the default location
/home/Shinobi
and using
Ubuntu 19.10.
How to view the Log stream for System, Monitor, or User
Learn about this
here
/system/logs
A Stream to Test Shinobi's Internal Processing
You can test Shinobi by creating a Monitor with these settings changed. This is a static file to simulate a livestream.
"bind failed: Address already in use" for UDP
On June 26, 2020 a community member by the name of
Tastiest#3464
in the Shinobi Community chat provided their solution to the problem.
-
Open the
Monitor Settings
-
In your
Connection
section Set
Automatic
to
No
-
Then fill
Path
with
-
Save and Enjoy!
Stream Delay (High Latency in Web Browser Stream)
One major factor is
I-Frames
(Keyframes) set in the camera's stream. If you choose not to encode (copy as video codec) then your camera will dictate this setting, usually from its web interface. In this instance set the
I-Frame
interval to match the FPS you are using. So if you are doing 15 FPS from the camera, set I-Frame to 15.
Although the latency is mainly because the
dashboard you are looking at is in a web browser.
There are multiple layers of processing upon the client-side CPU that occurs and that partially causes this delay, especially with H.264 based stream types.
So the best options for low latency are :
-
At the expense of slight delay but low CPU use. Set the
Stream Type
(blue section of Monitor Settings) to
Poseidon
over
Websocket
.
-
At the expense of CPU power on the server you can lower this delay. Set the
Stream Type
(blue section of Monitor Settings) to
Base64
or
MJPEG
.
Things to note :
-
Poseidon over Websocket has possibility of stutter under poor network-to-server connectivity conditions.
-
Base64 has no stream limit, watch as many of them on the dashboard as you want.
-
MJPEG is limited by the browsers HTTP connection limit.
-
Both MJPEG and Base64 will use a large chunk of CPU per stream.
Smudgy or Broken Streams
As with many issue this one is characteristic of many different problems. Here are some possible solutions.
-
Using a
Constant Bitrate
is the most common solution. Learn more
here
.
-
Toggling RTSP Transport from
TCP
to
UDP
.
-
To try UDP;
Open your Monitor Settings and set
Automatic
to
No
and set the
RTSP Transport
selector to
UDP
.
-
Shinobi defaults to TCP for RTSP Streams.
-
If using UDP does not help you should reset it to TCP.
Packet Loss or Black Stream - Disable Wall Clock Timestamps
Sometimes a camera that won't stream (appears black) correctly will better operate with this disabled.
-
Open
Monitor Settings
for the pesky camera.
The wrench icon.
-
Set
Use Camera Timestamps
to
Yes
.
The error below is one of many that can be fixed by this solution.
FFMPEG STDERRa few seconds ago
DTS 143301882296313, next:280367600000 st:0 invalid dropping PTS 143301882296313, next:280367600000 invalid dropping st:0
Black Stream - Analyzation Duration and Probe Size
You may fiddle with Analyzation Duration and Probe Size values to possibly acquire the stream. Try 1000000 for each field to start and work your way around.
Analyzation Duration
defined in the
FFmpeg Documentation
.
Specify how many microseconds are analyzed to probe the input. A higher value will enable detecting more accurate information, but will increase latency. It defaults to 5,000,000 microseconds = 5 seconds.
Probe Size
defined in the
FFmpeg Documentation.
Set probing size in bytes, i.e. the size of the data to analyze to get stream information. A higher value will enable detecting more information in case it is dispersed into the stream, but will increase latency. Must be an integer not lesser than 32. It is 5000000 by default.
Black Stream - Incompatible Codec
In many cases the audio data from the camera is causing FFmpeg process to hang. You can avoid this by doing one of the following in the Monitor Settings in Shinobi.
-
Set
Audio Codec
in the
Stream
section to
AAC
-
Set
Audio Codec
in the
Stream
section to
No Audio
It's possible that the stream data coming from the camera isn't compatible with web browsers. Usually that is H.265. You have three choices when that occurs.
-
Change the camera's internal settings (easiest)
-
Open the camera's internal settings and change the encoding type to
H.264
. This option's labelling and location varies based on manufacturer.
-
Have Shinobi encode the data into a viewable format (bad for hardware, easy)
-
Just set the
Video Codec
in the
Stream
section to
libx264
-
Use the Sub-stream from the camera to view in the browser with H.264 and still record in H.265.
Stuttering Streams, Dropping Streams
This can be a characteristic of different problems but usually just because of network connection issues from the client machine to the Shinobi server. The client machine is the machine you are using to view the Shinobi dashboard. Possible solutions to this are as follows.
-
Try setting the
Stream Type
to
HLS
.
-
This value is found in the
Stream
(blue)
section of your
Monitor Settings
.
-
Try setting the
Stream Type
to
MJPEG
.
-
This will use a fair amount of CPU power and has many limitation on the client side, use with caution.
-
Check that no other applications have an active connection to your camera.
-
Some cameras only allow a certain number of connections at a time due to resource limits on the camera itself.
-
Check that your wiring is maintained and isn't causing any packets to drop.
-
To learn more about reading camera logs please review the upcoming section of this article.
Reolink Camera Stream Drop
Try disabling the Auto-Reboot feature from the camera's web panel. This option is not exposed in the mobile app.
How to save Camera Logs
Be careful not to leave this on indefinitely if your camera has connection issues even if appearing to be fine. You will just be saving a whole lot of needless information into the database.
-
Open
Monitor Settings
for the camera.
The wrench icon.
-
Switch to
Advanced
view.
-
Scroll down to
Logging
.
-
Set to
Warning
and enable
Save to SQL
.
-
Wait a few moments or until problem strikes.
-
Click your
Username
in the top left then
Logs
.
Be careful not to forget that this is on after diagnosis. It may produce many log rows with many being useless information.
Getting logs from the Shinobi (camera.js) process
Open Terminal and run the following
pm2 logs --lines 100
Does the camera stream work in other applications?
Let's assume you have the correct stream URL. Try to run the stream in one of the following applications.
-
VLC
- media player by VideoLAN Organization -
https://www.videolan.org
-
FFprobe
- Stream analyzer included with FFmpeg and is meant to be used on command line. Shinobi has GUI that can be accessed by clicking your username in the top left, then Probe.
The camera does not work in VLC or FFprobe
-
Verify the network status of the camera.
-
You should be able to open the camera's IP Address in a web browser.
-
If your camera is connected by Ethernet then you should be able to see a light blinking (or solid if idle) on your internet router or network switch.
-
Verify connection status by opening a Web Browser and navigating to the IP Address of your camera.
-
Verify the type of stream coming from your camera by opening the internal settings.
-
Check for whether it is
MJPEG
or
H.264
. If possible to set H.264 then you should do for that.
-
If you are using
H.265
you can set the
Video Codec
in the
Stream
section to
Auto
or
libx264.
-
Verify the stream URL of your camera.
The camera works in one or both of these applications but not in Shinobi
Please put the hammer away and try each of the following.
-
Turn off all other applications that are accessing the streams, like ZoneMinder or VLC.
-
Check the connection information. See below.
-
Check your version of FFmpeg or swap it for another build. See below.
-
Check your file system permissions. See below.
Checking the Connection Information
In the
Connection
section try a combination of the following suggested option changes.
-
Set
Skip Ping
to
Yes
, if still not working you should switch it to
No
.
-
Set Automatic to
No
and turn
Force Port
to
Yes
.
-
Try the Web Port of the Camera in the
Port
field if you are using the RTSP port. Web port is the port you use to open the camera's internal settings from a web browser. The default Web Port is 80. The default RTSP port is 554. Leave
Protocol
as it is.
Check FFmpeg and Swap it for a static version
Open Terminal and run the following to see the version of FFmpeg. The Version number will be at the top of the output.
ffmpeg -version
If not at least version 3 or above then you need to update. There are three ways to achieve this.
-
Update your repositories to download the newest ffmpeg.
-
Install a static build with NPM.
-
Enter the Shinobi directory and run the following
cd /home/Shinobi
npm install ffmpeg-static
-
Install a static build from Johnathan's static Debian build hosted on Shinobi CDN
To uninstall the NPM static build and use the build just run the following.
cd /home/Shinobi
npm uninstall ffmpeg-static
Check file system permissions
Run the following commands to ensure your directories are accessible by Shinobi.
sudo chmod -R 755 /dev/shm/streams
sudo chmod -R 755 /home/Shinobi/videos
More info you might want to look at