« Back to home

LAN Production & Streaming

Note: My primary distro of choice is CentOS, so where I'm using linux, commands will be the redhat versions (YUM etc)

A fair few of you are probably aware that I regularly attend thebiggame's LAN parties. For the last event (TBG36), I was given the honour of heading up a streaming solution for the event.
First of all, let me give a rundown of what our aims were:

  • A 24/7 (ex outages) stream from a camera when there isn't anything going on
  • The ability to switch over to games being played, ideally from anyone's PC, and visa versa
  • Bringing in the sound feed from our stage microphone, such that stream can hear any announcements
  • Video out to projector (& audio for finals)
  • An internal-only stream to alleviate pressure from our already saturated downstream WAN pipe (this was an issue from TBG35)
  • WAN stream out to twitch.tv to begin, with expansibility to Youtube, etc in future
  • Ideally, all of this achieved with a budget of £0 or minimal

We could have gone for a full video production setup with ATEM production switchers and control panels, but this would easily cost a small fortune (£5,000+), and with a budget of 0 that's not really an option (or sensible). Instead, we opted for a full-network setup taking advantage of the already existing gigabit infrastructure in place.

Back at TBG35, we had a simple streaming solution which was quite simply a member of the crew connecting to twitch.tv and then pulling the stream back down when we wanted to view it. However, even with the exceptional 100/100mb pipe at the venue, we found stuttering was absolutely insane (pauses of 5s+ every 15 seconds or so). The first step, then, was to find a way to access the stream being sent to Twitch before it went WANside.

nginx-rtmp, have my babies

And then I stumbled upon this piece of magic. For the unaware, nginx is a extensible, lightweight, FOSS HTTP server & proxy, and is downright amazing on its own - I've been using it for all of my sites and applications for the best part of 5 years, and there's a good reason it powers ~15% of all sites on the web. If only there was some way to harness this wonderful server to mirror our own stream internally as well as sending it up to the web? Of course there is, I wouldn't be talking about it if there wasn't.
Introducing the primary piece of the puzzle, nginx-rtmp-module. This is a third-party module for nginx that extends with RTMP (flash)/HLS (apple)/MPEG-DASH (HTML5) support (including video on demand), relay support (push & pull), online transcoding, HTTP callbacks, and a HTTP control module. Oh my, and all this for Linux, FreeBSD, MacOS, or Windows. It's like a sysadmin's wet dream.

There's a few disadvantages that I'll go over quickly.

  • Network video introduces an inherant delay. In our setup, we were seeing about 5 seconds every time a stream is transcoded, which can add up (see later).
  • There's a minimal, but noticable, strain on system resources. If you start using the transcoding features available, this becomes extremely pronounced, to the point of needing a very powerful system.
  • RTMP uses a LOT of bandwidth, especially when redistributing the stream to multiple clients. In 3 days, we used almost 200gb of data (720p@30f). 100mb LAN is a must, ideally gigabit.

Compiling nginx with the rtmp module is easy peasy. First off, we'll need some dependencies:

yum install git gcc make pcre-devel zlib-devel openssl-devel

Grab either the stable or mainline (personal preference) nginx source from here:

mkdir /usr/src/nginx-rtmp
cd /usr/src/nginx-rtmp
wget http://nginx.org/download/nginx-*VERSION*.tar.gz
tar zxvf nginx-*VERSION*.tar.gz

and then get the latest GIT version of nginx-rtmp-module:
Note: It's important to keep nginx's source and the module's source in two separate folders

git clone https://github.com/arut/nginx-rtmp-module

Ensure that your favourite compiling beverage is available (mine's Cola) and get to work - this shouldn't take a huge amount of time, especially if you multithread it
Note: This is my personal configure command, feel free to switch it up or use any of the number of other options

cd nginx-*VERSION*

./configure --user=nginx --group=nginx --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-pcre --with-file-aio --with-http_ssl_module --add-module=/usr/src/nginx-rtmp-module

make -j (systhreads + 1)
make install

It's now time to give it a config file to munch on. I've stuck a sample config in a Github Gist along with comments on interesting parts. In summary, pushing a stream to the /out application will send it up to Twitch (but this requires a key in the URL), while /interna will only be available if you connect directly to the server (doesn't get pushed up). Plop this config at /etc/nginx/nginx.conf.

You're now ready to go. I'm pretty sure make install will give you an initscript which you can execute like usual with service nginx start. If it doesn't, you'll need to grab one from somewhere and stick it in /etc/init.d/nginx. To start on boot, do update-rc.d nginx defaults.

Wait how am I supposed to use this

Using any streaming application capable of sending H264 RTMP feeds (this includes Open Broadcaster Software, xSplit, and Flash Media Live Encoder (including dependents like vMix)), point your stream URL to rtmp://hostname/[out,interna]/ and set the stream key to out?psk=CHANGEMELOL (if streaming straight out) or some meaningful value if streaming to interna. If your application doesn't provide a space for the key, just add it to the end of the URL (ex rtmp://hostname/out/out?psk=CHANGEMELOL). Once connected, your stream will be available to connect to at the same URL from VLC or similar (including web players), excluding the psk for /out streams. nginx-rtmp will also push the stream up to any external sites you've added to the config.

Cameras

For this event, our aim was to have a camera mounted above the hall showcasing how clean and lovely everyone at the event was, as well as giving viewers something interesting to watch during downtime between games. We explored a few options including H264 IP cameras (which we'll probably invest in for the next event) and HDMI video cameras (camera + capture device too expensive), but settled on trying a slightly unorthodox solution, the Raspberry Pi + Camera.

I thought the Raspberry Pi was a kiddies toy

It may well help kids learn programming, but it also has some exceptional things going for it. It's only £30 (+£20 for camera) per unit, has onboard h264 encoding capabilities, and the camera can do 1080p@30f or 720@60f (most cheap IP cameras can't do this!). Combine this with the exceptionally low power requirement (sub-1A) and you're on to a winner, especially when I already owned two, so no money need be spilt.

There's quite a few guides already available that talk about streaming the output via either RTMP or JPEG, but RTMP introduces another significant delay (adding to the preexisting problem) and JPEG is very difficult to import to other devices (without installing software directshow drivers, which seems over the top). The solution is RTSP, which is basically from what I'm aware an open source version of RTMP, just with a much lower latency footprint. The Pi's best implimentation of this, h264v4l2rtspserver, isn't perfect (likes to drop frames, and also practically explodes if more than 1 client is connected at a time), but works exceptionally well for latency (sub-1s in my trials) and can be easily connected to using VLC or similar (which comes in very handy later on).

It requires registering v4l and other such fiddly stuff, but it's documented very well in the aformentioned link. There are some other ways of doing RTSP streaming, but this solution seems to work the best, as some others don't include correct metadata or similar which some players won't work without.

The one downside I've found with the Pi's standard camera is that it doesn't exactly do low light, well, at all:
lowlight We're going to try some more methods for improving this, but one idea that's come up is to use the Picam's NoIR variant (basically removes the IR filter from the camera) and bathe the room in IR light at night - if you've got any bright ideas (hurr), please do let me know!

Combination

Now that we have the ability for other people to stream their gameplay internally, and have a camera that we can connect to set up, how do we fade and switch between people? The answer is bulky, but simple: Open Broadcaster Software, or vMix, on a dedicated windows machine. You could have nginx running on this machine if it's powerful enough, but in my experience on a Phenom II 955 system it's a bit of a push. I use OBS at the moment as the AMD system isn't powerful enough for vMix, but vMix does have some extremely useful things (like stingers and transition effects, as well as better multi-source control) that wouldn't make me hesitate to recommend it if you have the computing grunt (and money) to make it happen.

For sound mixing, we used an external mixer already present at the event that outputted over standard headphone jack and then just brought that into OBS's Microphone input - you could also use Virtual Audio Cables and other snazzy methods if you'd prefer.

To mix together streams in OBS, make use of the beta (but still very stable) Video Source Plugin - which is basically just VLC in a plugin - and point it towards your camera's RTSP and game's RTMP streams (in separate scenes and / or as global sources, up to you). Set the output to twitch friendly settings, and point the URL to your nginx server, and you're good to go.

In Conclusion

There's some things that we're going to improve for the next event. See this document the summary of what we're going to be changing. I'll probably write up another post once TBG37 has been and gone (along with another scrubdown document).

If you'd like some basic help or guidance, you can contact me at the usual places, or drop an email to me@duck.me.uk. If you'd like me to come and work with you at a LAN in the UK to get streaming going (I'm based near Bournemouth, so if you're further north than London sadly I'll need help with travel costs) please also get in touch, and I'll be glad to help.

I hope the stuff that I've gone over helps you out in some way. Have fun, and best of luck!