The theme¶
In Nageru, most of the business logic around how your stream ends up looking is governed by the theme, much like how a theme works on a blog or a CMS. Most importantly, the theme governs the look and feel through the different scenes and transitions between them, such that an event gets a consistent visual language even if the operators differ. Instead of requiring the user to modify Nageru’s C++ core, themes are written in Lua, a lightweight scripting language made for embedding.
Themes contain a lot of logic, and writing one can seem a bit daunting at first. However, most events will be happy just tweaking one of the themes included with Nageru, and the operator (if different from the visual designer) will not need to worry at all.
Nageru ships with two themes, a default full-featured two-camera setup with side-by-side for e.g. conferences, and a minimal one that is easier for new users to understand.
Introduction to chains¶
Anything that’s shown on the stream, or on one of the preview displays, is created by a Movit chain. Movit is a library for high-quality, high-performance video filters, and Nageru’s themes can use a simplified version of Movit’s API where most of the low-level details are abstracted away.
Every frame, the theme chooses a chain and a set of parameters to it, based on what it thinks the picture should look like. Every chain consists of a set of inputs (which can be either live video streams or static pictures) and then a set of operators or effects to combine or modify each other. Movit compiles these down to a set of shaders that run in high speed on the GPU; the theme doesn’t see a pixel, and thus, Lua’s performance (even though good for its language class, especially if you use LuaJIT) will not matter much.
High- and low-quality chains¶
The simplest possible chain takes only in an input and sends it on to the display (the output of the last added node is always sent to the screen, and in this case, that would be the input):
local chain = EffectChain.new(16, 9) -- Aspect ratio.
local input = chain:add_live_input(false, false) -- No bounce override, no deinterlacing.
input:connect_signal(0) -- First input card. Can be changed whenever you want.
chain:finalize(hq)
Note the “hq” parameter. Every chain needs to be able to run in two situations: Both for the stream output (the ”live” pane) and for the preview displays (the big “preview” pane and the channels). The details have to do with Nageru internals (high-quality chains need to have an additional Y’CbCr output), but the distinction is also useful for themes. In particular, some operations, like scaling, can be done in various quality levels, and for a low-resolution preview display, you don’t need maximum quality. Thus, in a preview chain (hq=false), you can safely take shortcuts.
The live chain is always processed in full resolution (typically 720p) and then scaled down for the GUI. Preview chains are rendered in exactly the resolution required, although of course, intermediate steps could be bigger.
Setting parameters, and the get_chain entry point¶
Many effects support parameters that can vary per-frame. Imagine, for instance, a theme where you want to supports two inputs and fading between them. This means you will need a chain that produces two inputs and produces a mix of them; Movit’s MixEffect is exactly what you want here:
local chain = EffectChain.new(16, 9)
local input0 = chain:add_live_input(false, false)
input0:connect_signal(0)
local input1 = chain:add_live_input(false, false)
input1:connect_signal(1)
local mix_effect = chain:add_effect(MixEffect.new(), input0, input1)
chain:finalize(hq)
Every frame, Movit will call your get_chain function, which has this signature:
function get_chain(num, t, width, height, signals)
“width” and “height” are what you’d expect (the output resolution). t contains the current stream time in seconds. “num” contains 0 for the live view, 1 for the preview view, and 2, 3, 4, … for each of the individual stream previews. “signals“ contains a bit of information about each input signal, like its current resolution or frame rate.
get_chain is in turn responsible for returning two values:
- The first return value is a Movit chain, as described in these sections. For the live stream (num=0), you should return a high-quality chain; for all others, you should return a low-quality chain.
- The second parameter is an closure that will be called just before the chain is to be rendered. (The same chain could be used in multiple OpenGL contexts at the same time, so you can’t just set the values immediately before returning. If you set them in the closure, Nageru and Movit will deal with all the required threading for you.)
In the returned closure, you can set the parameters strength_first and strength_second; for instance like this:
function get_chain(num, t, width, height, signals)
-- Assume num is 0 here; you will need to handle the other
-- cases, too.
prepare = function()
input0:connect_signal(0)
input1:connect_signal(1)
local fade_progress = 0.0
if t >= 1.0 and t >= 2.0: -- Between 1 and 2 seconds; do the fade.
fade_progress = t - 1.0
elseif t >= 2.0:
fade_progress = 1.0
end
mix_effect:set_float("strength_first", 1.0 - fade_progress)
mix_effect:set_float("strength_second", fade_progress)
end
return chain, prepare
end
Note that in the case where fade_progress is 0.0 or 1.0 (you are just showing one of the inputs), you are wasting GPU power by using the fade chain; you should just return a simpler one-input chain instead.
The get_chain function is the backbone of every Nageru theme. As we shall see, however, it may end up dealing with a fair bit of complexity as the theme grows.
Chain precalculation¶
Setting up and finalizing a chain is relatively fast, but it still takes a measurable amount of CPU time, since it needs to create an OpenGL shader and have it optimized by the driver; 50–100 ms is not uncommon. Given that 60 fps means each frame is 16.7 ms, you cannot create new chains in get_chain; every chain you could be using must be created at program start, when your theme is initialized.
For any nontrivial theme, there are a lot of possible chains. Let’s return to the case of the MixEffect chain from the previous section. Now let us assume that we could deal with signals that come in at 1080p instead of the native 720p. In this case, we will want a high-quality scaler before mixing; ResampleEffect provides one:
local chain = EffectChain.new(16, 9)
local input0 = chain:add_live_input(false, false)
input0:connect_signal(0)
local input0_scaled = chain:add_effect(ResampleEffect.new()) -- Implicitly uses input0.
chain_or_input.resample_effect:set_int("width", 1280) -- Would normally be set in the prepare function.
chain_or_input.resample_effect:set_int("height", 720)
local input1 = chain:add_live_input(false, false)
input1:connect_signal(1)
-- The rest is unchanged.
Clearly, there are four options here; both inputs could be unscaled, input0 could be scaled but not input1, input1 could be scaled but not input0, or both could be scaled. That means four chains.
Now remember that we need to create all your chains both in high- and low-quality versions. In particular, this determines the “hq” parameter to finalize(), but in our case, we would want to replace ResampleEffect by ResizeEffect (a simpler scaling algorithm provided directly by the GPU) for the low-quality versions. This makes for eight chains.
Now also consider that we would want to deal with interlaced inputs. (You can check if you get an interlaced input on the Nth input by calling “signals:get_deinterlaced(n)” from get_chain.) This further quadruples the number of chains you’d need to write, and this isn’t even including that you’d want the static chains. It is obvious that this should not be done by hand. The default included theme contains a handy Lua shortcut called make_cartesian_product where you can declare all the dimensions you would want to specialize your chain over, and have a callback function called for each possible combination. Movit will make sure each and every of those generated chains runs optimally on your GPU.
Transitions¶
As we have seen, the theme is king when it determines what to show on screen. However, ultimately, it wants to delegate that power to the operator. The abstraction presented from the theme to the user is in the form of transitions. Every frame, Nageru calls the following Lua entry point:
function get_transitions(t)
(t is again the stream time, but it is provided only for convenience; not all themes would want to use it.) get_transitions must return an array of (currently exactly) three strings, of which any can be blank. These three strings are used as labels on one button each, and whenever the operator clicks one of them, Nageru calls this function in the theme:
function transition_clicked(num, t)
where “num” is 0, 1 or 2, and t is again the theme time.
It is expected that the theme will use this and its internal state to provide the abstraction (or perhaps illusion) of transitions to the user. For instance, a theme will know that the live stream is currently showing input 0 and the preview stream is showing input 1. In this case, it can use two of the buttons to offer “Cut“ or “Fade” transitions to the user. If the user clicks the cut button, the theme can simply switch input and previews, which will take immediate effect on the next frame. However, if the user clicks the fade button, state will need to be set up so that next time get_chain() runs, it will return the chain with the MixEffect, until it determines the transition is over and changes back to showing only one input (presumably the new one).
Channels¶
In addition to the live and preview outputs, a theme can declare as many individual channels as it wants. These are shown at the bottom of the screen, and are intended for the operator to see what they can put up on the preview (in a sense, a preview of the preview).
The number of channels is determined by calling this function once at the start of the program:
function num_channels()
It should simply return the number of channels (0 is allowed, but doesn’t make a lot of sense). Live and preview comes in addition to this.
Each channel will have a label on it; Nageru asks the theme by calling this function:
function channel_name(channel)
Here, channel is 2, 3, 4, etc.—0 is always called “Live” and 1 is always called “Preview”.
Each channel has its own chain, starting from number 2 for the first one (since 0 is live and 1 is preview). The simplest form is simply a direct copy of an input, and most themes will include one such channel for each input. (Below, we will see that there are more types of channels, however.) Since the mapping between the channel UI element and inputs is so typical, Nageru allows the theme to simply declare that a channel corresponds to a given signal, by asking it:
function channel_signal(channel)
if channel == 2 then
return 0
elseif channel == 3 then
return 1
else
return -1
end
end
Here, channels 2 and 3 (the two first ones) correspond directly to inputs 0 and 1, respectively. The others don’t, and return -1. The effect on the UI is that the user can right-click on the channel and configure the input that way; in fact, this is currently the only way to configure them.
Furthermore, channels can have a color:
function channel_color(channel)
The theme should return a CSS color (e.g. “#ff0000”, or “cyan”) for each channel when asked; it can vary from frame to frame. A typical use is to mark the currently playing input as red, or the preview as green.
And finally, there are two entry points related to white balance:
function supports_set_wb(channel)
function set_wb(channel, red, green, blue)
If the first function returns true (called once, at the start of the program), the channel will get a “Set WB” button next to it, which will activate a color picker. When the user picks a color (ostensibly with a gray point), the second function will be called (with the RGB values in linear light—not sRGB!), and the theme can then use it to adjust the white balance for that channel. The typical way to to this is to have a WhiteBalanceEffect on each input and set its “neutral_color” parameter using the “set_vec3” function.
More complicated channels: Scenes¶
Direct inputs are not the only kind of channels possible; again, any chain can be output. The most common case is different kinds of scenes, typically showing side-by-side or something similar. The typical UI presented to the user in this case is that you create a channel that consists of the finished setup; you use ResampleEffect (or ResizeEffect for low-quality chains), PaddingEffect (to place the rectangles on the screen, one of them with a transparent border) and then OverlayEffect (to get both on the screen at the same time). Optionally, you can have a background image at the bottom, and perhaps a logo at the top. This allows the operator to select a pre-made scene, and then transition to and from it from a single camera view (or even between different scenes) as needed.
Transitions involving scenes tend to be the most complicated parts of the theme logic, but also make for the most distinct parts of your visual look.
Image inputs¶
In addition to video inputs, Nageru supports static image inputs. These work pretty much the same way as live video inputs; however, they need to be instantiated in a different way. Recall that live inputs were created like this:
input = chain:add_live_input(false, deint)
Image inputs are instead created by instantiating ImageInput and adding them manually to the chain:
input = chain:add_effect(ImageInput.new("bg.jpeg"))
Note that add_effect returns its input for convenience.
All image types supported by FFmpeg are supported; if you give in a video, only the first frame is used. The file is checked once every second, so if you update the file on-disk, it will be available in Nageru without a restart. (If the file contains an error, the update will be ignored.) This allows you to e.g. have simple message overlays that you can change without restarting Nageru.