The theme¶
In Nageru, most of the business logic around how your stream ends up looking is governed by the theme, much like how a theme works on a blog or a CMS. Most importantly, the theme governs the look and feel through the different scenes and transitions between them, such that an event gets a consistent visual language even if the operators differ. Instead of requiring the user to modify Nageru’s C++ core, themes are written in Lua, a lightweight scripting language made for embedding.
Themes contain a lot of logic, and writing one can seem a bit daunting at first. However, most events will be happy just tweaking one of the themes included with Nageru, and the operator (if different from the visual designer) will not need to worry at all.
Nageru ships with two themes, a default full-featured two-camera setup with side-by-side for e.g. conferences, and a minimal one that is easier for new users to understand.
Introduction to scenes¶
Anything that’s shown on the stream, or on one of the preview displays, is created by a Movit chain, instantiated by a Nageru scene. Movit is a library for high-quality, high-performance video filters, and Nageru’s themes can use a simplified version of Movit’s API where most of the low-level details are abstracted away.
Every frame, the theme chooses a scene and a set of parameters to it, based on what it thinks the picture should look like. Every scene consists of a set of inputs (which can be either live video streams or static pictures) and then a set of operators or effects to combine or modify each other. Movit compiles these down to a set of shaders that run in high speed on the GPU; the theme doesn’t see a pixel, and thus, Lua’s performance (even though good for its language class, especially if you use LuaJIT) will not matter much.
Basic scenes¶
The simplest possible scene takes only in an input and sends it on to the display (the output of the last added node is always sent to the screen, and in this case, that would be the input):
local scene = Scene.new(16, 9) -- Aspect ratio.
local input = scene:add_input()
input:display(0) -- First input card. Can be changed whenever you want.
scene:finalize()
The live scene is always processed in full resolution (typically 720p) and then scaled down for the GUI. Preview scenes are rendered in exactly the resolution required, although of course, intermediate steps could be bigger.
Setting parameters, and the get_scene entry point¶
Many effects support parameters that can vary per-frame. Imagine, for instance, a theme where you want to supports two inputs and fading between them. This means you will need a scene that produces two inputs and produces a mix of them; Movit’s MixEffect is exactly what you want here:
local scene = EffectChain.new(16, 9)
local input0 = scene:add_input()
input0:display(0)
local input1 = scene:add_input()
input1:display(1)
-- Note that add_effect returns its input for convenience.
local mix_effect = scene:add_effect(MixEffect.new(), input0, input1)
scene:finalize()
Every frame, Movit will call your get_scene function, which has this signature:
function get_scene(num, t, width, height, signals)
“width” and “height” are what you’d expect (the output resolution). t contains the current stream time in seconds. “num” contains 0 for the live view, 1 for the preview view, and 2, 3, 4, … for each of the individual stream previews. “signals” contains a bit of information about each input signal (see Signal information queries).
get_scene in return should return a scene. However, before you do that, you can set the parameters strength_first and strength_second on it; for instance like this:
function get_scene(num, t, width, height, signals)
-- Assume num is 0 here; you will need to handle the other
-- cases, too.
local fade_progress = 0.0
if t >= 1.0 and t >= 2.0: -- Between 1 and 2 seconds; do the fade.
fade_progress = t - 1.0
elseif t >= 2.0:
fade_progress = 1.0
end
mix_effect:set_float("strength_first", 1.0 - fade_progress)
mix_effect:set_float("strength_second", fade_progress)
return scene
end
Note that in the case where fade_progress is 0.0 or 1.0 (you are just showing one of the inputs), you are wasting GPU power by using the fade scene; you should just return a simpler one-input scene instead.
The get_scene function is the backbone of every Nageru theme. As we shall see, however, it may end up dealing with a fair bit of complexity as the theme grows.
Scene variants and effect alternatives¶
Setting up and finalizing a scene is relatively fast, but it still takes a measurable amount of CPU time, since it needs to create an OpenGL shader and have it optimized by the driver; 50–100 ms is not uncommon. Given that 60 fps means each frame is 16.7 ms, you cannot create new scenes in get_scene; every scene you could be using must be created at program start, when your theme is initialized.
For any nontrivial theme, there are a lot of possible scenes. Let’s return to the case of the MixEffect scene from the previous section. Now let us assume that we could deal with signals that come in at 1080p instead of the native 720p. In this case, we will want a high-quality scaler before mixing; ResampleEffect provides one:
local scene = EffectChain.new(16, 9)
local input0 = scene:add_input()
input0:display(0)
local input0_scaled = scene:add_optional_effect(ResampleEffect.new()) -- Implicitly uses input0.
input0_scaled:set_int("width", 1280) -- Could also be set in get_scene().
input0_scaled:set_int("height", 720)
input0_scaled:enable() -- Enable or disable as needed.
local input1 = scene:add_input()
input1:display(1)
local input1_scaled = ... -- Similarly here and the rest.
input1_scaled:enable_if(some_variable) -- Convenience form for enable() or disable() depending on some_variable.
-- The rest is unchanged.
Clearly, there are four options here; both inputs could be unscaled, input0 could be scaled but not input1, input1 could be scaled but not input0, or both could be scaled. That means four scenes. However, you don’t need to care about this; behind the scenes (no pun intended), Nageru will make all four versions for you and choose the right one as you call enable() or disable() on each effect.
Beyond simple on/off switches, an effect can have many alternatives, by giving in an array of effects. For instance, it is usually pointless to use the high-quality resampling provided by ResampleEffect for the on-screen outputs; we can use ResizeEffect (a simpler scaling algorithm provided directly by the GPU) that instead. The scaling is set up like this:
local input0 = scene:add_input()
input0:display(0)
local input0_scaled = scene:add_effect({ResampleEffect.new(), ResizeEffect.new()}) -- Implicitly uses input0.
input0_scaled:set_int("width", 1280) -- Just like before.
input0_scaled:set_int("height", 720)
-- Pick one in get_scene() like this:
input0_scaled:choose(ResizeEffect)
-- Or by numerical index:
input0_scaled:choose(1) -- Chooses ResizeEffect
Actually, add_optional_effect() is just a wrapper around add_effect() with IdentityEffect as the other alternative, and disable() is a convenience version of choose(IdentityEffect).
All alternatives must have the same amount of inputs, with an exception for IdentityEffect, which can coexist with an effect requiring any amount of inputs (if selected, the IdentityEffect just passes its first input unchanged). Similarly, if you set a parameter with set_int() or similar, it must be valid for all alternatives (again excepting IdentityEffect); if there is one that can only be used on a certain alternative, you must set it directly on the effect:
local resample_effect = ResampleEffect.new()
resample_effect:set_float("zoom_x", 1.0001) -- Not valid for ResizeEffect.
local input0_scaled = scene:add_effect({resample_effect, ResizeEffect.new()})
input0_scaled:set_int("width", 1280) -- Set on both alternatives.
input0_scaled:set_int("height", 720)
-- This is also possible, as choose() returns the chosen effect:
input0_scaled:choose(ResampleEffect):set_float("zoom_y", 1.0001)
Actually, more versions are created than you’d immediately expect. In particular, the output format for the live output and all previews are different (Y’CbCr versus RGBA), which is also handled transparently for you. Also, the inputs could be interlaced, or they could be images, or videos (see Image inputs and Video inputs), creating many more options. Again, you generally don’t need to care about this; Movit will make sure each and every of those generated scenes runs optimally on your GPU. However, if the combinatorial explosion increases startup time beyond what you are comfortable with, see Locking alternatives.
Transitions¶
As we have seen, the theme is king when it determines what to show on screen. However, ultimately, it wants to delegate that power to the operator. The abstraction presented from the theme to the user is in the form of transitions. Every frame, Nageru calls the following Lua entry point:
function get_transitions(t)
(t is again the stream time, but it is provided only for convenience; not all themes would want to use it.) get_transitions must return an array of (currently exactly) three strings, of which any can be blank. These three strings are used as labels on one button each, and whenever the operator clicks one of them, Nageru calls this function in the theme:
function transition_clicked(num, t)
where “num” is 0, 1 or 2, and t is again the theme time.
It is expected that the theme will use this and its internal state to provide the abstraction (or perhaps illusion) of transitions to the user. For instance, a theme will know that the live stream is currently showing input 0 and the preview stream is showing input 1. In this case, it can use two of the buttons to offer “Cut“ or “Fade” transitions to the user. If the user clicks the cut button, the theme can simply switch input and previews, which will take immediate effect on the next frame. However, if the user clicks the fade button, state will need to be set up so that next time get_scene() runs, it will return the scene with the MixEffect, until it determines the transition is over and changes back to showing only one input (presumably the new one).
Channels¶
In addition to the live and preview outputs, a theme can declare as many individual channels as it wants. These are shown at the bottom of the screen, and are intended for the operator to see what they can put up on the preview (in a sense, a preview of the preview).
The number of channels is determined by calling this function once at the start of the program:
Nageru.set_num_channels(2)
0 is allowed, but doesn’t make a lot of sense. Live and preview comes in addition to this.
Each channel will have a label on it; you set it by calling:
Nageru.set_channel_name(2, "Side-by-side")
Here, channel is 2, 3, 4, etc.—by default, 0 is called “Live” and 1 is called “Preview”, and you probably don’t need to change this.
Each channel has its own scene, starting from number 2 for the first one (since 0 is live and 1 is preview). The simplest form is simply a direct copy of an input, and most themes will include one such channel for each input. (Below, we will see that there are more types of channels, however.) Since the mapping between the channel UI element and inputs is so typical, Nageru allows the theme to simply declare that a channel corresponds to a given signal:
Nageru.set_channel_signal(2, 0)
Nageru.set_channel_signal(3, 1)
Here, channels 2 and 3 (the two first ones) correspond directly to inputs 0 and 1, respectively. The others don’t, and return -1. The effect on the UI is that the user can right-click on the channel and configure the input that way; in fact, this is currently the only way to configure them.
Furthermore, channels can have a color, which is governed by Nageru calling a function your theme:
function channel_color(channel)
The theme should return a CSS color (e.g. “#ff0000”, or “cyan”) for each channel when asked; it can vary from frame to frame. A typical use is to mark the currently playing input as red, or the preview as green.
White balance¶
Finally, there are two entry points related to white balance. The first one is:
Nageru.set_supports_wb(2, true)
If the first function is called with a true value (at the start of the theme), the channel will get a “Set WB” button next to it, which will activate a color picker, to select the gray point. To actually apply this white balance change, add the white balance element to the scene:
scene:add_white_balance()
The white balance effect will automatically figure out which input it is connected to, and fetch its gray point if needed. (If it is connected to e.g. a mix of several inputs, such as a camera and an overlay, you will need to give the input to fetch white balance from as as a parameter.)
More complicated channels: Composites¶
Direct inputs are not the only kind of channels possible; again, any scene can be output. The most common case is different kinds of composites, typically showing side-by-side or something similar. The typical UI presented to the user in this case is that you create a channel that consists of the finished setup; you use ResampleEffect (or ResizeEffect for preview scenes), PaddingEffect (to place the rectangles on the screen, one of them with a transparent border) and then OverlayEffect (to get both on the screen at the same time). Optionally, you can have a background image at the bottom, and perhaps a logo at the top. This allows the operator to select a pre-made composits, and then transition to and from it from a single camera view (or even between different composites) as needed.
Transitions involving composites tend to be the most complicated parts of the theme logic, but also make for the most distinct parts of your visual look.
Image inputs¶
In addition to video inputs, Nageru supports static image inputs. These work pretty much the same way as live video inputs. Recall that you chose what input to display like this:
input:display(0)
Image inputs are instead created by instantiating ImageInput and displaying that:
bg = ImageInput.new("bg.jpeg") -- Once, at the start of the program.
input:display(bg) -- In get_scene().
All image types supported by FFmpeg are supported; if you give in a video, only the first frame is used. The file is checked once every second, so if you update the file on-disk, it will be available in Nageru without a restart. (If the file contains an error, the update will be ignored.) This allows you to e.g. have simple message overlays that you can change without restarting Nageru.
Locking alternatives¶
In some cases, Nageru may be building in alternatives to a scene that you don’t really need, resulting in combinatorial explosion. (If the number of instances is getting high, you will get a warning when finalizing the scene.) For instance, in some cases, you know that a given transition scene will never be used for previews, just live. In this case, you can replace the call to scene:finalize() with:
scene:finalize(false)
In this case, you guarantee that the scene will never be returned when get_scene() is called with the number 0. (Similarly, you can use true to only use it for the live channel.)
Similarly, inputs can hold four different input types, but in some scenes, you may always use them with a specific one, e.g. an image “bg_img”. In this case, you may add the input with a specific type right away:
scene:add_input(bg_img)
Similarly, for a live input, you can do:
scene:add_input(0)
You can still use scene:display() to change the input, but it needs to be of the same type as the one you gave to add_input().
Finally, you can specify that some effects only make sense together, reducing the number of possibilities further. For instance, you may have an optional crop effect followed by a resample, where the resample is only enabled if the crop is. If so, you can do this:
resample_effect:always_disable_if_disabled(crop_effect)
Also, you can disable an optional effect if a given other effect is enabled:
overlay1_effect:promise_to_disable_if_enabled(overlay2_effect)
overlay2_effect:promise_to_disable_if_enabled(overlay1_effect)
Note that the latter is a promise from the user, not automatic disabling; since it is mostly useful for mutual exclusions, Nageru wouldn’t know which of the two to disable. (If you violate the promise, you will get an error message at runtime.) It can still be useful for reducing the number of alternatives, though.
For more advanced exclusions, you may choose to split up the scenes into several distinct ones that you manage yourself. At some point, however, you may choose to simply accept the added startup time and a bit of extra RAM cost; ease of use and flexibility often trumps such concerns.
Signal information queries¶
As previously mentioned, get_scene() takes in a “signals” parameter that you can query for information about each signal (numbered from 0; live and preview are channels, not signals), like its current resolution or frame rate:
get_frame_width(signal), get_frame_height(signal): Width and height of the last frame.
get_width(signal), get_height(signal): Width and height of the last field (the field height is half of the frame height for an interlaced signal).
get_interlaced(signal): Whether the last frame was interlaced.
get_has_signal(signal): Whether there is a valid input signal.
get_is_connected(signal): Whether there is even a card connected to this signal (USB cards can be swapped in or out); if not, you will get a stream of single-colored frames.
get_frame_rate_nom(signal), get_frame_rate_den(signal): The frame rate of the last frame, as a rational (e.g. 60/1, or 60000/1001 for 59.94).
get_last_subtitle(signal): See Ingesting subtitles.
get_human_readable_resolution(signal): The resolution and frame rate in human-readable form (e.g. “1080i59.94”), suitable for e.g. stream titles. Note that Nageru does not follow the EBU recommendation of using frame rate even for interlaced signals (e.g. “1080i25” instead of “1080i50”), since it is little-used and confusing to most users.
You can use this either for display purposes, or for choosing the right effect alternatives. In particular, you may want to disable scaling if the frame is already of the correct resolution.
Audio control¶
Before you attempt to control audio from the theme, be sure to have read the documentation about Audio.
The theme has a certain amount of control over the audio mix, assuming that you are in multichannel mode. This is useful in particular to be able to set defaults, if e.g. one channel should always be muted at startup, or to switch in/out certain channels depending on whether they are visible or not.
In particular, these operations are available:
# Returns number of buses in the mapping.
local num_buses = Nageru.get_num_audio_buses()
# Gets the name from the mapping. All indexes start at zero,
# so valid indexes are 0..(num_buses-1), inclusive.
local name = Nageru.get_audio_bus_name(N)
# 0.0 is zero amplification, as in the UI. Valid range is
# -inf to +6.0, inclusive.
local level = Nageru.get_audio_bus_fader_level_db(N)
set_audio_bus_fader_level_db(N, level)
# Similar as the above, but valid range is -15.0..+15.0 (dB).
# Valid bands are Nageru.EQ_BAND_{BASS, MID, TREBLE}.
local eq_level = Nageru.get_audio_bus_eq_level_db(N, Nageru.EQ_BAND_BASS)
Nageru.set_audio_bus_eq_level_db(N, Nageru.EQ_BAND_BASS, level)
# A boolean. Does not affect the bus levels set/returned above.
local muted = Nageru_get_audio_bus_mute(N)
Nageru_set_audio_bus_mute(N, false)
Note that any audio operation is inherently unsynchronized with the UI, so if the user reduces the number of audio buses while the theme tries to access one that is going away, you may get unpredictable behavior, up to and including crashes. Thus, you will need to be careful with such operations.
Also, you cannot do anything with audio before the first get_scene() call, since the audio mixer is initialized only after the theme has been loaded and initialized. Thus, for things that should be done only the first frame, the recommended method is to put code into get_scene() and have a guard variable that makes sure it is only run once, ever.
Overriding the status line¶
Some users may wish to override the status line, e.g. with recording time. If so, it is possible to declare a function format_status_line:
function format_status_line(disk_space_text, file_length_seconds)
if file_length_seconds > 86400.0 then
return "Time to make a new segment"
else
return "Disk space left: " .. disk_space_text
end
end
As demonstrated, it is given the disk space text (that would normally be there), and the length of the current recording file in seconds. HTML is supported.