HTML Real-Time Graphics
In this time of remote working not everyone may have access to sophisticated rendering technologies. Here at Idonix we decided to spend some time investigating how we can still deliver CG-style real-time graphics without the help of expensive hardware that may not be accessible at this time.
No Renderer? No Problem!
There are plenty of ways to overlay web content onto video. We chose to try it out with OBS because a) it’s free, and b) there’s an NDI plugin that allows you to expose the OBS output as an NDI stream – we could then use the NDI Virtual Input tool to use the output as a webcam source and hey presto – overlay graphics on our Zoom/Skype/Teams meetings!
What About Control?
If we have a web page containing graphics, how do we input the content, and then animate them in and out? To solve this problem we chose to use websockets with nodejs. It works like this:
- A node server serves up the graphics html pages.
- The node server also serves up ‘control’ web pages with all the relevant text boxes to fill in and buttons to click to automate the graphics.
With our OBS solution, we could then embed the control web page inside OBS using its Custom Browser Docks feature:
To allow multiple instances of these graphics to be used simultaneously, we used the concept of channels. A graphics web page specifies what channel it is listening on by including it as a route parameter in its url – the control web page similarly specifies what channel it wants to broadcast to in its url. Our node server then knows what graphics pages to broadcast the control message to when it comes in.
This linkage allows multiple scenes to be served and driven simultaneously – all from the same ‘renderer’. Ultimate scaleability!
- Because the server is in the cloud, you can choose to have the control screen running on a different machine to the overlay, and even have one control screen broadcasting tx commands to multiple graphics overlays for added redundancy.
We decided to split out each graphic into its own file. That way we can have an overarching ‘scene’ (web page) that includes all the relevant graphical elements you need within it. In this example we have a Locator, Dog, and NameStrap graphic, all sitting in our IdonixL3 scene.
The Main Controller
These scenes can handle the control messages any way they see fit. We decided to implement a version of our stateless main controller pattern (as used with Viz, XPression and Unreal Engine) – this way the control code doesn’t need to keep state and know whether it needs to call GoIn, Update or GoOut on a graphic: it simply defines all the graphics that should currently be on-screen, and what data they should contain. The main controller then enquires of each graphic in turn and determines what needs to be done to that graphic to get it into this new desired state.
- For a deep dive into our main controller logic, see this blog post.
Anatomy of an HTML graphic
Here’s a look at everything contained within our custom graphic class:
A json object containing all the state information about our graphic – whether it has finished initialising (don’t try and animate me until I’ve finished initialising), whether it is in or not (useful for a main controller pattern to know whether to call GoIn or Update) etc.
A subset of the State object: a nested json object representing the current data displayed on the graphic. This defines the schema of data to be sent to the graphic and is passed in on every GoIn and Update call. The new data passed in to the update method can be used to compare against the data currently stored on the graphic in order to work out what parts of the graphic need to change. For example, if the top line of a strap has stayed the same, but the bottom line has changed, the graphic only needs to animate the bottom line off then on again.
The actual HTML model of the graphic. This is stored here, then used when the graphic is initialised to insert into the DOM.
We cache a list of all the HTML elements in our template for use later. This is useful for saving time in our animation methods by not having to query and find the elements we want to animate every time.
We chose to use the Greensock Animation Platform to create our animations. This uses the concept of timelines - for our graphic we will have a top-level timeline object. Every GoIn, Update and GoOut call builds a sub-timeline and adds it to this main timeline. This allows the graphic to queue up all its animates if they come in too quickly because, by default, adding to a timeline places the new animation at the end of any existing animations on that timeline (this can be defeated by overriding existing animations if required too).
Go In, Go Out and Update Methods
The Control Page
There’s not a lot to the control page really - this is how we approached it: