In this time of remote working not everyone may have access to sophisticated rendering technologies. Here at Idonix we decided to spend some time investigating how we can still deliver CG-style real-time graphics without the help of expensive hardware that may not be accessible at this time.

No Renderer? No Problem!

That’s where HTML comes in. If you’ve got a web browser, you’ve technically got a graphics renderer! Modern browsers on modern PCs have plenty of power to produce slick looking animations with the help of javascript and css:

Overlaying

There are plenty of ways to overlay web content onto video. We chose to try it out with OBS because a) it’s free, and b) there’s an NDI plugin that allows you to expose the OBS output as an NDI stream – we could then use the NDI Virtual Input tool to use the output as a webcam source and hey presto – overlay graphics on our Zoom/Skype/Teams meetings!

What About Control?

If we have a web page containing graphics, how do we input the content, and then animate them in and out? To solve this problem we chose to use websockets with nodejs. It works like this:

• A node server serves up the graphics html pages.
• Javascript on the page listens for messages on a websocket channel.
• The node server also serves up ‘control’ web pages with all the relevant text boxes to fill in and buttons to click to automate the graphics.
• Javascript on these control pages sends a websocket message back to the node server, which then broadcasts that message to any graphics pages listening on the relevant channel.

With our OBS solution, we could then embed the control web page inside OBS using its Custom Browser Docks feature:

Scaling

To allow multiple instances of these graphics to be used simultaneously, we used the concept of channels. A graphics web page specifies what channel it is listening on by including it as a route parameter in its url – the control web page similarly specifies what channel it wants to broadcast to in its url. Our node server then knows what graphics pages to broadcast the control message to when it comes in.

This linkage allows multiple scenes to be served and driven simultaneously – all from the same ‘renderer’. Ultimate scaleability!

• Because the server is in the cloud, you can choose to have the control screen running on a different machine to the overlay, and even have one control screen broadcasting tx commands to multiple graphics overlays for added redundancy.

Scenes

We decided to split out each graphic into its own file. That way we can have an overarching ‘scene’ (web page) that includes all the relevant graphical elements you need within it. In this example we have a Locator, Dog, and NameStrap graphic, all sitting in our IdonixL3 scene.

The Main Controller

These scenes can handle the control messages any way they see fit. We decided to implement a version of our stateless main controller pattern (as used with Viz, XPression and Unreal Engine) – this way the control code doesn’t need to keep state and know whether it needs to call GoIn, Update or GoOut on a graphic: it simply defines all the graphics that should currently be on-screen, and what data they should contain. The main controller then enquires of each graphic in turn and determines what needs to be done to that graphic to get it into this new desired state.

• For a deep dive into our main controller logic, see this blog post.

Anatomy of an HTML graphic

We define our graphic as a javascript class that extends HTMLElement. That way we can define it as a custom element for our scene to use. It also allows us to define the html content for this graphic and set it on itself by using the innerHTML property.

This definition is contained in a single javascript file per graphic, then each graphic we want to use in a particular scene, we include in our main html file.

Here’s a look at everything contained within our custom graphic class:

The Control Page

There’s not a lot to the control page really - this is how we approached it:

• We used Vue.js to bind text fields to data and store the values in local storage so the text is remembered the next time you visit the page.
• We used Bulma as a quick and easy way to style the fields nicely.