Hey again! Morgan here with another state of the tech conversation. First of all I would like to extend my deepest apologies to folks who have been experiencing issues across the last few weeks. We’re very grateful for the support tickets you have sent in which have been instrumental in tracking things down.
We’re facing down a couple of challenging bugs I wanted to inform you about, and taking an opportunity for a bit of an early spring cleaning while we’re at it. The two major issues I’ll talk about right now are issues with Voice and Video and some pretty drastic performance issues associated with certain “Smart” ad-blocker programs and extensions.
We’ve pulled aside a strike team of devs to focus on resolving these issues and doing a bunch of proactive bugfixes while we’re at it.
Voice And Video
We recently rolled out our new and improved Integrated Voice and Video feature, built from the ground up to provide more reliability and better quality. So why the uptick in Voice and Video issues? Well there are a few things going on:
Legacy Connection Issues
First, the majority of issues with connectivity have turned out to be mostly folks who have not moved over to the new Integrated Voice and Video yet. Integrated Voice and Video has much more stable and robust connection management, which has allowed us to increase video quality while reducing how many server resources are needed for good performance. Unfortunately, one type of server in the process is shared with our older Voice and Video solution. While Integrated was doing fine, the old Voice and Video has struggled to maintain connections on these now more congested and active servers.
In order to combat this we have significantly scaled up the number of servers we are using to give the old Voice and Video the room it needs to perform at least as well as it can. However, we do strongly encourage folks who are experiencing Voice and Video connectivity issues to move over to the new solution as soon as possible.
Integrated Voice and Video Missing Features and Bugs
We launched integrated Voice and Video missing a couple of key features. Namely: support for virtual devices (like an OBS Virtual Camera or NVidia Broadcast’s virtual Mic) and avatar swapping when GMs use the “Speaking As” feature. On top of this, we had some confusing UI elements surrounding hardware issues that have made it difficult for folks to troubleshoot.
A big chunk of our bughunt effort is going into fixing these hardware issues and providing users with better visibility into what is happening when there are hardware issues. And when we’re through that, those remaining feature parity issues are at the top of our priority list.
Performance Issues
You may have also seen folks mention (or experienced yourself) a pretty substantial slowdown on our site. This is primarily seen around Compendium and Art Library search as well as during Charactermancer usage. We’ve traced these issues back to the usage of two different types of external programs. Applications that heavily modify the contents of the browser (such as Grammarly and LastPass) have always caused some slowdowns on the VTT because those applications use some of your processing power to search through the contents of Virtual Tabletop and add things in.
But a new class of external apps have recently started to cause some issues. These programs are “smart” ad blockers, such as NordVPN’s “Threat Protection” tool and extensions like Privacy Possum. Rather than traditional ad blockers that just block a known list of URLs, these programs use algorithms to try to determine if particular network behavior is “normal” and then they block anything that seems odd. While this works pretty well for normal web applications, the Roll20 VTT works more like a game, with constant back and forth communication with our servers to update character data or token positions.
“Threat Protection” and similar tools try to monitor this activity, and in doing so, add just enough latency to cause performance to slip, and if too many requests get backed up behind each other it can even cause the VTT to freeze. This same functionality causes issues with lots of other intensive web applications. For example, NordVPN recommends turning off Threat Protection when watching streaming video to prevent similar performance issues.
We’ve been working hard on this issue for the last few days and we think we’ve made substantial headway (I’m using “Threat Protection” right now as I type this!). But a few of these other blockers are still causing issues. We’re going to do our best to track those down, but if you do see strange performance issues, we strongly recommend you disable these heavier external applications or add Roll20 to whitelists. You can find a list of suggestions on steps you can try in our Help Center article.
Additional Bugs
While we’re slowing down feature development to focus on core stability, we’re also taking this opportunity to go deep on our bug backlog. While we always work on bugs during our normal work, this is an opportunity to focus on them solely. Look out for lots of small quality of life fixes coming in our change log.
Closing
For those of you who have had a poor experience with either Voice and Video or these extension related performance issues, I again offer you my apologies. We’ve got our best folks on it right now and won’t stop until your VTT experience is stabilized. Please don’t hesitate to send in a support ticket if you see something broken or your experience is disrupted.
Have a great weekend and happy adventuring!