Last week we discussed some ongoing performance issues across our services. Over the past few days we’ve deployed multiple updates to resolve many of those. We’re seeing clear improvements, and have more updates coming to bring everything into normal territory, as well as improve beyond our baseline!
Here’s what’s shipped this past week:
- We optimized our Compendium Service, helping prevent delays and timeouts with each request
- This was the primary cause of the few minutes of downtime on July 18 and 19.
- These fixes were released on that weekend, and helped the service perform at scale.
- However, while monitoring the service early in that week, we found the performance was not at a level we felt comfortable with for weekend play demands.
- We optimized multiple database queries, on both the Sheet and Compendium services.
- We found and fixed a loop in the D&D Builder tied to 2014 legacy data.
- We also launched a fix for calculation delays on Token bars (a limited HP/AC fix went out Friday, a full fix for all sheets is in QA now).
- Once the service was stable, we were able to look at the specific trouble areas as they were reported in individual games.
- We were able to find and resolve the combinations of options that caused performance to fall off.
- Phase 1 of the D&D sheet performance rollout, which we outlined in our last post, has been rolling out.
- This work reduces the amount of data transferred between sheet and services. This does not significantly improve performance as experienced as a user, but is required for Phase 2 of this rollout.
- This change is being released slowly, and we are closely monitoring the games that have these changes.
- Games that have received this rollout are showing the improvements we expected, while games outside it are not, which tells us the fix works and the rollout needs to expand to all users, which we’re currently working on.
Sheet opening lag has been resolved for many of the customers who’ve been working with us directly (thank you so much to each of you!), and we’re getting positive reports from others (thank you as well!). More has happened behind the scenes, and multiple teams have hunted down numerous potential problems that did not pan out to be a root cause. However, the changes above have produced the most meaningful impacts.
What’s Coming Next
- An update to fix roll lag that many are experiencing is in QA and will launch very soon once it has been fully vetted
- There are some cases where certain character builds can have significantly more data than other, nearly identical builds. That data being processed is what causes the rolls to take so long to process.
- There will be additional work later to process rolls without sending the extra data, as well as a cleanup process to remove extra, unused data
- A fix for Token Bar lag, when connected to character attributes, is set to ship this week. This should resolve token bar latency for all sheets / systems.
- Phase 2 of the D&D sheet performance updates, where we expect further optimizations that are more widely noticeable, is planned over the coming weeks.
- A new Compendium Service is on the roadmap further out that will speed up multiple services that interact with it
- Data cleanup process for unused data stored on individual sheets to optimize for everyone
If your game nights are being negatively impacted, please let us know in a support ticket so we can be sure we resolve everything! To speed things along, include a link to the game, note if there’s specific pages experiencing issues, and note any sheets or tokens that are experiencing lag.