Roll20 is Hiring a Senior Infrastructure Engineer: Real-time Database

Remote Contract Position – US/Canada

Employment Type: Full Time

A BIT ABOUT US

Roll20 brings players, game masters, creators, and publishers together through a family of TTRPG brands and products, including the Roll20 virtual tabletop, Demiplane, DriveThruRPG, and Dungeon Scrawl. We are dedicated to bringing players together regardless of distance, experience, or hardware. We strive to lessen the technical burden on the participants, to facilitate the formation of new gaming groups, and to remove or reduce barriers to entry for those gathering around a table for camaraderie. To accomplish these goals, we sustainably provide a service that is an ever-improving resource to the gaming community.

THE OPPORTUNITY

Roll20 is moving its real-time database workloads from Firebase to Lark, a wire-compatible, high-performance realtime database built in Rust and Go.

We are looking for a Senior Infrastructure Engineer to help design, build, and operate the GCP platform that will support this migration. This is a hands-on role with meaningful ownership: you will help translate Lark’s dedicated-hardware reference architecture into a reliable cloud-native production environment, make thoughtful trade-offs around availability, recovery, security, and cost, and operate the platform as it grows.

You’ll partner closely with the engineers building on top of the platform and share operational ownership with Roll20’s infrastructure team. If you enjoy solving complex systems problems, building dependable stateful infrastructure, and documenting decisions that make a team stronger, we’d love to hear from you.

WHAT YOU WILL DO

Design and build:

• Work with the team to translate Lark’s reference architecture into a GCP design: compute shapes and storage (including local NVMe options), VPC topology, DNS, and the client connection path.
• Evaluate the trade-offs above (HA approach, backup strategy, encryption posture) and turn them into concrete, documented decisions.
• Provision and automate production infrastructure using Ansible or comparable infrastructure-as-code and configuration-management tools.
• Build, test, and maintain backup and recovery processes, with clear recovery objectives and regularly validated restore paths.
• Establish end-to-end observability, including logging, metrics, dashboards, alerts, health checks, and operational reporting.
• Support the migration from Firebase Real-time Database through capacity planning, cutover rehearsals, rollback planning, and production readiness work.

Operate and improve:

• Share responsibility for production reliability, performance, and capacity with the infrastructure team.
• Debug issues across the stack, from client connection problems at the proxy layer down to WAL, compaction, and storage behavior on the database servers.
• Test recovery procedures on a regular cadence, whatever form they take (failover, snapshot restore, or backup replay).
• Manage capacity as usage grows, including adding servers, rebalancing databases, and tuning resource allocation
• Improve the platform’s security posture through access management, secrets rotation, OS patching, and IAM hygiene.
• Participate in a shared on-call rotation and create practical runbooks that make incidents easier to resolve.

WHAT WE ARE LOOKING FOR

• 5+ years of experience operating production Linux infrastructure, including storage, filesystems, networking, systemd, and performance tuning.
• Production experience with Google Cloud Platform, including Compute Engine, VPC networking, Google Cloud Storage, and IAM; equivalent cloud depth paired with strong GCP knowledge will also be considered.
• Hands-on experience running stateful services in production, such as databases, message queues, or storage systems, including replication, failover, and recovery.
• Strong Ansible skills, or comparable experience with configuration management and infrastructure-as-code.
• Real backup-and-restore experience: you’ve designed retention policies, tested restores, and can reason about RPO/RTO trade-offs.
• Proficiency scripting in Python, Bash, or Go.
• Sound technical judgment and the ability to balance robust design with appropriate operational simplicity.
• Clear written communication skills, including design documentation, runbooks, and incident writeups.
• Experience taking end-to-end ownership of production services, from architectural design and capacity planning through deployment, monitoring, and incident response.

NICE TO HAVE

• Ability to read and troubleshoot Rust and/or Go codebases.
• Familiarity with Firebase Real-time Database or other real-time sync systems using WebSockets, pub/sub, or delta protocols.
• Experience with block-level replication (DRBD or similar), disk encryption, or other techniques from the dedicated-hardware world, useful for judging what to keep and what to drop.
• Performance-engineering experience involving io_uring, NVMe tuning and network-stack tuning.
• Familiarity with Borg or other encrypted, deduplicating backup tools.
• Hands-on experience with Terraform.

WHY THIS ROLE IS INTERESTING

This role begins at the design stage. Rather than inheriting a fixed platform, you will help decide how Lark should run on GCP, build the systems that support it, and become one of the people who deeply understands its behavior in production. Your work will directly influence the reliability and responsiveness of real-time experiences for Roll20 users.

STARTING SALARY: $115,000-$125,000USD
SALARY EXEMPT – This position is paid salary and ineligible for overtime

If you are interested in applying, please send a Resume to jobs [at] roll20 [dot] net.