Linux Remote Desktop Server: X11VNC & RustDesk Setup

You're trying to manage or support Linux machines remotely and tired of brittle ad-hoc solutions — SSH for shell access, copying large files manually, or sending someone a TeamViewer link every time. If you want a persistent, server-side remote desktop on Linux that starts at boot, survives reboots, and can be self-hosted behind your control, this tutorial walks through two practical server-side approaches: X11VNC for classic X11 sessions and the RustDesk server daemon for a modern, self-hosted relay/rendezvous option.

When to run a dedicated linux remote desktop server (and why)

A quick checklist to decide if a server-side remote desktop makes sense:

• You need headless or unattended access to a machine (lab servers, office desktops, kiosks).
• You want a single, always-on endpoint you can connect to without asking someone to launch a client first.
• You prefer self-hosting (no third-party cloud) or want a local relay to avoid exposing RDP/VNC ports directly.
• You want to combine classic X11 VNC access with modern NAT traversal/relay for client convenience.

X11VNC is a small, mature, server-side VNC daemon that exports whatever's on the X11 display (commonly :0). RustDesk's server components (hbbs + hbbr) provide the rendezvous and optional relay for peer-to-peer connections — useful when clients sit behind NAT. Both can coexist: X11VNC gives you an always-on VNC endpoint, and RustDesk gives you a managed way for remote clients to find your host without port-forwarding.

Option A — X11VNC: stable, simple, server-side X11 access

Use X11VNC when your machines run an X11-based desktop and you want a straightforward VNC server that starts on boot. X11VNC is battle-tested (common stable release: x11vnc 0.9.16 in many repos) and integrates nicely with systemd.

Install and secure x11vnc

On Debian/Ubuntu:

sudo apt update
sudo apt install -y x11vnc

Create a password file (use a strong passphrase). Replace 'remote' with the remote user's home directory owner.

sudo -u remote mkdir -p /home/remote/.vnc
sudo -u remote x11vnc -storepasswd /home/remote/.vnc/passwd
sudo chown -R remote:remote /home/remote/.vnc

Find the correct X authority file for your display manager. Common locations:

• LightDM: /var/run/lightdm/root/:0
• GDM (GNOME): /run/user/1000/gdm/Xauthority or check /home//.Xauthority

Start x11vnc manually once to validate:

sudo -u remote x11vnc -display :0 -auth /home/remote/.Xauthority -rfbauth /home/remote/.vnc/passwd -forever -shared -noxdamage -o /var/log/x11vnc.log

Systemd unit for an always-on service

Put this file at /etc/systemd/system/x11vnc.service — edit the User, Group, and -auth path to match your distro/display manager.

[Unit]
Description=x11vnc server for display :0
After=graphical.target

[Service]
Type=simple
User=remote
Group=remote
ExecStart=/usr/bin/x11vnc -display :0 -auth /home/remote/.Xauthority -rfbauth /home/remote/.vnc/passwd -forever -shared -noxdamage -repeat -o /var/log/x11vnc.log
Restart=on-failure

[Install]
WantedBy=graphical.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now x11vnc.service
sudo journalctl -u x11vnc -f

Network and security considerations

VNC by default is unencrypted. Options to harden a server-side VNC endpoint:

• Bind to localhost and require SSH tunneling: run x11vnc with -rfbport 5901 and use systemd to only listen on 127.0.0.1, then SSH -L 5901:localhost:5901.
• Use a VPN to access the host's LAN.
• Limit access with a firewall (ufw example below).
• If you need direct remote clients without SSH, put VNC behind an stunnel/NGINX TLS proxy (adds CPU and complexity).

# Basic UFW rule to allow local-network VNC only
sudo ufw allow from 192.168.0.0/16 to any port 5900 proto tcp
# Or bind to localhost and tunnel via SSH for remote access

Notes: X11VNC requires an X11 session. On Wayland (GNOME on some distros) use Wayland-compatible servers (e.g., wayvnc) or the desktop's built-in remote desktop (often RDP).

Option B — RustDesk server daemon: self-hosted rendezvous and relay

RustDesk gives you the ability to self-host the signaling (hbbs) and relay server (hbbr) so clients can find and reach your hosts without exposing raw VNC/RDP ports. If you already run X11VNC for the desktop session, you can front it with RustDesk for NAT traversal and an easier client experience. RustDesk server components are commonly packaged as docker images; check the project releases — example server tags include v1.2.0 (verify current tag on the RustDesk repo).

Simple Docker Compose example

This compose spins up hbbs (rendezvous) and hbbr (optional relay). The ports shown are common defaults used in community docs (adjust if upstream changes ports).

version: '3.7'
services:
  hbbs:
    image: rustdesk/rustdesk-server:latest
    container_name: rustdesk-hbbs
    restart: unless-stopped
    ports:
      - '21112:21112/tcp'   # rendezvous
    environment:
      - HBBS_KEY=your_secret_key_here

  hbbr:
    image: rustdesk/rustdesk-server:latest
    container_name: rustdesk-hbbr
    restart: unless-stopped
    ports:
      - '21113:21113/udp'   # relay

Notes:

• Replace HBBS_KEY (or other env vars per the current RustDesk instructions) with a secure value.
• RustDesk's official images and env var names change between releases — consult the RustDesk server repo before production.

Connecting clients

On the client side (RustDesk desktop/mobile), point the client to your hbbs server address (DNS name or public IP): e.g., 1.2.3.4:21112. If hbbr relay is available and needed, the client will use it to pass traffic when direct (P2P) fails. You can then configure the client to either remote-control a RustDesk agent running on the host or use RustDesk as a broker that connects to an existing VNC service on the host (for that you typically run the RustDesk agent on the host, which can in turn forward to the X11VNC session).

Systemd alternative to Docker

If you prefer not to use Docker, build rustdesk-server binaries following the project's documentation and install them as systemd services (hbbs and hbbr). Packaging varies by release; the Docker approach is the quickest way to get a reproducible server running.

Security, NAT traversal, and when to avoid exposing ports

Two high-level approaches to avoid exposing desktop ports directly:

1. Keep VNC/RDP bound to localhost; require SSH/VPN to reach the host. This is the simplest, most auditable option for single-admin setups.
2. Self-host a relay/rendezvous (RustDesk) and use TLS + authentication. This reduces open ports on the host but requires running and securing the relay server(s).

Firewall snippets (UFW):

# Allow only SSH from your office and block the rest
sudo ufw allow from 203.0.113.0/24 to any port 22 proto tcp
sudo ufw deny 5900/tcp

# If running RustDesk server on the relay box (example)
sudo ufw allow 21112/tcp
sudo ufw allow 21113/udp

Practical security checklist:

• Use strong authentication for the VNC or RustDesk agent account.
• Rotate or protect server keys (RustDesk HBBS key) and keep images up-to-date.
• Use IDS/monitoring to alert on port scans and failed logins.
• If you require encrypted desktop sessions, terminate TLS at a reverse proxy (Nginx/Caddy) in front of the relay and enforce TLS 1.2+ and strong ciphers.

Operational tips, troubleshooting, and maintenance

Common issues and fixes:

• No desktop visible over VNC: confirm the X display is :0 (ps aux | grep X) and that x11vnc uses the correct -auth file.
• Service won't start at boot: set the systemd WantedBy to graphical.target and confirm the display manager starts before x11vnc.
• RustDesk clients can't reach the server: confirm DNS and firewall; test with telnet/IP tools and inspect container logs (docker-compose logs -f).
• Performance is poor: enable -noxdamage for x11vnc (less tearing, lower CPU for some workloads) and consider adjusting compression/encodings on the client when available.

Maintenance playbook:

• Apply OS security updates weekly. On Debian/Ubuntu you can automate unattended-upgrades for minor patches.
• Track the RustDesk or x11vnc upstream repos for security fixes. If you use docker images, schedule an image refresh and redeploy pipeline.
• Backup configuration files and any TLS certs; store HBBS keys in a secrets manager if possible.

When a commercial tool or RDP might be the better choice

Honest trade-offs:

• TeamViewer / AnyDesk: They win at extreme ease-of-use for non-technical users, universal NAT traversal, and polished mobile apps. If you need instant, zero-ops support for hundreds of non-technical endpoints, a commercial SaaS may be worth the cost. See our comparison at rustdesk-vs-anydesk for specifics.
• RDP (Microsoft Remote Desktop): On Windows servers and desktops, native RDP usually gives better performance and features (clipboard, file transfer, sound). But RDP exposes higher-risk attack surface if not behind VPN or a bastion.

If your primary goal is self-hosting and privacy — and you are okay with a bit more initial setup and ongoing maintenance — the X11VNC + RustDesk server combination is a strong, practical approach.

Further reading and internal resources

If you want to avoid port-forwarding entirely, read our walkthrough: Remote desktop without port forwarding. For a higher-level view of deploying your own solution, see Self-hosted remote desktop guide. For security hardening best practices, check Remote desktop security.

Finally, Tenvo is focused on open, self-hosted remote desktop tooling — if you want an alternative client/server that is designed for self-hosting and cross-platform use, check our downloads or pricing pages to get started: /download and /pricing. We describe similar deployment patterns in other posts and keep examples current.

If you want a hand with a specific distro, display manager, or to tune a systemd startup for a particular environment, tell me the distro and display manager (e.g., Ubuntu 22.04 with GDM) and I’ll give you a tailored unit file and auth-path commands. When you're ready, download Tenvo or try building the stack described above — start at /download.