Linux remote desktop server: X11VNC and RustDesk daemon setup

You're trying to let people (or yourself) connect to a Linux desktop reliably, securely, and without bouncing through expensive proprietary services — and the manuals you've found are either too hand-wavy or written for GUI wizards. This guide shows two practical server-side approaches for a linux remote desktop server: a tried-and-true X11VNC setup for directly exposing an X session, and a self-hosted RustDesk daemon (hbbs/hbbr) for modern NAT traversal and relay — with real commands, systemd units, Docker examples, and concrete hardening steps.

Why run a linux remote desktop server? Quick decision tree

Before diving into commands: pick a model that fits your needs.

• If you need direct access to the physical X session on a machine (supporting the logged-in user, local session state, multiple monitors), X11VNC is the simplest server-side tool.
• If you want a client/server model that supports NAT traversal, ID servers, relays, and easier cross-platform clients — and you want to self-host those server components — run RustDesk's hbbs/hbbr daemon(s).
• If you just want a quick single-machine tunnel for one-off support, an SSH tunnel or using a hosted service may still be faster. See also our guide on self-hosted remote desktop for strategy and tradeoffs.

Note: commercial products like TeamViewer and AnyDesk often win on pure convenience (automatic NAT traversal and optimized codecs out of the box). They're reasonable choices if you need plug-and-play reliability and commercial support; see our comparison for feature tradeoffs in rustdesk-vs-anydesk.

1) X11VNC: a minimal linux remote desktop server exposing the physical X session

X11VNC connects to an existing X server and serves the current desktop. It’s not a separate virtual desktop — it mirrors the locally logged-in GUI. That makes it excellent for remote support and administration where you want to interact with the same session a local user sees.

Prerequisites and recommended versions

• Works on X11-based desktops. For Wayland, use compositor-specific remote APIs or a different approach.
• Install x11vnc >= 0.9.16 (0.9.16+ supports modern features and stability improvements).
• Make sure you have a display manager (gdm/lightdm/sddm) or an X session running on :0.

Install on Debian/Ubuntu (example):

sudo apt update
sudo apt install -y x11vnc xauth

Create a password file (store it securely):

sudo x11vnc -storepasswd /etc/x11vnc.pass
sudo chown root:root /etc/x11vnc.pass
sudo chmod 600 /etc/x11vnc.pass

Simple systemd unit for auto-starting on display :0 (place as /etc/systemd/system/x11vnc.service):

[Unit]
Description=x11vnc - VNC server for :0
After=display-manager.service

[Service]
Type=simple
ExecStart=/usr/bin/x11vnc -auth guess -forever -loop -noxdamage -repeat -rfbauth /etc/x11vnc.pass -rfbport 5900 -shared -ncache 10
User=root
Restart=on-failure

[Install]
WantedBy=graphical.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now x11vnc.service
sudo systemctl status x11vnc.service

Security notes for X11VNC

• Do not expose TCP/5900 directly to the internet without additional protections. VNC authentication is okay for LAN use but should be treated as weak over public networks.
• Prefer an SSH tunnel for remote access: ssh -L 5900:localhost:5900 user@your-server and then connect a VNC client to localhost:5900.
• If you need direct TLS, use stunnel or a VPN. Or place VNC behind a proper firewall and require VPN access.

Performance tips

• Use -ncache 10 to reduce bandwidth spikes when desktop content changes rapidly.
• When you see high CPU on 1–2 Mbps links, reduce color depth or use compression flags (x11vnc supports -compresslevel and -quality). Experiment — lower quality often yields better perceived responsiveness.

2) RustDesk daemon: self-hosted relay and ID server for modern remote access

RustDesk provides a client that can use a central ID server (hbbs) and a relay (hbbr) to connect peers even behind NAT. Running your own hbbs/hbbr gives you full control over IDs and relays — important if you want to avoid third-party servers. This is the server-side setup most people mean when they ask for a linux remote desktop server in a self-hosted model.

Why run hbbs/hbbr instead of a single binary? Hbbs is the ID server (authentication, assignment), hbbr is the relay server (TCP/UDP relay for media when direct P2P fails). Both are lightweight and commonly run in Docker.

Recommended versions: use rustdesk server components 1.1.9+ (or the latest stable tag at the time you deploy). Review release notes on the RustDesk project before production rollouts.

Example Docker Compose for hbbs + hbbr (minimal)

version: '3.3'
services:
  hbbs:
    image: rustdesk/rustdesk-server:1.1.9
    container_name: rustdesk_hbbs
    restart: unless-stopped
    ports:
      - "21115:21115/tcp"  # hbbs TCP (ID server)
    environment:
      - RUST_LOG=info
    volumes:
      - ./data/hbbs:/data

  hbbr:
    image: rustdesk/rustdesk-server:1.1.9
    container_name: rustdesk_hbbr
    restart: unless-stopped
    ports:
      - "21116:21116/tcp"  # hbbr TCP (relay)
      - "21116:21116/udp"  # hbbr UDP for hole punching/relay
    environment:
      - RUST_LOG=info
    volumes:
      - ./data/hbbr:/data

Notes on ports and NAT

• Default RustDesk ports commonly mapped are 21115 (hbbs ID server) and 21116 (hbbr relay). UDP is useful for NAT traversal; open both TCP and UDP where possible.
• Keep the server on a public IP or on a host with static IP/DNS. Use an A/AAAA record for the hostname you configure in clients.

Configuring the client side

Point the RustDesk client to your hbbs hostname and enable relay as needed. You can force relay usage for privacy or allow peer-to-peer if both clients can punch through NAT. The client configuration UI accepts your server’s hostname and port (e.g., server.example.com:21115).

Securing a self-hosted RustDesk daemon deployment

RustDesk's default traffic is encrypted between peers, but the server components authenticate and coordinate. Consider these hardening steps:

• Run hbbs/hbbr behind a firewall and only open necessary ports (21115 TCP, 21116 TCP/UDP). Use UFW or firewall-cmd; example: sudo ufw allow 21115/tcp; sudo ufw allow 21116/tcp; sudo ufw allow 21116/udp.
• Use TLS/HTTPS for any web-facing admin UI you add. If you terminate TLS at a reverse proxy (nginx/caddy), keep the backend isolated.
• Monitor logs and resource usage. RustDesk components are lightweight but you’ll want to watch connection counts and relay bandwidth.
• Consider rate-limiting and fail2ban on the host if you see brute-force attempts against the hbbs port.

When to pick RustDesk vs X11VNC

• Pick RustDesk when you need cross-platform client support (Windows/Mac/Linux/Android/iOS), NAT traversal, and a single self-hosted ID/relay for many endpoints. It’s a modern solution for distributed fleets.
• Pick X11VNC when you're servicing specific desktop machines and need to interact with the local X session (for example, troubleshooting the logged-in user or graphical boot issues).

Practical production notes and performance tuning

Bandwidth and CPU: expect that direct remote desktop sessions consume between 1–5 Mbps for typical office tasks with compressed codecs; screen-sharing of video or games will spike much higher. If you host your own relay (hbbr), budget for relay bandwidth: 100 concurrent sessions at 2 Mbps = ~200 Mbps continuous capacity.

Monitoring and autoscaling: for larger organizations, run hbbs with a small HA proxy or load balancer in front, and run multiple hbbr relays distributed across regions. Use standard container orchestration (Docker Swarm or Kubernetes) if you need auto-scaling; otherwise, a single beefy relay VM is fine for small teams.

Logging and troubleshooting

• x11vnc logs appear on systemd journal: sudo journalctl -u x11vnc.service
• RustDesk containers: docker logs rustdesk_hbbs and docker logs rustdesk_hbbr for startup errors. Check port reachability with ss or netstat and test UDP/TCP from a remote client.
• If direct connections fail, confirm UDP is not being blocked by intermediate NATs or firewalls; many carriers block uncommon UDP ranges.

Security comparison and vendor candid view

If security and privacy are your top priorities, self-hosting hbbs/hbbr gives you control of metadata and relay endpoints. However, proprietary vendors like TeamViewer or AnyDesk may provide stronger out-of-the-box NAT traversal, proprietary codecs with lower bitrates for poor links, and enterprise-grade support/SLAs. They can be better if you need guaranteed 24/7 commercial support and easier onboarding for non-technical users — but that convenience comes at a cost. See anydesk-pricing-explained for pricing differences and tradeoffs.

Practical checklist before going live

1. Decide which model suits you (X11VNC for physical sessions vs RustDesk for ID/relay-based remote access).
2. Harden the server: firewall, SSH keys only, fail2ban rate-limiting, and TLS where applicable.
3. Test from outside your network: verify relay behavior, latency, and failover.
4. Set monitoring (logs, bandwidth, process restarts) and an alert policy for outages.

Further reading and internal resources

If you're evaluating broader self-hosted options and tradeoffs, read our self-hosted guide at /self-hosted-remote-desktop. For a focused feature comparison between RustDesk and commercial options, see /rustdesk-vs-anydesk.

Final practical notes

Both approaches are maintainable, but they solve slightly different problems. X11VNC is simple and predictable for single desktops; RustDesk daemon(s) scale better for fleets and handle NAT traversal gracefully when configured correctly. In all cases, never expose unencrypted VNC directly to the internet — always use SSH tunnels, VPNs, or a properly hardened relay.

Ready to try it yourself? Download Tenvo (godeskflow) clients or check our server docs at /download — and if you need pricing or enterprise options, look at /pricing. Deploy a test instance, exercise your firewall rules, and validate client behavior before rolling out to users.