natbox.io

Blog / SSH report

Why some servers booted without SSH, and how we fixed it

14 November 2026 · Update · By Mohammad AlSaraf

On the afternoon of 14 November 2026 a customer opened a ticket: SSH to their new server answered "Connection refused", every time. The server was running and its network was fine, but nothing was listening on port 22. It wasn't anything they did. It was a mistake in our own Ubuntu 24.04 image, and it had quietly hit other servers too. This is what went wrong, why it only happened sometimes, and what we changed.

What customers saw

After a restart, some Ubuntu 24.04 servers came back with SSH switched off. The server itself kept running, and websites, VPNs and everything else on it kept working. Only logging in over SSH failed. The Console tab in the client area still worked, because it doesn't use SSH.

Most of the affected servers lost SSH during our scheduled maintenance restarts on the night of 13 to 14 November. One new server was affected from the moment it was created. When we checked port 22 on every server from our hosts, 18 of the 121 running Ubuntu 24.04 servers didn't answer. Servers on Debian 12, AlmaLinux 9 and Alpine were not affected.

systemctl status ssh.socket on a test server built from the old image: Active: inactive (dead). ss -ltn shows nothing listening on port 22.

Why it happened

Every natbox server starts from our own image. On first boot the server has to create its own SSH host keys, the keys that prove to your computer that it's talking to your server. Debian and Ubuntu don't create them by themselves on first boot, so we added a small startup step, natbox-hostkeys, that creates the keys before SSH starts.

The mistake was in how that step was ordered. We told systemd (the program that starts everything at boot) that our step must finish before SSH opens port 22. But systemd also adds default rules to every startup step, and one of them says it waits until the basic system is ready. On Ubuntu 24.04, SSH is started by a socket (ssh.socket), and sockets must be ready before the basic system counts as ready. Put together, that's a loop: SSH waits for our step, our step waits for the basic system, and the basic system waits for SSH.

The old natbox-hostkeys unit with Before=ssh.service ssh.socket, and systemd-analyze verify reporting an ordering cycle and deleting a job to break it.

systemd can't start a loop, so at boot it breaks it by dropping one of the steps. Which one it drops depends on timing, and that's why the problem was random. Usually the server came up fine. Sometimes the dropped step was the one that opens port 22, and the server booted without SSH.

Diagram of the loop: ssh.socket waits for natbox-hostkeys, which waits for basic.target, which waits for sockets.target, which waits for ssh.socket.

Why we didn't catch it sooner

When we test a new image, we create servers and log in to them, and that worked. The problem only showed up now and then: when we reproduced it afterwards on a test server, SSH was off on 1 boot out of 10 in one run and 1 out of 4 in another. A handful of test logins could easily all succeed, and they did. Our checks after the maintenance restarts confirmed that every server was running again, but not that SSH answered on each one.

What we fixed

The fix is small: our step now starts early, as soon as the disk is ready, instead of waiting for the basic system. That removes the loop, and SSH still never starts before the keys exist.

The fixed natbox-hostkeys unit with DefaultDependencies=no, After=local-fs.target and Before=sysinit.target, systemd-analyze verify finding 0 ordering cycles, and ssh.socket active (listening).

If you're ever locked out

The customer who reported this had already got back in on their own before we finished: they opened the Console tab in the client area and pressed Repair access, which turns off firewall rules, resets the SSH login settings and restarts SSH. That works whenever SSH stops answering, whatever the cause. See Secure your server in the docs.

In short

Get new tutorials in Telegram. Follow @natboxio: every new post, plus short server tips.

More from the blog: How much do natbox servers really use?, Keeping natbox clean: our first abuse and fraud report, We made disk writes on natbox over 100 times faster. Here's the trade-off. RSS feed.