Harden a new VPS in the first hour

Watch journalctl -u ssh on a server that has been online for an hour and the volume is alarming the first time you see it. It should not be. What you are looking at is the background radiation of the internet: a handful of scanning operations, some academic, some commercial, some criminal, that enumerate the entire IPv4 space continuously and hand the results to credential-guessing bots. Your address was reached because it exists, in numerical order, and it will be reached again in a few hours whatever you do.

This changes the shape of the problem in a useful way. You are not defending against an adversary who has chosen you and will adapt; you are defending against a script with a fixed repertoire — root, admin, ubuntu, test, git, oracle, postgres, and the twenty thousand passwords that have appeared in breach dumps. It has no patience, no creativity and no interest in a machine that does not answer on the first try. Turning off password authentication does not slow this attacker down. It removes it from the board entirely.

Two details are worth knowing. First, IPv6 is dramatically quieter, because a /64 cannot be swept — but the moment your AAAA record is public, or your machine's address appears in a mail header or a certificate-transparency log, the quiet ends. Never treat IPv6 as a hiding place; treat it as a smaller haystack. Second, an IPv4 address has a past. It had a tenant before you, and if that tenant ran a mail server badly or hosted something that got listed, you inherit the reputation until it decays. If mail matters to you, check the address on the usual blocklists before you build on it — that is a five-minute check that saves a fortnight of deliverability debugging.

Almost every real compromise of a small server begins in one of two places: a password that could be guessed, or a service that was listening and should not have been. The corresponding fixes are PasswordAuthentication no and a firewall with a default drop policy. They are unglamorous, they take fifteen minutes together, and they are worth more than every other control on this page combined.

The reason is structural rather than statistical. Both are closed by default — they fail toward safety. A keys-only sshd cannot be brute-forced no matter how many attempts arrive, because there is no path through the code that accepts a password. A default-deny firewall protects services you have not installed yet, including the database you will add in three months and forget to bind to localhost. Everything else in a hardening checklist is enumeration of badness: a list of specific things to switch off, which is only ever as complete as the list.

So if you read one section and close the tab, read this one, do steps 2 through 5 below, and consider the hour well spent. The rest is genuinely useful and genuinely secondary.

On Debian 13 and Ubuntu 24.04, /etc/ssh/sshd_config opens with an Include /etc/ssh/sshd_config.d/*.conf line, and cloud images ship a file in that directory — usually 50-cloud-init.conf — that already sets PasswordAuthentication. Editing the main file and appending your own directives at the bottom feels natural and produces a config that does not do what it says.

The rule is the one thing about sshd that surprises nearly everyone: for each keyword, the first value obtained wins. This is the opposite of almost every other config-merging system you have used. The Include sits near the top of the main file, so a drop-in is read before the body of sshd_config — and among drop-ins, alphabetical order decides. A file named 10-hardening.conf beats 50-cloud-init.conf. A file named 60-hardening.conf loses to it, silently, and you will not learn this from a restart that reports success.

Which is why you never trust the file you just wrote. Ask the daemon what it actually resolved:

sshd -t                 # syntax check — silence means valid
sshd -T | grep -Ei '^(permitrootlogin|passwordauthentication|pubkeyauthentication|kbdinteractive|allowusers|port) '

sshd -T prints the effective configuration after every include, override and default has been resolved. If that output says passwordauthentication yes, then password authentication is on, regardless of what any file on disk claims. Nothing else counts as verification.

The second trap is in the same family. Ubuntu 24.04 starts sshd through socket activation: ssh.socket owns the listening port, and Port in sshd_config is ignored entirely. Check with systemctl is-enabled ssh.socket; if it is enabled and you want a different port, change it with systemctl edit ssh.socket and a ListenStream= override — not in sshd_config, where the setting will look right and do nothing.

Let us be honest about this one, because the internet is not. Moving sshd to port 2222 or 47000 stops zero competent attackers. A full TCP scan of one host takes seconds, the service announces itself in the banner, and anyone who has decided to attack you specifically will find it before finishing their coffee.

What it does do is cut your authentication log by something like ninety-five percent, and that has real value: it is the difference between logs you skim and logs you actually read. Signal you can see beats signal you have buried. If you monitor anything at all, a quiet auth.log is the single cheapest way to make an anomaly visible.

The costs are small but they are real, and they are the reason we call it optional rather than recommended. You will forget the port when you come back in eight months. A non-standard port above 1024 can, in principle, be claimed by an unprivileged process if sshd is ever not listening. Restrictive outbound firewalls on client networks block odd ports, so you will occasionally be unable to reach your own server from an office or a hotel. And on a socket-activated system you have to change it in the right place, per the previous section. Do it if quiet logs matter to you. Do not do it and then feel secure, and never do it instead of keys-only.

A firewall that lists what to block is a filing system. A firewall that lists what to allow is a security control. The distinction is the entire game, because only the second one covers the service you install next month, the debug port you opened on a Friday, and the container that decided to expose itself.

On our platform you have two layers, and they are independent. The edge filter is optional, configured per server in the panel and enforced on the hypervisor, so blocked traffic never reaches your guest at all — useful for layer-4 rules you want applied before your kernel spends a cycle on them, and for keeping a compromised guest from being reachable while you work on it. The guest firewall is entirely yours: nftables, iptables, pf, whatever your image ships. We never touch it. Use both if the machine matters; use at least the second one always.

Two things in the ruleset in step 5 deserve explanation, because most copy-pasted rulesets get them wrong. Do not drop all ICMP. It feels tidy and it breaks path-MTU discovery, which produces the worst class of bug there is: small requests work, large responses hang, and nothing in your logs mentions the firewall. Accept at minimum destination-unreachable, time-exceeded and parameter-problem. Do not filter ICMPv6 by type unless you know the list. IPv6 relies on ICMPv6 for neighbour discovery and router advertisement; block it broadly and your IPv6 connectivity dies in a way that looks like a routing problem. Accepting all ICMPv6 on a single host is a reasonable trade and the ruleset below does exactly that.

One warning before you run flush ruleset: if Docker is installed, that line removes the NAT and filter rules Docker wrote, and container networking stops until systemctl restart docker puts them back. Load your ruleset first, restart Docker second, and read the next section before you publish a single container port.

Ask the machine what it is listening on. Not what you think you installed — what is bound right now:

ss -tulpn | grep -vE '127\.0\.0\.1|\[::1\]'

Everything that survives that filter is reachable from somewhere other than the machine itself. On a stock image the usual finds are rpcbind on 111 (nothing you run needs it), an exim4 listener left over from the base install, and a systemd-resolved stub that is harmless on loopback and an open resolver if it is not. The dangerous ones arrive later, with software you installed on purpose: PostgreSQL on 0.0.0.0:5432 because a tutorial said to edit listen_addresses, Redis with no password because Redis had no password by default for most of its life, an Elasticsearch node, a Prometheus exporter, a Jupyter notebook. Every one of those has been the first step of a real breach many times over.

Now the gotcha that catches careful people. Docker does not ask your firewall for permission. When you write -p 5432:5432, Docker inserts DNAT rules in the nat table's PREROUTING chain, which the kernel evaluates before your input chain ever sees the packet; the traffic is then forwarded to the container. Your default-deny input policy is not consulted, ufw status shows the port closed, and the database is on the public internet. This is documented Docker behaviour, it has been this way for a decade, and it has exposed an enormous number of databases.

The fix is one string, and it is the habit worth building:

# wrong — reachable from the entire internet, whatever your firewall says
docker run -p 5432:5432 postgres

# right — bound to loopback, reachable only through an SSH tunnel or a proxy
docker run -p 127.0.0.1:5432:5432 postgres

Finally, get an outside opinion. A port scan from the server tells you what the kernel thinks; a scan from somewhere else tells you what the world sees, which is the only number that matters. Run nmap -Pn -p- <your-ip> from your laptop, not from the machine, and compare the result with the list you expected.

The window that gets small servers compromised is not the zero-day. It is the four weeks between a CVE going public with a working proof of concept and the next time you happen to log in. Attack tooling absorbs a new remote bug within days; a machine you patch “when you get to it” is exposed for the entire interval, and every honest operator knows how long that interval really is.

The objection to automatic updates is fear of breakage, and it is worth taking seriously — then narrowing. Restrict the automation to the security pocket of your distribution, where maintainers backport fixes into the packaged version rather than shipping new upstream releases. A Debian security update to openssl is a patched build of the same version you already run; the risk of it changing behaviour is far smaller than the risk of leaving a remote hole open for a month. Feature upgrades stay manual, where they belong.

The part people skip is the reboot. A patched libssl on disk does nothing for the process that mapped the old one at start-up, and a kernel update does nothing at all until you boot into it. Either accept an unattended reboot in a window you choose, or install needrestart and let it tell you which services are running against deleted libraries — but do one of them. “Automatic updates are on” while uptime reads 340 days is a comfortable illusion, and it is the most common one we see.

fail2ban reads your logs, notices repeated failures from an address and bans it for a while. CrowdSec does the same and shares the verdicts across a network of participants, so you can block an address that has misbehaved elsewhere before it reaches you. Both are good software. Neither is doing what most people think on a keys-only SSH server.

Once PasswordAuthentication is no, an SSH brute-force cannot succeed. Not “is unlikely to” — there is no code path. Banning an address after five failures therefore prevents nothing; it reduces log volume and a trivial amount of CPU. That is a real benefit, it is just not a security benefit, and the trade is not free: a badly written jail that watches the wrong log has locked more administrators out of their own servers than it has ever locked attackers out.

Where these tools genuinely earn their place is one layer up, on the services you actually expose: a login form, a WordPress admin, an API with rate limits, a mail server with SMTP AUTH. Those do accept passwords, they are brute-forceable, and a ban list is exactly the right control. So our verdict is narrow and specific. Skip the SSH jail. Put CrowdSec or fail2ban in front of the thing that has a password field. And whichever you choose, whitelist your own management address first — ignoreip exists for the evening you will otherwise remember for the wrong reasons.

Prevention is what you can do in an hour. Detection is what tells you the hour was not enough. It does not need to be elaborate, and on a single server three cheap things cover most of the ground.

Keep the logs. On many images the journal lives in /run and evaporates at reboot, which means the record of an incident disappears in the reboot that follows it. Creating /var/log/journal is a one-line fix and the highest-value thing in this section.

Get told about logins. A line in /etc/ssh/sshrc that fires logger on every session start costs nothing and gives you a clean, greppable record separate from sshd's own chatter. One caveat that catches people: sshd runs ~/.ssh/rc instead of /etc/ssh/sshrc when the user has one, so the system-wide file is skipped for exactly the account most likely to have a customised dotfile. If you need a hook that cannot be shadowed, use pam_exec instead.

Know what the filesystem looked like on day one. AIDE records hashes of your binaries, libraries and unit files and reports what changed since. The catch is fundamental and usually ignored: a database stored on the machine it is watching can be regenerated by whoever compromised it, and then the check reports “no changes” forever. Copy the database off the box, or at minimum record its hash somewhere else, and the tool becomes worth its twenty minutes. Left in place, it is a comfort blanket.

The general principle is worth stating on its own: the log you cannot trust is the one on the compromised machine. Anything you genuinely intend to rely on — a journal, an integrity database, a backup — should have a copy somewhere the same attacker does not control. A second small server in another jurisdiction is a legitimate answer to that, and a five-dollar one is enough.

Everything above is perimeter work, and it is worth being precise about the boundary so you do not mistake a locked front door for a safe.

It does not protect data on a running machine. A hardened kernel and a closed firewall are irrelevant to anyone with hypervisor access, and to the contents of RAM while the machine is up. If your concern is the disk when the server is off, seized or decommissioned, that is disk encryption, and it is a different procedure with a different failure mode — see encrypting a VPS disk with LUKS.

It does not make you anonymous. A server can be flawlessly hardened and still announce who owns it, through a WHOIS record, a reused SSH key, an analytics tag, an SSH client that connects from home without a hop, or a certificate that ties two identities together. Paying with Monero and then logging in from your own address undoes the payment. That failure mode has its own page: the mistakes that deanonymize you.

It does not fix your application. An SQL injection, an unauthenticated admin endpoint or a dependency with a backdoor does not care about your sshd settings. Most compromises of well-run servers arrive through the port you opened deliberately, on purpose, to a service you wrote or installed.

It is not a backup. Ransomware, rm -rf with a variable that was empty, and a failed upgrade all end the same way. Take a copy, put it somewhere else, and restore from it once before you need to — the same discipline that makes a move to another host survivable makes a bad Tuesday survivable.

None of this is an argument against the hour. It is an argument for knowing exactly what the hour bought.