SSH refused, emergency mode, a locked root account. Why it happens on Compute Engine, and three ways out.
Your VM says RUNNING in the console, and SSH is refused:
$ gcloud compute ssh gce-rescue-test --zone=us-east1-b
ssh: connect to host 35.227.x.x port 22: Connection refused
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].ssh: connect to host 34.73.x.x port 22: Connection refused
The serial console shows why:
You are in emergency mode. After logging in, type "journalctl -xb" to view
And it won’t let you in to fix it:
Cannot open access to console, the root account is locked.
SSH was refused, not timed out, so the VM’s network answered and nothing was listening on port 22. The boot stopped before sshd started.
First, is it a boot problem?
Check the end of the serial output:
$ gcloud compute instances get-serial-port-output $VM --zone=$ZONE | tail -30
- If it ends at a normal login prompt: The OS booted successfully. Your issue is on the network side (VPC Firewalls, SSH keys, OS Login, or IAP configurations). Try running gcloud compute ssh $VM –troubleshoot.
- If it ends in an error: Errors likeemergency mode, grub rescue>, or a kernel panic, the guest isn’t finishing its boot.
The failure: one bad line in /etc/fstab
To reproduce a common failure, add an fstab entry for a disk that doesn’t exist and reboot. Use a throwaway VM, because this makes it unreachable:
gcloud compute instances create gce-rescue-test --zone=us-east1-b --machine-type=e2-small --image-family=debian-12 --image-project=debian-cloud
gcloud compute ssh gce-rescue-test --zone=us-east1-b --command "echo 'UUID=deadbeef-1234-5678-9abc-def012345678 /data ext4 defaults 0 2' | sudo tee -a /etc/fstab && sudo reboot"
SSH drops with exit code 255 on the reboot. About two minutes later, the serial console shows this:
[ TIME ] Timed out waiting for device dev-disk-by-uuid-deadbeef-1234-5678-9abc-def012345678.
[DEPEND] Dependency failed for data.mount - /data.
[DEPEND] Dependency failed for local-fs.target - Local File Systems.
You are in emergency mode. After logging in, type "journalctl -xb" to view
Without nofail, systemd treats the /data mount as required and waits up to 90 seconds for the device. When that times out, local-fs.target fails and systemd drops to emergency.target, which starts a sulogin shell. sulogin asks for the root password, and Google’s public images ship with root locked.
Three ways out
1. Fix it from inside, through the serial console. With a root password, or single-user mode by editing the kernel line in GRUB, you can repair configuration problems in place. Google’s fstab troubleshooting guide walks through it. It needs interactive serial access, varies by distro, and doesn’t help once the bootloader or initramfs is the problem.
2. Fix it from outside, with a rescue VM. Google’s boot troubleshooting guide sends most failures (GRUB, initramfs, filesystem corruption, kernel panics) this way. You snapshot the disk, create a temporary VM, attach the broken disk, mount it, chroot in and fix it, then swap the disk back and clean up. It works for almost anything, but it’s a dozen-plus manual steps across two VMs while something is down.
3. Automate rescue VM method. GCE Rescue does the same disk swap on the same VM, scripted end to end, with no second instance. For the common failures, that comes down to two commands, diagnose and repair. Here's how it works.
What GCE Rescue does to your VM
GCE Rescue has four commands:
- diagnose reads the serial console and identifies why the VM won’t boot. It’s read-only.
- repair diagnoses the problem and fixes it automatically.
- rescue swaps in a rescue disk with your original disk mounted, so you can investigate and fix it yourself over SSH or RDP. It changes which disk is attached, never the disk’s contents.
- restore reverses rescue: it puts your fixed boot disk back and starts the VM.
repair chains the others together: it runs diagnose, then rescue, applies the fix automatically, and finishes with restore. Here's what that does to your VM:
- Checks permissions, org policy and Local SSDs while the VM is still running, so a failure here costs nothing.
- Stops the VM, snapshots the boot disk and detaches it.
- Attaches a temporary rescue disk as the boot disk (Debian 12 by default on Linux, a matching image for Windows and Arm VMs, or your own with –rescue-image).
- Boots the VM, attaches your original disk as a secondary disk, mounts it at /mnt/sysroot and applies the fix from the startup script. No SSH is needed.
- Stops the VM, puts your original disk back as the boot disk, restores your metadata, starts the VM and deletes the rescue disk.
If a step fails, the earlier ones are rolled back.
Walkthrough on the broken VM
Install in Cloud Shell. You need roles/compute.instanceAdmin.v1 on the project (roles/compute.viewer is enough to diagnose).
pip install --upgrade gce-rescue
Diagnose (read-only)
$ gce-rescue diagnose gce-rescue-test --zone=us-east1-b
Diagnosis: gce-rescue-test (us-east1-b)
Status: RUNNING
OS: Linux (debian-12-bookworm, x86_64, Free)
Result: 2 issues found (1 critical, 1 error)
CRITICAL [FSTAB] UUID specified in /etc/fstab cannot be found
Identifier: UUID=deadbeef-1234-5678-9abc-def012345678
Serial console:
[ 3.850139] systemd[1]: Expecting device dev-disk-by-uuid-deadbeef-1234-5678-9abc-def012345678.de... <--
ERROR [SYSTEMD_EARLY] Emergency shell unreachable - root account is locked (no root password)
Serial console:
Cannot open access to console, the root account is locked. <--
See sulogin(8) man page for more details.
To fix these issues:
Auto-repair (recommended):
$ gce-rescue repair gce-rescue-test --zone=us-east1-b
Or fix manually:
(rescue, repair boot configuration, restore: steps omitted)
diagnose reads the serial console and matches it against known failure patterns. The <– marks the log line each finding is based on, and the VM is never touched. The second finding, the locked root account, is a consequence of the first. repair flags it as needing manual repair, but it goes away once the VM boots.
Repair
$ gce-rescue repair gce-rescue-test --zone=us-east1-b
Repair plan:
1. Create backup snapshot of boot disk
2. Enter rescue mode (stop VM, swap boot disk)
3. Fix /etc/fstab (comment out UUID=deadbeef-1234-5678-9abc-def012345678)
4. Restore original boot disk and start VM
Proceed? [y/N]: y
(1/3) Rescue: Stopping VM -> Creating snapshot -> Creating rescue disk -> Starting rescue VM -> Mounting disk done.
(2/3) Repair: Applying fix -> Verifying fix done.
(3/3) Restore: Stopping VM -> Restoring boot disk -> Starting VM done.
Repair results:
[FIXED] fstab: Commented out entry for /data (deadbeef-123...)
1 issue fixed.
Original fstab backed up to: /etc/fstab.gce-repair-backup
Backup snapshot: pre-rescue-gce-rescue-test-1792940000
Repair complete. Instance [gce-rescue-test] is now running. (7m 32s)
Boot verification: VM is booting normally.
The process took 7m 32s, most of it Compute Engine stopping and starting the VM.
Verify
$ gcloud compute ssh gce-rescue-test --zone=us-east1-b --command "uptime && cat /etc/fstab && ls -l /etc/fstab.gce-repair-backup"
15:02:00 up 4 min, 0 user, load average: 0.05, 0.05, 0.01
PARTUUID=1d18xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx / ext4 rw,discard,errors=remount-ro,x-systemd.growfs 0 1
PARTUUID=4ec1xxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /boot/efi vfat defaults 0 0
# GCE-REPAIR: commented out entry matching diagnosed error
#UUID=deadbeef-1234-5678-9abc-def012345678 /data ext4 defaults 0 2
-rw-r--r-- 1 root root 243 Oct 25 14:56 /etc/fstab.gce-repair-backup
SSH works again. Only the bad line changed, and it was commented out, not deleted.
It’s not just fstab
Four failures have auto-repair on Linux today:
- Bad /etc/fstab entry (emergency mode, “Dependency failed for /data”): comments out only the matching entry, after backing up the file.
- Broken GRUB (a missing grub.cfg, a grub rescue> prompt): regenerates grub.cfg and reinstalls the bootloader on Debian/Ubuntu and RHEL-family systems.
- Missing or corrupt initramfs (Kernel panic – not syncing: VFS: Unable to mount root fs): rebuilds it for the kernel that failed, and falls back to the previous kernel if that fails. Not attempted for LVM or LUKS roots.
- Full boot disk (No space left on device): resizes the disk (+5 GB by default) and grows the filesystem, deleting nothing.
Beyond those, diagnose recognizes 21 Linux failure categories (kernel, storage, services, resource problems) and common Windows boot errors, even where no repair exists. You still get the root cause, the evidence line and manual fix steps.
What to know before running it
- Your ephemeral external IP can change. Because the VM is stopped during the swap. Reserve a static IP first if anything depends on it.
- There’s downtime of several minutes.
- Local SSD data is lost when a VM stops. The tool asks before continuing, and –quiet mode refuses unless you pass –force.
- Confidential VMs aren’t supported.
- Restricted image policies can block the default rescue image. The tool stops before touching the VM and tells you which –rescue-image to use.
- Snapshots aren’t deleted or labeled. List them with gcloud compute snapshots list –filter="name~^pre-rescue-". I had 43 left from testing.
When there’s no auto-fix
gce-rescue rescue $VM --zone=$ZONE # swap in a rescue disk
gcloud compute ssh $VM --zone=$ZONE # your disk is at /mnt/sysroot
gce-rescue restore $VM --zone=$ZONE # put your fixed disk back
If you already know the fix, repair –fix-script=fix.sh runs your own script against the mounted disk, then restores and verifies. On Windows, repair needs –fix-script.
Next time a VM won’t boot
Start with diagnose. It's read-only, takes seconds, and tells you which of the three paths you're on. The source, issues and contributions live in the GitHub repo. New detection patterns are picked up automatically, so a pull request for a failure it missed is a small change.
GCE Rescue is open source and not an officially supported Google Cloud product, so there’s no support contract behind it. The views here are my own.
GCP VM Won’t Boot? How to Fix It Without a Rescue VM was originally published in Google Cloud – Community on Medium, where people are continuing the conversation by highlighting and responding to this story.
Source Credit: https://medium.com/google-cloud/gcp-vm-wont-boot-how-to-fix-it-without-a-rescue-vm-7810498151e0?source=rss—-e52cf94d98af—4
About the Author
