; Date: Sun May 03 2020

Tags: Ubuntu »»»» Multipass »»»» Windows 10

Multipass is an excellent tool for running Ubuntu instances on a host system. Builds are available for Linux, macOS and Windows, and it is claimed to work well on all three. In my case, I spent three days solid trying to launch an Ubuntu instance and nearly gave up in frustration. But I found a simple problem, with a simple solution, and along the way learned about Hyper-V, WSL2, and other deep stuff about Windows.

My task is to update my book, Node.js Web Development, for the fifth edition. This edition is focusing on changes related to Node.js 14.x. But this book has always shown how to deploy Node.js apps on a server, using Ubuntu as the example. For this release it seemed the easiest way to experiment in that area is with Multipass, since it promises to be an excellent way to have a full Ubuntu server environment without leaving ones laptop.

While preparing for the fifth edition I heard about WSL2, the second version of the Windows Subsystem for Linux. Microsoft makes WSL2 out to be a major advance over WSL, and to be an excellent way of running Linux instances on a Windows developer machine. That may be - but my experience over the last few days has shown me WSL2 is at best not ready for prime time (e.g. it requires setting Windows 10 to the Insiders Program, and to receive early access updates, which means your machine could easily break) because it is lacking some important features and reliability.

To learn about WSL - About Windows Subsystem for Linux

To learn about WSL2 - About WSL2

This post isn't a comparison about Multipass versus WSL2, it's how to debug Multipass on Windows. Those links will teach you about WSL2. My point for bringing it up is to describe the beginning state of my laptop:

• Windows 10 Pro
  • On the Insiders Program - on the Fast Ring - because that was described as the way to install WSL2
• Docker for Windows installed but disabled
• Windows Features
  • To access the screen for enabling or disabling Windows features, in the task bar search area enter "Windows Features", and that gives you access to a window showing all possible features
  • Hyper-V is enabled - both WSL2 and Multipass use Hyper-V
  • The Virtual Machine Platform optional component is enabled in Windows Features
  • During the time I was actively tinkering with WSL2, the Windows Subsystem for Linux feature was enabled

When the Windows Subsystem for Linux feature is enabled, Windows will install the related code and make WSL2 features available. That means the wsl.exe command line tool and other things. Similarly when that feature becomes disabled all those things are removed.

The problem with running Multipass on Windows

That's the system onto which I tried to install Multipass. Windows 10 Pro, Insiders Program, Fast Ring, with WSL2 installed.

I had previously installed Multipass on my macOS laptop, and had a joy filled time easily installing Ubuntu instances and playing with deploying the application from my book. So I approached installing Multipass on Windows with the same excitement. But ...

After installing Multipass, and eagerly typing multipass launch to set up an Ubuntu instance, after a long wait, this was printed:

PS C:\Users\heavy> multipass launch
launch failed: The following errors occurred:
unassuming-tailorbird: timed out waiting for response

I tried finding advice - other people facing the same problem - etc. The other things I found suggested that Multipass was not compatible with the Windows Subsystem for Linux v2 (WSL2) because of something about how WSL2 deals with the Hyper-V virtualization engine. Supposedly the Docker team also had problems, until they reworked Docker for Windows to use Hyper-V in a way that was compatible with what WSL2 does with Hyper-V.

My troubleshooting process is documented in " Multipass Launch timed out waiting for response on Windows 10 #1512" on Github. What follows is a whole lot of details, including at the end detailed notes from troubleshooting.

Summary of observations

The core problem is the timeout shown above. Most of the time when this happened, the instance showed as Running in multipass list but there was no IP address assigned nor was it possible to use multipass shell to enter the instance. Further, multipass stop would hang for a long time, the longest recorded time being two hours, without result.

One finding is that in some instances there was accompanying corruption of C:\WINDOWS\System32\drivers\etc\hosts.ics. This file is somewhat like the /etc/hosts file on Unix/Linux/macOS systems meaning it shows IP addresses and host names. It is autogenerated by some background process. The corruption was spurious extra text that did not constitute a well formed entry in the file.

Another finding is that at one time I had Multipass working fine, whith WSL2 disabled. Then as soon as WSL2 was enabled problems started cropping up. In other words, using WSL2 with Multipass is definitely not recommended. Since Multipass is superior to WSL2 it's not much loss to leave it disabled.

The last finding is that the Ubuntu 18.04 image works great, the Ubuntu 20.04 image always caused timeouts and failures. It took a long time to realize this was a problem. But, I had this result:

• Multipass was up and running fine, no hosts.ics corruption, etc
• Successfully launch an 18.04 instance,
• Fail at launching a 20.04 instance,
• Successfully launch an 18.04 instance,
• Fail at launching a 20.04 instance,
• Fail at stop a previously failed 20.04 instance

Fail in this case means having the Multipass command result in the timeout message. This sequence showed that Multipass was freely able to launch Ubuntu 18.04, even after failing to launch Ubuntu 20.04.

I do not have records that previous failures to launch an instance were on 20.04 or 18.04. My notes do not include that detail.

A reliable cure

Uninstall Multipass - telling the Uninstall wizard to not delete instances and data.

Delete C:\WINDOWS\System32\drivers\etc\hosts.ics

Run this:

Get-HNSNetwork | ? Name -Like "Default Switch" | Remove-HNSNetwork

Then reboot the system, and reinstall Multipass.

Hyper-V default switch, Hyper-V switches

One suggestion was

• Check that the Multipass VM is connected to the default switch
• Do you have other Hyper-V virtual switches?

Not knowing anything about Hyper-V - I only knew of it as a virtualization service for Windows, akin to HyperKit on macOS - I learned that it's much more. Seems that Hyper-V is a powerful system for managing virtual machines and has lots of interesting features.

To get visibility into this, launch the Hyper-V Manager -- you'll have to search for it in the task bar search box. This tool gives you visibility into the virtual machines running on your current machine, and possibly it can also manage Hyper-V instances on other machines.

In the Actions sidebar there is a choice Virtual Switch Manager. Click on that and you can view the Virtual Switches, and administer them.

A Virtual Switch seems to be akin to an ethernet hub (or switch) but is a software emulation of that idea. It seems that you can create multiple virtual networks inside your computer, with different access rights, etc. For example you might have a database service for which access must be strictly limited, and you can implement the necessary barriers using Hyper-V Virtual Switches.

In my case all that was in the default state. Namely, a a single Virtual Switch with the default setup.

In the Multipass guide for troubleshooting on Windows, this command was suggested:

Get-HNSNetwork | ? Name -Like "Default Switch" | Remove-HNSNetwork

This must be executed on an Administrator-Priveleged PowerShell instance. It forcibly deletes some aspect of the default virtual switch. After running this, reboot the computer, and supposedly Windows will automatically recreate whatever-that-is. In my case this made no difference.

Using Hyper-V to create a normal virtual machine

I did not know this - but Hyper-V is cable of running virtual machines just like VirtualBox or Parallels or VMWare. Namely, you can download an operating system ISO image, and install it on a Hyper-V virtual machine, and have a running computer.

Hyper-V even has a Quick Create option that lets you set up an Ubuntu virtual machine.

I did this using both the Quick Create option, as well as the download-ISO-from-ubuntu.com. Both resulted in a running Ubuntu VM with no difficulty. It worked as advertised, but the performance was very low.

In other words, even though getting Multipass to run reliably is very difficult, using a regular VM works very well (other than performance).

Using WSL2 to set up an Ubuntu instance

I also explored using WSL2 to set up the application shown in my book. Remember that the task on hand was to document exploring Node.js service deployment on Ubuntu by running a machine emulator on a laptop.

Again, this is not about comparing WSL2 versus Multipass. But I do have to say that WSL2 has serious deficiencies in this regard. Bottom line is that what is delvered for WSL2 is a very incomplete Ubuntu instance. The biggest flaw is that SystemD is not running, and therefore you cannot deploy a persistent background task. While you can run background tasks, they do not automatically execute when the host computer (or the WSL2 subsystem) reboots.

Uninstalling WSL2

Since WSL2 is a Windows Feature, it is not an application that can be installed and uninstalled.

If it's correct that WSL2 and Multipass are incompatible, then it seemed necessary to uninstall WSL2.

How that works is to go into Windows Features and deselect the Windows Subsystem for Linux choice. Once you do that Windows reboots, and during the reboot it is on the blue screen doing Windows updates. After rebooting, the wsl.exe and other WSL2 features are missing.

Hyper-V still exists, and you can use it either directly or via Multipass.

Multipass long pause and timeout on various operations

For many of the Multipass commands, what's actually happening is an API request to a virtualization server. On Windows that virtualization server is Hyper-V.

When this is printed:

PS C:\Users\heavy> multipass launch
launch failed: The following errors occurred:
unassuming-tailorbird: timed out waiting for response

Before that message there are several messages printed by Multipass as it is setting up the machine instance. It might download an image from the Ubuntu server, and certainly it will unpack the image, and configure the image for use. At each stage Multipass prints messages, and shows a twirling indicator. In every case I observed, the message was printed after Multipass had shown Starting for a long time. After the Starting state is shown, multipass launch shows another state, whose name I do not remember, then after that the instance is properly launched. In other words, when this timeout happens it's because the Starting state takes a long time and then times out.

Also in multipass list the machine that was attempted to be launched is listed in the Running state. But it is not actually Running but in an almost running state.

There are other commands like multipass stop, multipass delete, and multipass purge, that also require interaction with a virtualization server.

Getting timeout does not always mean a problem occurred

Consider:

PS C:\Users\heavy> multipass list
Name                    State             IPv4             Image
focal                   Running           UNKNOWN          Ubuntu 20.04 LTS
svc-notes               Stopped           --               Ubuntu 18.04 LTS
svc-userauth            Stopped           --               Ubuntu 18.04 LTS
PS C:\Users\heavy>
PS C:\Users\heavy> multipass start svc-userauth
start failed: The following errors occurred:
svc-userauth: timed out waiting for response
PS C:\Users\heavy> multipass list
Name                    State             IPv4             Image
focal                   Running           UNKNOWN          Ubuntu 20.04 LTS
svc-notes               Stopped           --               Ubuntu 18.04 LTS
svc-userauth            Running           UNKNOWN          Ubuntu 18.04 LTS
PS C:\Users\heavy> multipass list
Name                    State             IPv4             Image
focal                   Running           UNKNOWN          Ubuntu 20.04 LTS
svc-notes               Stopped           --               Ubuntu 18.04 LTS
svc-userauth            Running           172.21.196.116   Ubuntu 18.04 LTS
PS C:\Users\heavy> multipass shell svc-userauth
Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-99-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
...

It's an 18.04 instance and it started up, but the startup timed out. Problem? I waited long enough, and it got into a correctly running state with an IP address assigned and I could shell into the instance.

During this process I had a Hyper-V Connect window attached to the instance. It was showing the normal Linux startup messages. Eventually the startup messages finished, and I was greeted with a normal Login prompt at which time multipass list showed the IP address above.

This instance has a couple file system mounts that need to be there. But upon shelling into the instance, the mount directories are empty indicating the mounts failed.

The Hyper-V Connect window

Since Hyper-V is a general purpose virtual machine manager, it has a window (the Connect window) with which one can connect to the console of virtual machine. If the VM is booted into a graphical system, we instead see the desktop.

To start this - in Hyper-V, select the VM and in the Actions side-bar some actions will appear for the selected VM. One of those is Connect. Click on that and a window will pop up.

This window may be useful for debugging issues in Multipass/Hyper-V. For example in the previous section we were able to see that the Multipass instance was booting normally, but it was simply taking a long time.

Corruption in hosts.ics file

A key file to Hyper-V is C:\WINDOWS\System32\drivers\etc\hosts.ics. This file is kind of like /etc/hosts on Unix/Linux in that it lists IP addresses and host names.

The Multipass troubleshooting guide for Windows discusses the possibility of extra entries in this file. What I found is it included spurious extra text.

# Copyright (c) 1993-2001 Microsoft Corp.
#
# This file has been automatically generated for use by Microsoft Internet
# Connection Sharing. It contains the mappings of IP addresses to host names
# for the home network. Please do not make changes to the HOSTS.ICS file.
# Any changes may result in a loss of connectivity between machines on the
# local network.
#

172.20.32.1 HEAVY.mshome.net # 2025 5 5 2 16 11 42 960

5 95

Notice this file has a DO NOT EDIT warning. It is auto-generated by certain software supplied by Microsoft, and clearly editing this file could cause the early heat death of the Universe. Or some other badness could result.

The first non-comment entry is the information for my laptop. It is an HP EliteBook 2650p that, for a 13" laptop, is very heavy, so I called it HEAVY.

But notice that extra bit of text.

This was the cure for the problem. Namely,

• Make sure that WSL2 is disabled
• Make sure to uninstall Multipass
• Delete that file then reboot the computer.

After reboot the file was still missing. The file was recreated automatically when the normal VM (the one created for Ubuntu) was launched, at which time hosts.ics again existed.

Once I got the normal VM launched correctly, I had enough confidence to install Multipass again. And trepeditously I typed this command:

PS C:\Users\heavy> multipass list
No instances found.
PS C:\Users\heavy> multipass launch
Launched: set-suricate

Sure enough, the launch command made it all the way to having a completely booted instance. Excited, I asked to launch a second instance, which also worked, and then I checked the hosts.ics file to find this:

172.20.32.1 HEAVY.mshome.net # 2025 5 5 2 16 14 25 208
172.20.45.220 set-suricate.mshome.net # 2020 5 0 10 16 11 42 960
172.20.32.171 svc-notes.mshome.net # 2020 5 0 10 15 55 35 914
172.20.45.200 svc-notes2.mshome.net # 2020 5 0 10 16 14 25 208

By the way - those domain names are real names that can be used to access the virtual machine. In any case, the important thing to notice is that all entries are well formed and are correct.

The entries for set-suricate and svc-notes2 were added by Multipass, the entry for svc-notes was added when launching the Ubuntu VM with Hyper-V.

Further experimentation

After getting it to a working state, I wanted to explore further and see what's what.

In the previous section, I'd gotten the system to a state where multipass launch could readily start new instances.

So I enabled WSL2 again. Then I launched Ubuntu under WSL2, and hosts.ics became corrupted.

Then attempting to launch a Multipass instance got back to the timeout problem. The result was like this:

PS C:\Users\heavy> multipass launch
launch failed: The following errors occurred:
unassuming-tailorbird: timed out waiting for response

Disabled WSL2, and rebooted. THere was a blank line in hosts.ics Running multipass list took a long time and didn't finish. So I edited hosts.ics to clean up the file, and rebooted.

Found hosts.ics to have extra text. Edited it to remove the extra text. Running multipass stop -- the goal would have been to delete the half-launched instance from the previous attempt -- this took a long time, so I instead rebooted the machine.

Found corruption in hosts.ics. Uninstalled Multipass, telling it to not delete the existing machines and settings. Rebooted, again found corruption in hosts.ics. Edited the file again, and rebooted.

Installed Multipass, and it found the previous instances. Could readily stop/delete/purge the instances that had been halfway installed. This indicates it could readily access the Hyper-V service and do things.

However - in the process hosts.ics became corrupted. The corruption was a blank line before the block of hosts, and some spurious text afterward. Attempted to launch a new instance, and got a timeout.

Uninstall Multipass again - telling it to not remove the instances and settings.

hosts.ics has a blank line both before the block of hosts, and afterwards. Rebooted, and hosts.ics still in that form.

Installed Multipass again. Readily shows the list of instances, readily allows stop/delete/purge existing half-installed instances.

hosts.ics has a blank line both before and after the block of hosts.

Rebooted without editing hosts.ics, and after reboot it has excess text:

# Copyright (c) 1993-2001 Microsoft Corp.
#
# This file has been automatically generated for use by Microsoft Internet
# Connection Sharing. It contains the mappings of IP addresses to host names
# for the home network. Please do not make changes to the HOSTS.ICS file.
# Any changes may result in a loss of connectivity between machines on the
# local network.
#

172.21.128.1 HEAVY.mshome.net # 2025 5 5 2 22 59 50 156
192.168.82.98 set-suricate.mshome.net # 2020 5 0 10 22 25 21 876
172.20.32.171 svc-notes.mshome.net # 2020 5 0 10 15 55 35 914
192.168.92.220 svc-notes2.mshome.net # 2020 5 0 10 22 25 17 413
413

There's a blank line before the block of host names. Maybe that blank line is okay? Don't know. But the 413 at the end is clearly spurious, and is possibly related to the 413 on the end of the previous line.

With it in this state - multipass list works readily. Running multipass start for an existing machine that had been properly installed, that worked readily, and then multipass shell to access that machine works readily.

However, hosts.ics gets even more corrupt:

# Copyright (c) 1993-2001 Microsoft Corp.
#
# This file has been automatically generated for use by Microsoft Internet
# Connection Sharing. It contains the mappings of IP addresses to host names
# for the home network. Please do not make changes to the HOSTS.ICS file.
# Any changes may result in a loss of connectivity between machines on the
# local network.
#

172.21.128.1 HEAVY.mshome.net # 2025 5 5 2 23 3 11 956
192.168.82.98 set-suricate.mshome.net # 2020 5 0 10 22 25 21 876
172.20.32.171 svc-notes.mshome.net # 2020 5 0 10 15 55 35 914
172.21.140.75 svc-notes2.mshome.net # 2020 5 0 10 23 3 11 956
3
413

At that point I tried multipass launch and got a timeout as shown previously. Tried multipass launch using a different instance name, and that too got a timeout.

Uninstalled Multipass w/o removing instances. Deleted hosts.ics. The thinking is that hosts.ics is thoroughly bad, and the way I'd fixed it before was to delete this file and reboot. Rebooted.

After reboot hosts.ics has a single entry like so:

# Copyright (c) 1993-2001 Microsoft Corp.
#
# This file has been automatically generated for use by Microsoft Internet
# Connection Sharing. It contains the mappings of IP addresses to host names
# for the home network. Please do not make changes to the HOSTS.ICS file.
# Any changes may result in a loss of connectivity between machines on the
# local network.
#

172.21.128.1 HEAVY.mshome.net # 2025 5 5 2 23 3 11 956

Note there is a single blank line before and after the block of hosts. Maybe it is okay to have blank lines, but not okay to have other spurious text??

Installed Multipass. Running multipass list runs okay.

Running multipass start for an instance that had been properly installed runs okay. The distinction is that a properly installed instance is one that successfully ran multipass launch, whereas a partially launched instance is one where multipass launch timed out.

However, running multipass launch for an instance that was partially launched (remember - previously I tried to launch two instances and got the timeout) did not work, gave a timeout.

After getting that timeout, used multipass launch on another instance that had been properly launched, and that ran fine, and was able to multipass shell to access the instance.

Wanting to delete the instances that had partially launched, I tried multipass stop on one, but that took a long time. So I rebooted while it was still running.

After reboot hosts.ics looks okay:

# Copyright (c) 1993-2001 Microsoft Corp.
#
# This file has been automatically generated for use by Microsoft Internet
# Connection Sharing. It contains the mappings of IP addresses to host names
# for the home network. Please do not make changes to the HOSTS.ICS file.
# Any changes may result in a loss of connectivity between machines on the
# local network.
#

172.22.176.1 HEAVY.mshome.net # 2025 5 5 2 23 58 43 100
172.22.178.171 set-suricate.mshome.net # 2020 5 0 10 23 58 43 100
172.22.181.190 svc-notes2.mshome.net # 2020 5 0 10 23 58 41 298

Note there is a blank line before the block of hosts.

multipass list still shows two instances that were incompletely launched. Trying to use multipass stop on one, and this takes a long time. I've let it run for about an hour with no response. So have deleted hosts.ics and rebooted.

multipass list shows readily, has two partly launched instances, one of which is shown in Running state. But multipass shell to that instance fails. This is a common occurrence, that list shows it Running but it never completed its setup.

Ran: Get-HNSNetwork | ? Name -Like "Default Switch" | Remove-HNSNetwork and deleted hosts.ics and rebooted

After reboot multipass list takes a long time and there is no hosts.ics. Started Hyper-V and it shows the expected VM's in a Saved state. After it shows the VM's transition to a Running state, multipass list finishes, and hosts.ics appears. Presumably a background process was busy generating that file.

Tried to multipass stop to prepare to delete/purge. But this hung for a long time.

Several reboots ensued. The overall attempt was to Remove-HNSNetwork, delete hosts.ics and to uninstall Multipass. However for every attempt to uninstall Multipass and delete all the instances and config data, those attempts paused for a long time and never finished the uninstall.

Another request that hung for a long time was multipass stop on a partially installed instance. In one case I left that running for 2 hours before giving up.

Eventually - I tried

• Multipass uninstall w/o delete instances and data
• delete hosts.ics
• Remove-HNSNetwork

After reboot there was excess text in hosts.ics.

# Copyright (c) 1993-2001 Microsoft Corp.
#
# This file has been automatically generated for use by Microsoft Internet
# Connection Sharing. It contains the mappings of IP addresses to host names
# for the home network. Please do not make changes to the HOSTS.ICS file.
# Any changes may result in a loss of connectivity between machines on the
# local network.
#

172.27.224.1 HEAVY.mshome.net # 2025 5 6 3 6 27 33 38
172.27.225.15 set-suricate.mshome.net # 2020 5 1 11 6 27 32 211
172.27.238.127 svc-notes2.mshome.net # 2020 5 1 11 6 27 33 38
 8

But, I used notepad to clean that up. Then installed Multipass again.

After that was able to stop/delete/purge the partially installed instances.

Running multipass launch (default image means Ubuntu 18.04) and this succeeded.

Running multipass launch specifying focal as the image (20.04) failed on timeout.

Subsequent to that ran multipass launch (not specifying an image, means default of Ubuntu 18.04) and that succeeded.

Trying again to multipass launch specifying focal failed.

Was able to freely create and delete images using the 18.04 base image, but every attempt to do anything with 20.04 images resulted in timeouts.

No hosts.ics corruption.

Was unable to remove the partly completed instances - all of which were based on 20.04 image.

Therefore, rebooted the computer to see how it would behave after reboot.

After reboot no hosts.ics corruption.

The 20.04 instances showed in a Suspended state. Was able to stop/delete/purge them to remove them.

Was able to freely create 18.04 instance, except for a strangeness of this:

PS C:\Users\heavy> multipass launch --name svc-userauth
launch failed: The following errors occurred:
svc-userauth: timed out waiting for response
PS C:\Users\heavy> multipass list
Name                    State             IPv4             Image
svc-userauth            Running           172.29.201.106   Ubuntu 18.04 LTS
PS C:\Users\heavy> multipass shell svc-userauth
Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-96-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

 System information disabled due to load higher than 1.0

0 packages can be updated.
0 updates are security updates.



The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@svc-userauth:~$

Namely, The launch showed a timeout, but list showed it in Running state and there was an IP address assigned. And, shell let me into the instance. So it worked even though it showed the timeout?