Arroyo.cc / plan9 / 9grid bringup

This is not a guide; the 9front FQA is a good starting point for anyone looking to get started with 9front. Rather, it is a diary of my efforts to run 9front in a way that integrates well with my existing network, which is mostly running Linux.

Pre-setup inventory and planning

I have two machines; a 4-core fanless intel n150 mini-pc, and a beefier 8-core AMD workstation with 5TB of SSD storage. I have a laptop as well, which I just plan to use as a terminal. I am using Guix SD, a Linux distribution, across all my PCs. I plan to run my grid as a set of VMs across these machines.

My networking setup is a little unique, and is described here. I've setup each machine with its own IPv6 network, and some daemons to allocate interfaces with addresses in that network, and associate those addresses with domain names, all on demand. So each VM is reachable over the internet, with its own (public) domain name.

Here is the initial setup I'm going for:

auth fs cpu desktop auth fs cpu minipc (baremetal)

This setup is kind of overcomplicated; I could colocate the cpu/fs/auth roles within the same VM. But I want to get some practice managing multiple Plan 9 systems. I also want to be able to turn off my desktop when I'm away, or to take it down to upgrade or replace hardware, or to just terminate what's running on to free resources for gaming. So there's 2 of everything. I keep the 2 auth servers in sync manually when I add or remove a user.

I am running 9front directly on the mini server, since I don't intend to do anything on it that requires Linux and it's good to use real hardware as a reference. The rest of this post will cover the setup of the VMs on my workstation; the bare metal machine's setup is pretty much what you would get from following the 9front FQA.

Once the file server and auth servers are setup, I don't intend to touch them very often; since everything depends on them, it's better if they don't go down. On the other hand, the CPU server will be updated frequently; I might reboot it repeatedly while working on the kernel. I will also have a way to spin up copies of the CPU server, either to test out a patch, or experiment with a new configuration.

For this post, I will focus just on setting up the VMs on my desktop. My configuration is a GUIX program, so it would be straightforward to replicate the configuration on another system after I get it working. I haven't looked into how to keep the auth servers in sync.

Setting up the file server

I start with the file server, with very few deviations from the FQA. Here is the expression in my server config for the file server:

(runvm 'ultan
  #:cores "2"
  #:mem "1G"
  #:disks '("cache=none,aio=native,file=/dev/mapper/crypt9,format=raw")
  #:provides '(runvm-fs)
  #:requires '(ipmux-external phonebook)
  #:publish "/run/phonebook/publish"
  #:network "/run/ipmux/external")

A lot of the complexity is hidden away in my Guix channel, but this gets me a qemu VM with the suggested shape, that starts up at boot, with direct access to one of my block devices. The VM gets two DNS records: ultan.internal and ultan.ymar.aqwari.net, where ultan.internal is a ULA address, routable only within the host system, and ultan.ymar.aqwari.net is the global ipv6 address assigned to the VM from my ISP's prefix. The global zone is delegated from a public DNS domain I own.

The disk is an entire SSD that I've encrypted with LUKS, which is opened when the host system boots. 9front does have whole-disk encryption support, which I may switch to using in the future. Doing it this way has the advantage that I can't accidentally boot the host system into 9front (some may see that as a disadvantage!).

The VM has files under the /run directory that I can use to connect to its serial console, a graphical display, and the qemu monitor. For the initial setup, I connected to the qemu monitor to load up the 9front install CD:

$ screen /run/runvm-fs-mon
(qemu) change ide1-cd0 /tmp/9front.iso
(qemu) system_reset

From there, I followed the normal install process, choosing gefs as my file system, and accepting the defaults elsewhere. I used MBR, because it is simpler to boot a qemu VM from disk using qemu's BIOS.

Enable authentication

To serve 9P requests over the network, I first need to enable authentication. The first step is to put credentials for the "hostowner" in nvram, which is just a small partition near the beginning of the disk, which the kernel reads on boot:

% auth/wrkey
bad nvram des key
bad authentication id
bad authentication domain
authid: glenda
authdom: aqwari.net
secstore key: <enter>
password: hunter2
confirm password: hunter2
enable legacy p9sk1[no]: no

Serve 9P to the network

The file server (gefs) is running, but it's only accepting local requests. I need to configure a listener for 9P requests. To do that, I make this server a cpu server:

% 9fs 9fat
% echo 'service=cpu' >> /n/9fat/plan9.ini
% echo 'console=0' >> /n/9fat/plan9.ini
% fshalt -r

Adding console=0 lets me use the serial console, which qemu attaches to a pty, using screen. This has the benefit over VNC of being able to copy/paste between the VM and host machine. However, the cpu profile also starts a listener for rcpu(1), so I can connect with drawterm, which is even better:

drawterm -u glenda -h ultan.internal -a ultan.internal

there I can start rio(1) and get a graphical display with clipboard integration and easy access to the host file system under /mnt/term. I have to make sure to make the file server's devices visible if I want to make changes to plan9.ini:

cpu% bind -a '#S' /dev
cpu% 9fs 9fat
cpu% sam /n/9fat/plan9.ini

The next thing to do is replace bootargs:

bootargs=local!/dev/sdC0/fs -A

with

nobootprompt=local!/dev/sdC0/fs -a tcp!*!564

so it will boot without prompting, and instruct gefs to receive authenticated sessions from the network on the well-known 9p port. After a reboot, the file server is ready to use, for the hostowner.

I don't want to login as the hostowner; while the hostowner is much less privileged than the unix root superuser, it still has the ability to read anything on the file system. I can add a user account to the file server by following gefs(8):

ultan# mount /srv/gefs /adm adm
ultan# sam /adm/users

The file /adm/users will look something like this:

-1:adm:adm:glenda
0:none::
1:tor:tor:
2:glenda:glenda:
10000:sys::glenda
10001:map:map:
10002:doc::
10003:upas:upas:glenda
10004:font::
10005:bootes:bootes:

The file format is documented in users(6). I can add a line for my user:

10006:myuser:myuser:

and I'll add my user to the upas and sys groups, so I can create email queues (upas) and perform some administrative tasks like updating the system (sys):

10000:sys::glenda,myuser
10003:upas:upas:glenda,myser

after saving the file, I need to tell gefs to reload it:

ultan# con -C /srv/gefs.cmd
gefs# users
refreshed users

and follow the remaining commands in gefs(8) to create home and tmp directories for my user. Now I have my own account, but I cannot use it, because I can't authenticate; I need to add a user with the same name to the auth server.

Setting up the auth server

I create another VM:

(runvm 'talos
  #:kernel (file-append 9front-amd64 "/amd64/9pc64")
  #:cores "1"
  #:mem "1G"
  #:disks '("cache=none,file=/var/lib/runvm/p9auth.qcow2")
  #:provides '(runvm-p9auth)
  #:requires '(ipmux-external phonebook)
  #:publish "/run/phonebook/publish"
  #:network "/run/ipmux/external")

This is a little different from the file server; instead of booting from a disk device, it boots from a qcow2 image. The auth server needs a small amount of storage to store user credentials; it's not intended for use as a general file server. It could be diskless, using the file server for storage, but then there would be little point to separating it, from a security perspective.

After following the normal install procedure, configuring nvram with the hostowner's credentials, I need to convince this VM that it is, in fact, an auth server. Because this is a single-purpose VM, and it won't be sharing its files with any other, I can get away with simply adding auth=talos to its plan9.ini file and rebooting. I can confirm that the auth server is running after reboot:

talos# ps -a | grep auth
glenda          302    0:00   0:00      124K Await    listen [tcp * /rc/bin/service.auth]

now I can add a user:

talos# auth/keyfs
talos# auth/changeuser myuser

the final task is to instruct the file server to use this auth server.

Configuring /lib/ndb/local

The network database (see ndb(6)) is like DNS on steroids; it lets you define arbitrary tuples in a hierarchical set of files starting with /lib/ndb/local. If a tuple contains an ip address and netmask, it will match any system with an address (as seen in the /net/ndb file) matching that prefix.

When a system boots, it will consult the network database to find the name or address of the auth server(s), performing this query:

% ndb/query -cia sys $sysname auth

that is, it searches for any auth keys in tuples that contain sys=$sysname, or any tuples that match addresses learned from tuples that match sys=$sysname. I can read /net/ndb to find any addresses that were added by slaac or dhcp:

ultan# cat /net/ndb
ip=fd64:cafe::746a:ff:fe55:babe ipmask=/64 ipgw=::
	sys=ultan
	dns=fd64:cafe::53

ip=2603:7000:cafe:babe:746a:ff:fe55:babe ipmask=/64 ipgw=fe80::beef:20ff::cafe
	sys=ultan
	dns=fd64:cafe::53

Since multiple systems will eventually use this file system and read the same /lib/ndb/local file, it would be good to define the auth server in a way that applies to all the systems in my grid. I can do something like this:

ipnet=nyc1 ip=2603:7000:cafe:ba00:: ipmask=/56 ipgw=fe80::beef:20ff::cafe
	authdom=aqwari.net auth=talos.ymar.aqwari.net

The name of the ipnet, as far as I can tell, is for documentation; I'm giving a short, loosely location-based name. If I try running the query for the auth server, I should get results now:

ultan# ndb/query -i sys $sysname auth
talos.ymar.aqwari.net

At this point I can log onto the file server with my new user:

drawterm -h ultan.ymar.aqwari.net -a talos.ymar.aqwari.net -u myuser

On first login, I run /sys/lib/newuser which sets up my home directory and an email queue. The file server is good to go, but I want a separate CPU server that I do most of my work on.

CPU server setup

I'll add a diskless VM which uses my file server for storage. First I have to create a new user for the cpu server to use; we'll call it cpu0, and use the same name for its hostname, as well. I create the cpu0 user in the same way I created myuser, with its own, unique password. Then I can create the VM:

  (runvm 'cpu0
         #:kernel (file-append 9front-amd64 "/amd64/9pc64")
         #:initrd (plan9.ini
                   #:sysname "cpu0"
                   #:auth "talos.internal"
                   #:fs "ultan.internal"
                   #:nobootprompt "tcp!-6 ether /net/ether0 ra6 recvra 1"
                   #:service "cpu")
         #:cores "4"
         #:mem "8G"
         #:provides '(runvm-cpu0)
         #:requires '(runvm-p9auth runvm-fs ipmux-external phonebook)
         #:publish "/run/phonebook/publish"
         #:network "/run/ipmux/external")

I'll find the boot process held up at the following prompt:

authid:

Since it's diskless, the VM has no stored credentials. I can fill out the prompt with the credentials I created for the cpu0 user, and the boot process will complete. However, I want this VM to boot without intervention, so I 'll create a small drive to use as nvram:

# umask 077
# truncate -s 1M /var/lib/runvm/cpu0.nvram

Due to the sector size on my encrypted block device, the smallest it can be is 1M, but usually the nvram partition is only 512 bytes. I'll add it to my VM definition:


(runvm 'cpu0
       #:kernel (file-append 9front-amd64 "/amd64/9pc64")
       #:initrd (plan9.ini
                 #:sysname "cpu0"
                 #:auth "talos.internal"
                 #:fs "ultan.internal"
                 #:nobootprompt "tcp!ether /net/ether0"
                 #:service "cpu")
       #:disks '("cache=none,format=raw,file=/var/lib/runvm/cpu0.nvram")
       #:cores "4"
       #:mem "8G"
       #:provides '(runvm-cpu0)
       #:requires '(runvm-p9auth runvm-fs ipmux-external phonebook)
       #:publish "/run/phonebook/publish"
       #:network "/run/ipmux/external")

then, from within the VM, after answering the auth/wrkey prompt, I can partition the disk to add an nvram partition:

init: starting /bin/rc
talos# disk/mbr -m /386/mbr /dev/sdC0/data
talos# disk/fdisk -baw /dev/sdC0/data
talos# disk/prep -bw -a nvram /dev/sdC0/plan9
nvram 1
talos# auth/wrkey
...

After filling out the credentials for the hostowner and rebooting, the auth server starts up and mounts the file server without intervention.

As I'm doing this, I'm keeping in mind that I want to be able to create VMs on-demand; I think, for now, for those short-lived VMs, it would be acceptable to fill in the boot prompt with my own credentials, but in the future it would be nice to be able to pass a connection to factotum(4) into the VM somehow, where I can pre-load credentials for the VM to use.

At this point I can start adding users to my auth and file servers. I'll start with two: one for myself, and one for the CPU server to access the file server on boot.

A note

I recognize that this setup is incredibly complicated. I am still figuring things out, and admittedly doing a bit of role-playing to get a feel for what it would be like to manage a larger grid of plan 9 machines.