FreeBSD jails. Alternative to Oracle VM VirtualBox

Posted on

Just recently I wrote about my frustration with developing this website on my laptop. Apple’s changes to the operating system and my tendency to experiment with various —apparently at times conflicting— tools and libraries introduce problems that get in the way when the job at hand is simply writing a new blog post. My solution was the creation of a virtual machine using Oracle VM VirtualBox and Ubuntu as the host system. That worked well, but soon a question came to my mind: “Can we have more fun with a virtual machine?” The answer is yes. Read on.

A Mac Mini Waiting to Be Put to Good Use

Every day I walk by a 2012 Mac Mini sitting on a shelf in my home. It is not supported by Apple anymore, but it is still a capable computer that was just waiting to be put to use. Installing an operating system with native support for virtualized environments seemed like a good opportunity to try out an alternative to an Ubuntu based development environment running inside Virtual Box. The question is: Which operating system to choose?

One possible operating system is the OmniOS Community Edition. I used it before on a —now defunct— server. You can read about it on this blog. Both, Linux and FreeBSD are viable choices as well; they all support virtualization to create a separate environment where my website development tools can live independent of anything else installed.

Let’s go on a tangent: ZFS

My choice to use OmniOS in 2016 was solely influenced by its stellar support for ZFS[1]. That was then, now is now and the ZFS landscape has changed. Today OpenZFS is the de-facto umbrella organization driving the development of ZFS. Different operating systems base their ZFS implementation on the OpenZFS version, albeit with different levels of integration.

Famously, Linux does not integrate ZFS into the kernel. A decision that stands, until as Linus Torvald puts it, he gets “an official letter from Oracle that is signed by their main legal counsel or preferably by Larry Ellison himself that says that yes, it’s ok to do so”. The main problem is an incompatibility between the license that Linux is based on (GPL 2.0) and the license that was used when ZFS was open sourced (CDDL 1.0). You would think that wouldn’t make much of a difference, until you learn that Oracle pulled ZFS back into closed source. Add a heavy dose of expensive lawyering and you have the threat of a lawsuit.

In 2016 Ubuntu has deployed their own lawyers to conduct a legal review and concluded they are free to include ZFS. However, as of this writing OpenZFS notes that “The Ubuntu installer still has ZFS support, but it was almost removed for 22.04”. Others are wondering whether there is a future for ZFS on Ubuntu. I guess if you want unconditional support getting a dog is still the only way to go? Or, for the problem at hand we could turn to the FreeBSD Foundation who proclaims in a post in 2016 that they are “happy that ZFS is available in FreeBSD as a fully integrated, first class file system”. “Fully integrated” here means that there is support for ZFS boot environments. I experienced the virtues of beadm(1) when I was running OmniOS. The ability to simply snapshot the state of the current boot environment before an upgrade took a lot of worries away as botched upgrades are easily undone. Ubuntu requires me to use the —now apparently unsupported— zsys desktop installer or a third-party application like ZFSBootMenu; less than ideal. FreeBSD it is!

FreeBSD Virtualization: Jails or bhyve

Although bhyve, short for “BSD Hypervisor” made it into the title of this subsection, it wasn’t much of a contender. It allows you to install a virtual environment that runs a different operating system as the host. A FreeBSD host running Microsoft Windows in a virtual environment? Sure. FreeBSD for the ZFS support but you are more comfortable running Linux? Again, bhyve is the way to go. bhyve on FreeBSD is very much equivalent to Oracle VM VirtualBox on the Mac. The main disadvantage being the amount of resources required. A separate operating system needs more memory, more disk space, more cpu power. All I need is a lightweight virtualization that keeps my website stuff isolated. That’s what FreeBSD’s jails do. Easy choice.

The first Jail

I installed FreeBSD on the Mac Mini; a straightforward process outlined here: Chapter 2. Installing FreeBSD | FreeBSD Documentation Portal. Besides a few tools (ssh, sudo, vim, samba, smartctl) and setting up the Dma Dragonfly Mail Agent I’m leaving the host alone. The services that this machine will provide are going to be run in jails. To make it easy to move those virtualized environments to a different computer should the need arise[2], they will be installed on a separate zpool. Luckily, I still had two 1TB SSDs from my old server, containing a mirrored zpool with deduplication[3] turned on. It was created on OmniOS, but FreeBSD imported that zpool without a hitch. After upgrading the pool to enable the latest ZFS pool feature flags (I wanted to use zstd to compress files) and creating a new dataset to store the jails (and a second dataset for the website environment called “hexo”) I was ready to create my first jail:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ sudo zpool upgrade ssdPool
This system supports ZFS pool feature flags.

Enabled the following features on 'ssdPool':
  redaction_bookmarks
  redacted_datasets
  bookmark_written
  log_spacemap
  livelist
  device_rebuild
  zstd_compress
  draid
  
$ sudo zfs set compression=zstd ssdPool

$ sudo zfs create -o mountpoint=/jails ssdPool/jails
$ sudo zfs create  ssdPool/jails/hexo

Creating a FreeBSD jail is quick process. With the dataset created, a simple command gets the process started:

1
$ sudo bsdinstall jail /jails/hexo

The installer asks a few questions. I disabled all optional system components but enabled sshd services to be started at boot. The third and last screen asks for a user to be created, which I did. To boot the virtual machine a file /etc/jail.conf has to be created on the host. Here is mine for reference, although the part that is specific to each virtual machine depends on the host, the network organization and the disk layout and thus will have to be adjusted:

1
2
3
4
5
6
7
8
9
10
mount.devfs;                               # Mount devfs inside the jail
exec.clean;
exec.start = "/bin/sh /etc/rc";            # Start command
exec.stop = "/bin/sh /etc/rc.shutdown";    # Stop command
hexo {
    host.hostname = hexo.ottmar.org;       # Hostname
    ip4.addr = 192.168.3.10/24;            # IP address of the jail
    interface = "bge0";
    path = "/jails/hexo";                  # Path to the jail
}

Let’s boot it:

1
2
$ sudo service jail start hexo
Starting jails: hexo.

and log in:

1
sudo jexec hexo /bin/sh

At this point I configure sshd so I can directly log into the virtual machine without having to go through the host and install samba which allows me to mount the home directory on the virtual machine on my laptop, making file editing possible via my beloved BBEdit.

Installing the Website Development Tools

The main ingredients to install Hexo are Node.js and Git. I had hoped to install Node.js with nvm, but nvm does not have a binary for Node.js for FreeBSD. So I opted to install Node.js directly:

1
$ sudo pkg install node-18.7.0

followed by the node package manager compatible with Node.js version 18:

1
$ sudo pkg install npm-node18-8.19.1

git can be easily installed as well:

1
$ sudo pkg install git

All the pre-requisites are there, now we can download Hexo as outlined in their Documentation | Hexo[4]

1
$ sudo npm install hexo-cli -g

After initializing the website

1
2
3
4
$ hexo init server1Website
INFO  Cloning hexo-starter https://github.com/hexojs/hexo-starter.git
INFO  Install dependencies
INFO  Start blogging with Hexo!

I can download the hexo-theme-next that I’m using for this website:

1
2
3
4
5
6
7
8
:~/server1Website $ git clone https://github.com/next-theme/hexo-theme-next themes/next
Cloning into 'themes/next'...
remote: Enumerating objects: 6194, done.
remote: Counting objects: 100% (105/105), done.
remote: Compressing objects: 100% (78/78), done.
remote: Total 6194 (delta 30), reused 69 (delta 25), pack-reused 6089
Receiving objects: 100% (6194/6194), 1.30 MiB | 1.38 MiB/s, done.
Resolving deltas: 100% (4005/4005), done.

At this point I copied all the posts and configuration files for the website into the virtual machine, letting me generate the website:

1
hexo generate --watch

and deploy it locally for development

1
hexo server

Pointing the web-browser at port 4000 on the virtual machine will then serve the content and let me check the website as I’m writing a new blog post.

Picture Credits:

  1. More accurately the illumos kernel OmniOS is using provides the ZFS support. ↩︎

  2. The Mac Mini is old. You never know how long it will keep going. ↩︎

  3. Deduplication saves a lot of disk space if there are multiple virtual machines installed. Deduplication will simply point to the existing files instead of duplicating the data on disk. ↩︎

  4. I ran the install using sudo even though it is discouraged. Reason being that the workaround, installing npm with nvm failed as nvm does not have a binary for npm on FreeBSD. Since I’m operating in a virtual environment there isn’t much downside to overriding the installation errors using sudo. ↩︎