Oracle Linux on Btrfs for the Raspberry Pi

Enterprise comes to the micro server.

Oracle
Linux 7 has been released for the Raspberry Pi 3. The release
packages Btrfs as the root filesystem on the UEK-branded Linux 4.14 Long
Term Support (LTS) kernel. A bootable disk image with a minimal install is
provided along with a standard ISO installer.

CentOS appears
to support only the "Mustang" Applied Micro X-Gene for
AArch64, and it provides the older AArch32 environment for all models of the
Raspberry Pi. Oracle Linux is a compelling option among RPM distributions
in supporting AArch64 for the Pi Model 3.

This is not to say that Oracle AArch64 Linux is without flaw, as Oracle
warns that this is "a preview release and for development purposes only;
Oracle suggests these not be used in production." The non-functional WiFi
device is missing firmware and documentation, which Oracle admits was
overlooked. No X11 graphics are included in the image, although you can
install them. The eponymous database client (and server) are absent. Oracle
has provided a previous example of orphaned software with its Linux for
SPARC project, which was abandoned after two minor releases. There's no
guarantee that this ARM version will not suffer the same fate, although
Oracle has responded that "our eventual target is server class platforms".
One possible hardware target is the Fujitsu A64FX, a new server processor
that bundles 48 addressable AArch64 cores and 32GB of RAM on one die,
asserted to be the "fastest server processor" that exists.

AArch64 on the Pi

You'll need a Raspberry Pi Model 3 to run Oracle Linux. The 3B+ is the best
available device, and you should choose that over the predecessor Model 3B and
all other previous models. Both Model 3 boards retain the (constraining)
1GB of RAM—a SODIMM socket would be far more practical. The newer board
has a CPU that is 200MHz faster and a Gigabit-compatible Ethernet port
(that is limited to 300Mbit due to the USB2 linkage that connects it). A
Model A also exists, but it lacks many of the ports on the 3B. More
important, the Model 3 platform introduces a 64-bit CPU.

ARM was very tardy to 64-bit addressing capabilities compared to other
well known microprocessor families, announcing this extension in 2011.
Intel's first attempt to migrate the x86 market to the abortive Itanium
64-bit architecture shipped in 2001, ultimately yielding to AMD64, which
debuted in 2003. MIPS and SPARC made this transition much, much earlier
(1991 and 1995, respectively).

Despite the late arrival, the ARM AArch64 CPU architecture is now the
dominant mobile platform. All supported iPhones are now required to run it,
and most modern Android devices have migrated to it. ARM has guarded a competitive
edge in power efficiency, as vast changes in its instruction set ideology
and implementation have allowed ARM to maintain its market leadership in
mobile.

The Acorn/Advanced RISC Machine (ARM) began with the retroactively named
AArch32 assembly and machine language that was designed by Furber and
Wilson for the Archimedes desktop computer, where tremendous power
efficiency "was a complete accident". Desktop performance remained an
architectural focus for a decade after the birth of ARM.

Apple's decision to base the (failed) Newton on ARM opened a new market of
mobile device applications that prompted a new instruction set for
ultra-compact assembly—Thumb 1 and
2. These are distinct from AArch32 and
AArch64, and they focus on code density and minimal footprint for mobile
devices. Thumb 2 is a 16-/32-bit variable length instruction set, which is
dynamically translated to AArch32. Many other ARM extensions exist, but
Thumb appears to have both great flexibility and persistence across ARM
implementations.

When mobile devices neared the 4Gb RAM limit of 32-bit ARM, the designers
decided to break from the past. The primary mistakes in AArch32 have long
been known:

Design errors, like having r15 as the program counter or making every
instruction conditional, are problems for CPU architects rather than
programmers, and it's no surprise that they disappeared in the 64-bit
version of the ARM architecture. They must have made it awfully hard to
implement superscalar, out-of-order execution.

The changes in AArch64 bring it much closer to the spirit of MIPS, the most
notable of which are:

• 
  Conditional
  Execution has been removed, facilitating out-of-order
  processing in multiple pipelines.



• 
  The R15/PC
  register can now be manipulated only by a small number of
  jumping and branching instructions, vastly simplifying branch prediction.
  

These performance improvements, along with the increase of pointer sizes to
64-bits, come at the cost of code density—programs compiled for native
AArch64 will be larger than the equivalent for AArch32. Despite these
enhancements, the majority of Intel desktop processors of the last decade
easily will beat the Pi in most benchmarks (but they will not do so with a
10-Watt power supply). I examine the code density impact of AArch64
in greater detail below.

Installation

Most Raspberry Pi users rely on flash memory, which comes in two
grades. Multi-Level Cell (MLC) media is cheap and offers large amounts of
storage, but it can decay very quickly (cells typically are destroyed after
5,000 write operations). Most retail flash media (SD cards, USB flash
drives) are MLC, and will not have a long lifetime under high I/O usage,
despite the "wear-leveling" electronics that attempt to distribute
writes over the whole device evenly. Single-Level Cell (SLC) media is
more expensive and offers smaller amounts of storage, but it drastically
increases the number of write operations until cell failure (100,000). Both
types of memory can be "rehabilitated" by heating
them, but this is not
feasible for memory in most plastic packaging. If you anticipate large
amounts of I/O, plan to purchase the correct grade of flash.

One great benefit of the new Model 3 is the ability to boot from USB. A
standard hard disk drive is now a boot option. SLC media is also more
plentiful and inexpensive in the USB flash format than as microSD cards.
Choose the format that fits your expected I/O usage.

Once you have selected and obtained your media, you're ready to download
and uncompress the following file:

$ xz -dkv rpi3-ol7.6-image-20181116.img.xz
rpi3-ol7.6-image-20181116.img.xz (1/1)
  100 %   266.4 MiB / 5120.0 MiB = 0.052  55 MiB/s   1:32

The full size of the boot image is 5GB—your boot media must be at least
this size:

$ ll rpi3-ol7.6-image-20181116.img*
-rw-r--r-- 1 root root 5368709120 Jan 13 18:52
rpi3-ol7.6-image-20181116.img
-rw-r--r-- 1 root root  279309592 Jan 13 18:52
rpi3-ol7.6-image-20181116.img.xz

Insert your boot media and ensure that it is detected, but not mounted:

# dmesg | tail -3
[  378.540649] mmc0: new high speed SDHC card at address 0002
[  378.544104] mmcblk0: mmc0:0002 00000 7.83 GiB
[  378.548395]  mmcblk0: p1

Finally, write
the image to the raw media:

# dd if=rpi3-ol7.6-image-20181116.img of=/dev/mmcblk0 bs=4M

Assuming there are no write errors, the media is now prepared to boot the Raspberry
Pi Model 3.

Operation

Load the media into the Pi, connect an HDMI cable to a monitor, and attach
an ethernet cable. After the power supply is connected, the Pi will boot
(there is no power switch).

The Pi might fail to boot with older USB peripherals. When a well-worn IBM
89P8800 USB 1.1 keyboard was attached, the firmware issued the message
Timeout poll on interrupt endpoint and refused to boot. Trying a slightly
newer Lenovo 41A5248 keyboard, the boot proceeded but was greatly delayed.
Both keyboards worked without error once the OS was running. It might be
wise to boot initially without any non-essential USB hardware.

Once the boot is complete, a login: prompt should be displayed. The user
root with the password oracle will allow you to
select a new root password,
then drop to Bash.

The /proc/cpuinfo file will list four processor cores with the following
descriptions:

[root@rpi3 ~]# cat /proc/cpuinfo
processor   : 0 ... 1 ... 2 ... 3
BogoMIPS    : 38.40
Features    : fp asimd evtstrm crc32 cpuid
CPU implementer   : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part    : 0xd03
CPU revision      : 4

The root filesystem is on Btrfs, which is a change from the XFS that you
normally see on the x86_64 (AMD64) version of
Oracle/Red Hat/CentOS/Scientific Linux 7. The /boot filesystem is on EXT4,
likely due to bootloader considerations:

[root@rpi3 ~]# mount | egrep 'btrfs|ext4'
/dev/mmcblk0p4 on / type btrfs
(rw,noatime,ssd,space_cache,subvolid=5,subvol=/)
/dev/mmcblk0p2 on /boot type ext4 (rw,noatime,data=ordered)

Observe also the ssd mount option above. Btrfs detected this option automatically
and was previously
unsafe with a "a negative impact on
usability and lifetime" of the flash media, but it's now appropriate in the
4.14 kernel. Observe that the autodetected SSD mount option was not
specified in /etc/fstab (I've removed the UUIDs and LABELs):

[root@rpi3 ~]# sed -r 's/^(UUID|LABEL)[^ ]*[ ]*//' /etc/fstab
#Generated by RootFS Build Factory
/boot/efi vfat noatime 0 0
/boot ext4 noatime 0 0
swap swap noatime 0 0
/ btrfs noatime 0 0
tmpfs   /tmp       tmpfs   rw,nodev,nosuid,size=128M    0 0

The root filesystem is on the p4 partition of the SD card:

[root@rpi3 ~]# fdisk -l

Disk /dev/mmcblk0: 7948 MB, 7948206080 bytes, 15523840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000164f6

        Device Boot    Start      End   Blocks   Id  System
/dev/mmcblk0p1          2048   526335   262144    c  W95 FAT32
 ↪(LBA)
/dev/mmcblk0p2        526336  1550335   512000   83  Linux
/dev/mmcblk0p3       1550336  2074623   262144   82  Linux
 ↪swap / Solaris
/dev/mmcblk0p4       2074624  10463231  4194304   83  Linux

Note above that I'm running on an 8GB SD card, but the last third of the
card is unused, as it does not lie in a partition. You can
add the unused space to the root filesystem by first expanding the partition:

[root@rpi3 ~]# growpart /dev/mmcblk0 4
CHANGED: partition=4 start=2074624 old: size=8388608
↪end=10463232 new: size=13449183,end=15523807

And then expanding the Btrfs filesystem into the newly allocated space:

[root@rpi3 ~]# btrfs filesystem resize max /
Resize '/' of 'max'

The root filesystem now occupies the rest of the flash device:

[root@rpi3 ~]# fdisk -l

Disk /dev/mmcblk0: 7948 MB, 7948206080 bytes, 15523840 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000164f6

        Device Boot  Start      End   Blocks   Id  System
/dev/mmcblk0p1        2048   526335   262144    c  W95 FAT32
 ↪(LBA)
/dev/mmcblk0p2      526336  1550335   512000   83  Linux
/dev/mmcblk0p3     1550336  2074623   262144   82  Linux
 ↪swap / Solaris
/dev/mmcblk0p4     2074624 15523806  6724591+  83  Linux

Btrfs is an extremely powerful filesystem, similar in capabilities to ZFS.
It's capable of transparent compression, mirroring, error detection and
it has self-healing capabilities. (Watch for a future article on in-depth Btrfs
coverage.) Oracle Linux on the Raspberry Pi provides a useful learning
environment for many new tools and features, and the addition of Btrfs is
chief among them.

A number of missing utilities are present in minimal
installs on x86_64. In no particular order of preference, some are
ethtool,
less, man and nmtui. Assuming
internet connectivity to Oracle, a yum
install man will bring in less as a dependency (and let you begin reading
all the Btrfs manual pages). The yum whatprovides command is useful for
searching the contents of uninstalled packages for a particular utility.

Busybox is an alternative to some of the native Oracle AArch64 packages.
Those unfamiliar with Busybox might review my previous
container article published in Linux Journal that details its use. The 1.28.1
release offers several ARM binaries of
Busybox (listed below):

[root@rpi3 ~]# for x in busybox-arm*
               do ls -l $x; file $x; ./$x | head -1; done

-rwxr-xr-x 1 root root 1132724 Jan 10 17:23 busybox-armv5l
busybox-armv5l: ELF 32-bit LSB executable, ARM, version 1
 ↪(SYSV)...
BusyBox v1.28.1 (2018-02-15 14:34:02 CET) multi-call binary.
-rwxr-xr-x 1 root root  836560 Jan 10 17:23 busybox-armv7m
busybox-armv7m: ELF 32-bit LSB shared object, ARM, version 1
 ↪(SYSV)...
BusyBox v1.28.1 (2018-02-15 14:34:02 CET) multi-call binary.
-rwxr-xr-x 1 root root 1079156 Jan 10 17:23 busybox-armv7r
busybox-armv7r: ELF 32-bit LSB executable, ARM, version 1
 ↪(SYSV)...
BusyBox v1.28.1 (2018-02-15 14:34:02 CET) multi-call binary.
-rwxr-xr-x 1 root root 1078504 Jan 10 17:23 busybox-armv8l
busybox-armv8l: ELF 32-bit LSB executable, ARM, version 1
 ↪(SYSV)...
BusyBox v1.28.1 (2018-02-15 14:34:02 CET) multi-call binary.

Notice above that the busybox-armv7m binary is substantially smaller than
all the rest. This binary is apparently composed of Thumb 2 code; the ARM 7
M and R architectures appear to exclude both AArch32 and AArch64:
"ARMv7-M...
No ARM instruction set support (Thumb only)." Thumb might
explain the smaller size, but its use may come at some cost to performance.

Oracle includes an AArch64
compiler environment, but this isn't likely to
be capable of emitting ARMv7-M code. Oracle doesn't provide a
glibc.aarch32 or glibc.thumb2 package in the same way that it provides a
glibc.i686 on AMD64, nor are there any 32-bit libraries in /usr/lib. ARM
itself provides Thumb-capable
GNU compilers, as do other sources. Using
Thumb as a compiler target will conserve memory at the potential cost of
performance. This might be a reasonable choice for standing dæmons that
are not CPU-intensive.

One glaring lack in Oracle Linux on the Raspberry Pi is the missing WiFi
device. The kernel
dmesg has a clue to the problem:

brcmfmac: brcmf_fw_map_chip_to_name: using
        brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221)
         ↪rev 0x000006
usbcore: registered new interface driver brcmfmac
brcmfmac mmc1:0001:1: Direct firmware load for
                         brcm/brcmfmac43455-sdio.bin failed
                          ↪with error -2

You can find one source of the missing firmware at this link,
although you
also can find it within Raspbian. Installing the firmware will cause a
wlan0 device to appear, but all my attempts to configure it have failed.
It doesn't appear to be functional in the current release, despite the
brcmfmac
kernel module:

[root@rpi3 ~]# cd /usr/lib/firmware/brcm/
[root@rpi3 ~]# ll brcmfmac43455*

-rw-r--r-- 1 root root 600487 Jan 14 19:33
 ↪brcmfmac43455-sdio.bin
-rw-r--r-- 1 root root  14036 Jan 14 19:49
 ↪brcmfmac43455-sdio.clm_blob
-rw-r--r-- 1 root root   2054 Jan 14 19:41
 ↪brcmfmac43455-sdio.txt

It appears that the WiFi and Bluetooth devices on the Raspberry Pi work
through the SD card's SDIO
interface. Once those files are in place, reboot,
and the WiFi driver should appear in the dmesg. Note this will use a small
amount of additional memory:

mmc1: new high speed SDIO card at address 0001
brcmfmac: brcmf_fw_map_chip_to_name: using
        brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221)
         ↪rev 0x000006
brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0:
      Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY)
       ↪FWID 01-4fbe0b04
Bluetooth: Generic Bluetooth SDIO driver ver 0.1

There is no mention at all of the WiFi hardware on the Raspberry Pi within
Oracle's documentation on the AArch64 release, which Oracle claims was an
oversight. This also appears to be an issue in CentOS, where it is at least
discussed
at some length.

Code Density

As 1GB of RAM included on the Pi is constraining, you should have some idea
of the penalty AArch64 imposes.

Below is a script that I've used to size all the ELF binaries in Raspbian
Linux running on the original Raspberry Pi, storing this in the file
a32.txt:

for x in /bin/*
do [ -f "$x" ] &&
   case "$(file "$x")" in
        *ELF*) stat -c %n %s "$x";;
   esac
done > a32.txt

Moving this file to Oracle Linux running on the Raspberry Pi Model 3 B+, I
run the following to find the size differences:

while read p s
do [ -f "$p" ] &&
   case "$(file "$p")" in
        *ELF*) echo $p $s $(stat -c %s "$p");;
   esac
done < a32.txt | awk '
    a+=$2; b+=$3; print $1,$2,$3,$3/$2
END print a,b,b/a' > a64.txt

For this small sample of 66 files, I found the results shown in Table 1.

Table 1. Results of Size Differences of 66 Files

Program	Raspbian	OL7.6	% increase
/bin/bash	912712	971728	1.06466
/bin/cat	30560	70408	2.30393
/bin/chgrp	51084	70944	1.38877
/bin/chmod	46956	70840	1.50865
/bin/chown	51092	71000	1.38965
/bin/cp	104592	204296	1.95327
/bin/cpio	118460	141752	1.19662
/bin/date	83868	70368	0.839033
/bin/dd	63424	136456	2.15149
/bin/df	67876	137848	2.03088
/bin/dir	108804	138240	1.27054
/bin/dmesg	59484	78296	1.31625
/bin/echo	26404	69904	2.64748
/bin/false	22304	69880	3.13307
/bin/findmnt	52144	71992	1.38064
/bin/grep	173656	204048	1.17501
/bin/gzip	80476	137400	1.70734
/bin/hostname	13964	69048	4.94471
/bin/journalctl	63204	538448	8.51921
/bin/kill	22020	70432	3.19855
/bin/kmod	128560	203960	1.5865
/bin/less	151392	219472	1.44969
/bin/lessecho	9688	68752	7.09661
/bin/lesskey	14460	70320	4.86307
/bin/ln	46976	70848	1.50817
/bin/login	39112	70032	1.79055
/bin/loginctl	42732	538280	12.5966
/bin/ls	108804	138240	1.27054
/bin/lsblk	67756	138336	2.04168
/bin/mkdir	63472	137080	2.15969
/bin/mknod	55248	71272	1.29004
/bin/mktemp	34668	70288	2.02746
/bin/more	34708	69824	2.01176
/bin/mount	34872	68840	1.97408
/bin/mountpoint	9896	68944	6.96686
/bin/mv	100504	138480	1.37786
/bin/netstat	106676	211912	1.9865
/bin/ping	55720	70208	1.26001
/bin/ps	83624	137000	1.63829
/bin/pwd	26452	70056	2.64842
/bin/readlink	34628	70448	2.03442
/bin/rm	51076	71056	1.39118
/bin/rmdir	34628	70072	2.02356
/bin/sed	84100	71904	0.854982
/bin/sleep	26416	69984	2.6493
/bin/stty	59240	70240	1.18569
/bin/su	31016	69008	2.22492
/bin/sync	26424	69912	2.64578
/bin/systemctl	161680	738032	4.56477
/bin/systemd-ask-password	9948	70000	7.03659
/bin/systemd-escape	9936	69816	7.02657
/bin/systemd-hwdb	67520	136520	2.02192
/bin/systemd-inhibit	14040	337728	24.0547
/bin/systemd-machine-id-setup	18128	69912	3.85658
/bin/systemd-notify	9936	69728	7.01771
/bin/systemd-tmpfiles	50988	202912	3.9796
/bin/systemd-tty-ask-password-agent	26324	135920	5.16335
/bin/tailf	22288	69488	3.11773
/bin/tar	327644	350288	1.06911
/bin/touch	71584	70640	0.986813
/bin/true	22304	69880	3.13307
/bin/udevadm	395336	469248	1.18696
/bin/umount	22436	68856	3.069
/bin/uname	26416	69928	2.64718
/bin/vdir	108804	138240	1.27054
/bin/wdctl	26408	70256	2.66041
	5107652	9795488	1.91781

These programs take nearly twice the space in Oracle Linux as they do in
Raspbian. This somewhat explains the CentOS decision to remain on AArch32
with smaller 32-bit binaries. Oracle's pursuit of AArch64 is likely due to
similar platforms that it supports or may support in the future.

Should Oracle elect to provide a Thumb2 development environment in the same
way that it supports 32-bit x86, then Oracle could produce even smaller
binaries than are found in Raspbian while still running a 64-bit kernel, at
some cost to performance. This assumes that all target platforms support
Thumb2; by all appearances, the Fujitsu A64FX does not.

It might be useful to examine commonly run server dæmons, system
libraries and extract the text/data/bss segment sizes within all of these
programs to see greater detail on the AArch64 penalty paid here. Those with
large ARM deployments are encouraged to do so.

Conclusion

It's refreshing to have a new Linux distribution where legacy support is
slashed in a way that never would be tolerated within Intel/AMD64
environments. There is substantial complexity and inertia in the
maintenance of the systems of decades past.

Still, the relative silence in the documentation on questions of (the
overlooked WiFi) hardware support and the legacy Thumb and AArch32
instruction sets is unsettling. Operating system vendors should be clear
on what their products can and cannot do with the target hardware. While
there are issues with Oracle AArch64 Linux where that clarity is lacking,
it must be conceded that this is a pre-production release, and the desired
clarity and supported server-grade AArch64 target platforms may not yet
exist. To reiterate, one possible hardware target is the Fujitsu A64FX,
which designers are asserting as the fastest processor in the
world. Amazon
also recently began running ARM workloads in its EC2 cloud with its
Graviton processor, but Gravitons are not expected to outperform the
Fujitsu A64FX, and the relationship between Amazon and Oracle is not warm.
Oracle also may be developing its own AArch64 specifically tuned for
Oracle's database. Oracle previously has maintained SPARC in this capacity,
and continues
to dominate the TPC benchmark with it; the company also may decide to
do this with its own ARM processor.

On the subject of the Oracle Database, the absence of any discussion or
mention of it also is a cause for substantial concern on the longevity of
the platform.

In any case, Oracle AArch64 Linux likely will be used by many pursuing
low-power, large memory applications. The Raspberry Pi may be able to
provide a development environment for those larger systems. It's
encouraging to see ARM move into the enterprise space, and the prospect of
a legacy-free computing environment without all the problems (Meltdown),
scandals (ME/PSP) and firmware concerns of Intel is quite refreshing.

Resources