Booting FreeBSD 9.0-CURRENT from NS-K330’s RAM

Almost a month ago I obtained an NS-K330 “NAS”  — a tiny embedded computer with one 10/100Mbps LAN port and two USB 2.0 ports, intended for home use as a ultra-cheap Network Attached Storage unit. This post summarises my initial experiments with installing and booting FreeBSD 9.0-CURRENT (as of early 2011) via uBoot boot loader and TFTP.

The NS-K330

The NS-K330 is based on a 250MHz Cavium Networks “CNS2132” ARM processor with 32MB RAM. More basic details can be found in my earlier post. The device’s uBoot boot loader is normally this is completely hidden to the end user, but you can access it via the on-board serial port to do low-level re-flashing of the NS-K330.

Cross-compiling FreeBSD 9.0-CURRENT

My home FreeBSD server (gjabkup2) is running FreeBSD 8.0-STABLE #2 r202063 at the moment, which is fine for cross-compiling FreeBSD 9-CURRENT for the ARM-based NS-K330. We will be building and installing world on external USB storage, and booting the kernel image from RAM via TFTP.

I created a new 3Gbyte ZFS dataset (filesystem) specifically for storing my FreeBSD 9-CURRENT world and kernel source tree, and mounted it on /usr/nsk330-dev. I’ve enabled compression and turned off atime updates to speed things along a little.

gjabkup2# zfs set compression=on data/usr/nsk330-dev
gjabkup2# zfs set atime=off data/usr/nsk330-dev

Grab the FreeBSD 9-CURRENT source tree

gjabkup2# cd /usr/nsk330-dev
gjabkup2# svn co http://svn.freebsd.org/base/head .
 [....]
gjabkup2#

[Result is svn revision 219528, 2011-03-12 06:40:49 +1100 (Sat, 12 Mar 2011)]

Then applied  the magic sauce — a patch file (cns21xxnas.diff) supplied by my colleague, Mattia Rossi.

gjabkup2# patch -p0 < cns21xxnas.diff

Now, kick things off with a completely clean buildworld.

gjabkup2# cd /usr/obj
gjabkup2# rm -rf arm.arm/
gjabkup2#
gjabkup2# cd /usr/nsk330-dev/
gjabkup2# make -j4 -s TARGET=arm buildworld
--------------------------------------------------------------
>>> World build started on Sat Mar 19 13:46:53 EST 2011
--------------------------------------------------------------
--------------------------------------------------------------
>>> Rebuilding the temporary build tree
--------------------------------------------------------------
--------------------------------------------------------------
>>> stage 1.1: legacy release compatibility shims
--------------------------------------------------------------
===> tools/build (obj,includes,depend,all,install)
--------------------------------------------------------------
>>> stage 1.2: bootstrap tools
--------------------------------------------------------------
===> lib/clang/libllvmsupport (obj,depend,all,install)

	[...blah... blah... long time...]

===> usr.sbin/ypset (all)
===> usr.sbin/zic (all)
===> usr.sbin/zic/zic (all)
===> usr.sbin/zic/zdump (all)
--------------------------------------------------------------
>>> World build completed on Sat Mar 19 14:16:17 EST 2011
--------------------------------------------------------------
gjabkup2#

Cross compile kernel
Then build kernel (without the “-j” option — it just seemed to work more reliably). In this case my custom config file was CNS21XXNAS_PPP_SMALL

gjabkup2# make -s TARGET=arm KERNCONF=CNS21XXNAS_PPP_SMALL buildkernel

--------------------------------------------------------------
>>> Kernel build for CNS21XXNAS_PPP_SMALL started on Sat Mar 19 15:30:07 EST 2011
--------------------------------------------------------------
===> CNS21XXNAS_PPP_SMALL
 [.....]
linking kernel
   text    data     bss     dec     hex filename
2965488  179140  150804 3295432  3248c8 kernel
--------------------------------------------------------------
>>> Kernel build for CNS21XXNAS_PPP_SMALL completed on Sat Mar 19 15:32:07 EST 2011
--------------------------------------------------------------
gjabkup2#

The newly compiled FreeBSD 9-CURRENT kernel is now located at /usr/obj/arm.arm/usr/nsk330-dev/sys/CNS21XXNAS_PPP_SMALL/kernel.gz.tramp.bin

Install world on external USB storage
At this stage FreeBSD 9-CURRENT doesn’t have support for treating the NS-K330’s flash as a root file system. Nor have we explored crunching a stand-alone FreeBSD kernel+world down small enough to fit on the 8MB flash. So the next step is to prepare external storage device to hold world. I used an old, spare Kensington 512MB USB memory stick.

gjabkup2# tail -40 /var/log/messages
	[..]
Mar 12 16:36:42 gjabkup2 kernel: da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
Mar 12 16:36:42 gjabkup2 kernel: da0:  Removable Direct Access SCSI-2 device
Mar 12 16:36:42 gjabkup2 kernel: da0: 40.000MB/s transfers
Mar 12 16:36:42 gjabkup2 kernel: da0: 489MB (1001472 512 byte sectors: 64H 32S/T 489C)
gjabkup2#
gjabkup2# bsdlabel -w /dev/da0s1
gjabkup2# ll /dev/da0*
crw-r-----  1 root  operator    1,  63 Mar 12 21:42 /dev/da0
crw-r-----  1 root  operator    1,  66 Mar 12 21:47 /dev/da0s1
crw-r-----  1 root  operator    1,  69 Mar 12 21:47 /dev/da0s1a
gjabkup2#
gjabkup2# newfs /dev/da0s1a
/dev/da0s1a: 489.0MB (1001424 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 122.25MB, 7824 blks, 15680 inodes.
super-block backups (for fsck -b #) at:
 160, 250528, 500896, 751264
gjabkup2#
gjabkup2# mount /dev/da0s1a /mnt
gjabkup2# df -h /mnt
Filesystem    Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    473M    4.0K    436M     0%    /mnt
gjabkup2#
gjabkup2# make TARGET=arm KERNCONF=CNS21XXNAS_PPP_SMALL DESTDIR=/mnt installworld distrib-dirs distribution
mkdir -p /tmp/install.HgN9xr5
[...lots of output...veeery slow copying of data across to the USB drive under /mnt....]

(NOTE: Installing the man pages takes ages — lots of links. For subsequent experiments I created a tarball of the world installed under /mnt, and simply un-tar’d this tree onto other USB drives that I also used to hold world.)

Tweak FreeBSD 9’s world to work inside NS-K330 context

With the USB drive still mounted on gjabkup2:/mnt, we need to make a few tweaks to the system configuration with which the NS-K330 will eventually boot under FreeBSD 9-CURRENT.

The USB drive will appear as /dev/da0s1a on the NS-K330 (assuming it is the first USB drive polled during boot), so I put the following line into gjabkup2:/mnt/etc/fstab

/dev/da0s1a / ufs rw 1 1

Added the following to gjabkup2:/mnt/etc/rc.conf (I decided the NS-K330 would be 10.1.1.67 on the home LAN).

defaultrouter=""
hostname="ns-k330"
ifconfig_ece0="10.1.1.67"
ifconfig_ece0_ipv6="ACCEPT_RTADV"
keymap="us.iso"
sshd_enable="YES"

Enable root login via ssh by editing /mnt/etc/ssh/sshd_config

PermitRootLogin yes
PermitEmptyPasswords yes

Finally, I un-mounted the USB disk from gjabkup2  (“cd / && umount /mnt”) and plugged it into the NS-K330.

Boot new FreeBSD 9-CURRENT kernel on NS-K330 via TFTP

uBoot enables us to load and boot a kernel via TFTP, so this is the first approach I use to confirm the FreeBSD kernel does indeed work (while leaving the NS-K330’s current SnakeOS 1.2.0 firmware untouched).

First, set up a tftp server on gjabkup2 (10.1.1.12) to host the kernel image file. Create the /tftpboot directory to host content, and copy over the bootable kernel image.

gjabkup2# mkdir -p /tftpboot
gjabkup2# cp /usr/obj/arm.arm/usr/nsk330-dev/sys/CNS21XXNAS_PPP_SMALL/kernel.gz.tramp.bin /tftpboot/
gjabkup2#

Edit /etc/inetd.conf to uncomment the tftp line (enabling tftpd from inetd), then restart inetd.

gjabkup2# /etc/rc.d/inetd onerestart
inetd not running? (check /var/run/inetd.pid).
Starting inetd.
gjabkup2#
gjabkup2# sockstat -4
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     inetd      8486  5  udp4   *:69                  *:*
	[...blah...]
gjabkup2#

Now power up the NS-K330, with serial console attached. Enter uBoot (hit <space> at the “Hit any key to stop autoboot:” prompt) and load the kernel image into RAM at address 0x600000.

Star Equuleus # setenv ipaddr 10.1.1.67
Star Equuleus # setenv serverip 10.1.1.12
Star Equuleus # tftpboot 0x600000 kernel.gz.tramp.bin
Check Link Status .Up
Waiting 3 seconds ..
TFTP from server 10.1.1.12; our IP address is 10.1.1.67
Filename 'kernel.gz.tramp.bin'.
Load address: 0x600000
Loading: #################################################################
         #################################################################
         #################################################################
         #################################################################
         ###################################################
done
Bytes transferred = 1591390 (18485e hex)
Star Equuleus #

Finally, tell uBoot to execute the code at 0x600000 — this boots the FreeBSD kernel, which has been statically configured to treat /dev/da0s1a (the attached USB drive) as the root file system.

Star Equuleus # go 0x600000
## Starting application at 0x00600000 ...
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.0-CURRENT #0 r219528M: Sat Mar 19 15:31:55 EST 2011
gja@gjabkup2.space4me.com:/usr/obj/arm.arm/usr/nsk330-dev/sys/CNS21XXNAS_PPP_SMALL arm
Preloaded elf kernel "elf kernel" at 0xc0922b70.
CPU: FA526 rev 1 (ARM9TDMI core)
DC enabled IC enabled WB enabled LABT branch prediction enabled
8KB/16B 2-way Instruction cache
8KB/16B 2-way write-back-locking-B Data cache
real memory = 33554432 (32 MB)
Physical memory chunk(s):
00000000 - 0x5fffff, 6291456 bytes (1536 pages)
0x96f000 - 0x1f36fff, 22839296 bytes (5576 pages)
avail memory = 28798976 (27 MB)
ULE: setup cpu 0
wlan: <802.11 Link Layer>
mem: <memory>
null: <null device, zero device>
random: <entropy source, Software, Yarrow>
nfslock: pseudo-device
econaarm0: <ECONA device bus> on motherboard
pmc0: <pmc> mem 0x77000000-0x77ffffff on econaarm0
PLL Clock: 250Mhz
CPU Clock: 250Mhz
AHB Clock : 125Mhz
APB Clock : 62Mhz
uart0: <Non-standard ns8250 class UART with FIFOs> mem 0x78000000-0x787fffff irq 9 on econaarm0
uart0: fast interrupt
uart0: console (38461,n,8,1)
uart1: <Non-standard ns8250 class UART with FIFOs> mem 0x78800000-0x78ffffff irq 10 on econaarm0
uart1: fast interrupt
timer0: <Econa CPU Timer> mem 0x79000000-0x79ffffff irq 0,1 on econaarm0
Misc probe
misc0: <misc> mem 0x76000000-0x76ffffff on econaarm0
spi0: <SPI> mem 0x71000000-0x71ffffff on econaarm0
spibus0: <spibus bus> on spi0
m25p800: <M25p80 Flash Family> at cs 0 on spibus0
ohci0: <Econa integrated OHCI controller> mem 0xe0000000-0xe7ffffff irq 23 on econaarm0
usbus0 on ohci0
usbus0: bpf attached
ohci0: usbpf attached
ehci0: <CNS11XX USB EHCI> mem 0xe8000000-0xefffffff irq 24 on econaarm0
usbus1: EHCI version 1.0
usbus1: <CNS11XX USB EHCI> on ehci0
usbus1: bpf attached
ehci0: usbpf attached
ece0: <Econa Ethernet Controller> mem 0x70000000-0x70ffffff irq 18,19,20,21,22 on econaarm0
ece0: Generating random ethernet address.
miibus0: <MII bus> on ece0
ukphy0: <Generic IEEE 802.3u media interface> PHY 0 on miibus0
ukphy0: OUI 0x007072, model 0x0025, rev. 1
ukphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
ece0: bpf attached
ece0: Ethernet address: 62:73:64:12:a6:b1
CPU Clock 250000000 APB clock 62500000
Timecounter "CPU Timer" frequency 62500000 Hz quality 1000
Timecounters tick every 10.000 msec
lo0: bpf attached
m25p800: found m25p80 flash eon en25p32
GEOM: new disk flash/spi0
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Cavium> at usbus0
uhub0: <Cavium OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Cavium> at usbus1
uhub1: <Cavium EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus1
ugen1.2: <Kingston> at usbus1
umass0: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.00, addr 2> on usbus1
Root mount waiting for: usbus1
umass0:0:0:-1: Attached to scbus0
Trying to mount root from ufs:da0s1a []...
mountroot: waiting for device da0s1a ...
(probe0:umass-sim0:0:0:0): SCSI status error
(probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0
(probe0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(probe0:umass-sim0:0:0:0): SCSI status: Check Condition
(probe0:umass-sim0:0:0:0): SCSI sense: UNIT ATTENTION asc:28,0 (Not ready to ready change, medium may have changed)
(probe0:umass-sim0:0:0:0): Retrying command (per sense data)
pass0 at umass-sim0 bus 0 scbus0 target 0 lun 0
pass0: <Kingston DataTraveler 2.0 1.00> Removable Direct Access SCSI-2 device
pass0: 40.000MB/s transfers
GEOM: new disk da0
da0 at umass-sim0 bus 0 scbus0 target 0 lun 0
da0: <Kingston DataTraveler 2.0 1.00> Removable Direct Access SCSI-2 device
da0: 40.000MB/s transfers
da0: 489MB (1001472 512 byte sectors: 64H 32S/T 489C)
warning: no time-of-day clock registered, system time will not be set accurately
start_init: trying /sbin/init
Setting hostuuid: 6c3dbf08-4cf6-11e0-a390-62736412a6b1.
Setting hostid: 0x3711d759.
No suitable dump device was found.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/da0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/da0s1a: clean, 143963 free (75 frags, 17986 blocks, 0.0% fragmentation)
Mounting local file systems:.
Setting hostname: ns-k330.
ece0: link state changed to DOWN
ifconfig: ACCEPT_RTADV: bad value
Starting Network: lo0 ece0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet 127.0.0.1 netmask 0xff000000
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ece0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=b<RXCSUM,TXCSUM,VLAN_MTU>
ether 62:73:64:12:a6:b1
inet 10.1.1.67 netmask 0xffffff00 broadcast 10.1.1.255
inet6 fe80::6073:64ff:fe12:a6b1%ece0 prefixlen 64 scopeid 0x3
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (none)
status: no carrier
Starting devd.
Starting Network: usbus0.
Starting Network: usbus1.
ece0: link state changed to UP
add net default: gateway 10.1.1.3
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
Generating host.conf.
Creating and/or trimming log files.
Starting syslogd.
/etc/rc: WARNING: Dump device does not exist. Savecore not run.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
Clearing /tmp (X related).
Updating motd:.
Generating public/private rsa1 key pair.
Your identification has been saved in /etc/ssh/ssh_host_key.
Your public key has been saved in /etc/ssh/ssh_host_key.pub.
The key fingerprint is:
8b:18:a0:57:84:ec:b3:7a:f1:1d:af:06:4c:9d:af:f2 root@ns-k330
The key's randomart image is:
+--[RSA1 1024]----+
| . ..            |
|  o.             |
| .. .. .         |
| .oo. o          |
|. .=.  .S        |
| .o oo....       |
| . o.o.+.        |
|. . o + .        |
| .   +E.         |
+-----------------+
Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
The key fingerprint is:
2b:57:de:2f:32:a1:f6:74:a6:2f:4c:d0:95:8c:5f:c0 root@ns-k330
The key's randomart image is:
+--[ DSA 1024]----+
|           +.o   |
|          . E .  |
|         . o .   |
|        . . .    |
|        S..      |
|         +o.     |
|      . o+o.+    |
|       oo.=+..   |
|       . .o=...  |
+-----------------+
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
c2:d5:4b:20:63:cb:01:b4:a1:4a:43:8a:ca:a7:1e:e1 root@ns-k330
The key's randomart image is:
+--[ RSA 2048]----+
| . .+.= .        |
|+  . = = o       |
|oo. . o . o      |
|+..  . . . .     |
|oo .  o S .      |
|. +    .         |
| E               |
|. .              |
| .               |
+-----------------+
Starting sshd.
Starting cron.
Starting background file system checks in 60 seconds.

Sat Mar 12 22:59:14 UTC 2011

FreeBSD/arm (ns-k330) (ttyu0)

login:

The preceding boot took a long time (seemed like a minute or so), primarily in generating the rsa and dsa host ssh key pairs.

Login to FreeBSD 9-CURRENT

We can now login as root on the console and have a look around to confirm we have a running FreeBSD 9-CURRENT system.

login: root
Mar 12 22:59:39 ns-k330 login: ROOT LOGIN (root) ON ttyu0
FreeBSD 9.0-CURRENT (CNS21XXNAS_PPP_SMALL) #0 r219528M: Sat Mar 19 15:31:55 EST 2011

Welcome to FreeBSD!

Before seeking technical support, please use the following resources:

o  Security advisories and updated errata information for all releases are
   at http://www.FreeBSD.org/releases/ - always consult the ERRATA section
   for your release first as it's updated frequently.

o  The Handbook and FAQ documents are at http://www.FreeBSD.org/ and,
   along with the mailing lists, can be searched by going to
   http://www.FreeBSD.org/search/.  If the doc distribution has
   been installed, they're also available formatted in /usr/share/doc.

If you still have a question or problem, please take the output of
`uname -a', along with any relevant error messages, and email it
as a question to the questions@FreeBSD.org mailing list.  If you are
unfamiliar with FreeBSD's directory layout, please refer to the hier(7)
manual page.  If you are not familiar with manual pages, type `man man'.

You may also use sysinstall(8) to re-enter the installation and
configuration utility.  Edit /etc/motd to change this login announcement.

ns-k330#
ns-k330# ps
  PID  TT  STAT      TIME COMMAND
 1187  u0- I      0:00.02 sh /etc/rc autoboot
 1188  u0- I      0:00.04 logger -p daemon.notice -t fsck
 1189  u0- I      0:00.03 sleep 60
 1192  u0  Ss     0:00.31 login [pam] (login)
 1193  u0  S      0:00.46 -csh (csh)
 1197  u0  R+     0:00.04 ps
ns-k330#
ns-k330# uptime
11:00PM  up 5 mins, 1 user, load averages: 0.49, 0.66, 0.34
ns-k330#
ns-k330# top
last pid:  1207;  load averages:  0.38,  0.63,  0.33    up 0+00:04:53  23:00:13
12 processes:  1 running, 11 sleeping
CPU:  0.0% user,  0.0% nice,  1.5% system,  0.5% interrupt, 98.0% idle
Mem: 6360K Active, 7056K Inact, 7928K Wired, 8K Cache, 7264K Buf, 6836K Free
Swap:

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 1207 root        1  21    0 10308K  2012K RUN      0:00  0.98% top
 1193 root        1  20    0 10188K  2684K pause    0:01  0.00% csh
 1192 root        1  22    0 10680K  1936K wait     0:00  0.00% login
  812 root        1  20    0  9864K  1372K select   0:00  0.00% syslogd
 1121 root        1  20    0  9924K  1428K nanslp   0:00  0.00% cron
 1110 root        1  20    0 11540K  2188K select   0:00  0.00% sendmail
 1188 root        1  52    0  9756K   896K piperd   0:00  0.00% logger
 1114 smmsp       1  52    0 11540K  2012K pause    0:00  0.00% sendmail
 1102 root        1  52    0 13412K  1924K select   0:00  0.00% sshd
 1189 root        1  52    0  1636K   800K nanslp   0:00  0.00% sleep
 1187 root        1  52    0 10168K  1136K wait     0:00  0.00% sh
  643 root        1  52    0  8260K   380K select   0:00  0.00% devd

ns-k330#

Conclusion

Cool! We can boot a patched version of FreeBSD 9.0-CURRENT that’s been TFPT’d into RAM, although an external USB drive is required to hold World. Next step is to install the kernel into NS-K330’s flash, so we don’t need to manually TFTP the kernel each time.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: