Home
VPN WorkPlace
Web Sites/Apps
Small Business
Other Apps
Contact
Guides
Raspberry Pi Cluster Running Ubuntu Server 20.04
This cluster utilizes NFS to store all of the file systems onto a single drive.
For easy setup & maintenance, I am utilizing small SD Cards to boot the kernel instead of PXE. Trust me life is so much easier with a pack of cheap SD Cards. As they are only used to boot, speed and longevity is not an issue.
Index
Environment
-
All Nodes Share One Storage Device Mounted on Node 01
-
Node 01 OS Storage on USB Storage Device
All other Nodes OS Storage utilizing Node 01 storage via NFS
-
All Nodes boot partitions are on SD cards instead of PXE boot.
PXE boot is possible, but a pain as you have to constantly manually update the kernel and vmlinux config file.
As the SD card is only accessed during boot and kernel updates, speed, size and longevity is not an issue.
2-4GB SD Cards will due fine.
Example IP Structure
Node |
Name |
IP |
Role |
1 |
pnode01 |
192.168.4.1 |
Storage Host, NFS Server, DHCP Server |
2 |
pnode02 |
192.168.4.2 |
NFS Client |
3 |
pnode03 |
192.168.4.3 |
NFS Client |
4 |
pnode04 |
192.168.4.4 |
NFS Client |
5 |
pnode05 |
192.168.4.5 |
NFS Client |
6 |
pnode06 |
192.168.4.6 |
NFS Client |
7 |
pnode07 |
192.168.4.7 |
NFS Client |
Harware Example
- 6x Raspberry Pi 4
- 1x Raspberry Pi 2 (Had one still kicking around)
- 1x GeeekPi Raspberry X828 USB3 SATA Shield
- 1x Westurn Digital 500GB SSD
- 1x Gigabit USB3 Ethernet Adapter
- 1x Gigabit Netgear 8 Port Switch
- 1x Raspberry Pi 4 USB-C 3amp Power Adapter
- 1x 5amp 5v Switching Power Adapter (X828 Power Supply)
- 1x RAVPower 60w 12amp 6-Port USB Charger
- Various USB Cables
- 7x SanDisk Extreme 16GB microSDHC Card
- Many 20mm M2.5 Brass Standoffs
- 1x RPi RTC (Pi Realtime Clock Module) (ds1307 Type)
Initial SD Card Setup
Using Raspberry Pi Imager, write 2 SD Cards with Ubuntu Server. 32bit or 64bit will do. 1st SD Card will be to utilize USB
storage and the 2nd will be to utilize NFS. We will create an image of the 2nd to make easy duplicates for Nodes 03+.
Setup Node 01 (pnode01)
Insert and boot the first Pi (pnode01). Wait for the Pi to fully start the first time. You will know this because it will display certificate information then ask to login.
Login Username & Password will both be ubuntu. If you try to login with these credentials and they do not work, wait a little more, first boot is not completed.
Now plug in your USB storage drive. USB3 is way better then USB2.
Use the following guide to setup the usb storage as Ext4 with a partition label of 'osstorage' and temporarily mount to /osstorage
Copy OS to USB Storage.
sudo cp -axv /. /osstorage/.
sudo cp -axv /dev/. /osstorage/dev/.
Now power off the Pi and remove the SD Card
sudo poweroff
Now insert the SD Card of pnode01 into your computer. Open the small visible partition and edit the file 'cmdline.txt'.
Alter root=LABEL=writable to root=LABEL=osstorage
root=LABEL=writable
root=LABEL=osstorage
Now save & eject the SD Card. Now when you boot your Pi (pnode01), it will be working from the USB storage drive.
You can verify this as the green SD indicator light at the rear of the Raspberry Pi will no longer blink after bootup.
IP Addressing Options.
-
pnode01 as DHCP Server & Gateway.
This is how I cluster my Pi nodes. I use a USB3 Gigabit Ethernet adapter as the External adapter and the Integrated Ethernet for Internal as it is faster for serving NFS. I then use IP Tables to forward traffic as needed.
DHCP Server Guide coming soon.
-
Use Existing Network DHCP Server
You will need to reserve a static IP for each Pi node.
Setup NFS on pnode01
Install required package
sudo apt install nfs-kernel-server
Now create directory structure
sudo mkdir /nfsroot
Create a directory for each node.
sudo mkdir /nfsroot/pnode02
sudo mkdir /nfsroot/pnode03
sudo mkdir /nfsroot/pnode04
sudo mkdir /nfsroot/pnode05
sudo mkdir /nfsroot/pnode06
sudo mkdir /nfsroot/pnode07
Create a shared directory for use by all nodes.
sudo mkdir /nfspublic
Now we configure the NFS Exports
sudo nano /etc/exports
Your config will need to look something like this below.
/nfsroot/pnode02 192.168.4.2(rw,no_root_squash,async,insecure,no_subtree_check)
/nfsroot/pnode03 192.168.4.3(rw,no_root_squash,async,insecure,no_subtree_check)
/nfsroot/pnode04 192.168.4.4(rw,no_root_squash,async,insecure,no_subtree_check)
/nfsroot/pnode05 192.168.4.5(rw,no_root_squash,async,insecure,no_subtree_check)
/nfsroot/pnode06 192.168.4.6(rw,no_root_squash,async,insecure,no_subtree_check)
/nfsroot/pnode07 192.168.4.7(rw,no_root_squash,async,insecure,no_subtree_check)
/nfspublic 192.168.4.*(rw,no_root_squash,async,insecure,no_subtree_check)
Now save and export configuration.
sudo exportfs -rv
Configure first NFS node pnode02
Lets start by unloading a couple of packages that will have a problem with NFS.
sudo apt remove cryptsetup
sudo apt remove mdadm
sudo snap remove lxd
Now we can add NFS support
sudo apt install nfs-common
Next to mount the NFS share that we need to set as OS Storage
sudo mkdir /osroot
sudo mount -t nfs -onolock 192.168.4.1:/nfsroot/pnode02 /osroot
And to copy OS
sudo cp -axv /. /osroot/.
sudo cp -axv /dev/. /osroot/dev/.
Now you can power off and remove the SD Card.
sudo poweroff
Now place the SD Card from pnode02 into your computer.
Using Computer Manager -> Disk Management, you can remove the linux partition. (Do not delete the small boot partition.)
Like the first SD Card, you will need to locate the cmdline.txt file and edit the following.
Replace the content of cmdline.txt with the following.
console=serial0,115200 console=tty root=/dev/nfs nfsroot=192.168.4.1:/nfsroot/pnode02,vers=3 rw ip=dhcp rootwait elevator=deadline
Now save and close cmdline.txt
Now you can duplicate the boot SD Card as needed. Remember to update the nfsroot= for each SD Card.
Using a utility such as Win32 Disk Imager, create an image of the SD Card. With Win32 Disk Imager, check Read Only Allocated Partitions.
Duplicate first NFS Node
Log into pnode01 (Node with USB Storage) and duplicate the NFS node.
sudo cp -axv /nfsroot/pnode02 /nfsroot/pnode03
sudo cp -axv /nfsroot/pnode02 /nfsroot/pnode04
sudo cp -axv /nfsroot/pnode02 /nfsroot/pnode05
Etc as needed.
Setup Shared Directory /nfspublic
On each NFS node edit the /etc/fstab file and add the following
sudo nano /etc/fstab
192.168.4.1:/nfspublic /nfspublic nfs _netdev,noauto,x-systemd.automount,nolock,soft,intr,nfsvers=3,tcp,bg,rw 0 0
I find this configuration works best to quickly load the NSF resource for services that may utilize the NSF such as Apache.
You should now have all of your Nodes working
You can now configure each node as you desire.
First you should update the hostnames of each node.
sudo nano /etc/hostname
Configure The RTC Module if you have one.
I run the RTC Module on pnode01 and then host a Network Time Service to supply the other nodes.
Edit Boot Settings and add the following to 'usercfg.txt' found on Fat partition of Micro SD Card (You will have to remove the SD Card and edit it on your computer.)
dtoverlay=i2c-rtc,ds1307
Now Sync the module
sudo timedatectl set-local-rtc false
Check if time sync is already active
sudo systemctl is-active systemd-timesyncd
Next create the service
sudo nano /etc/systemd/system/hwclock-sync.service
[Unit]
Description=Time Synchronisation from RTC Source
After=systemd-modules-load.service
RequiresMountsFor=/dev/rtc
Conflicts=shutdown.target
[Service]
Type=oneshot
ExecStart=/sbin/hwclock -s
TimeoutSec=0
[Install]
WantedBy=time-sync.target
Install TimeSyncD
sudo apt install systemd-timesyncd
Enable Service
sudo systemctl enable hwclock-sync systemd-timesyncd
sudo systemctl start hwclock-sync systemd-timesyncd
Check Time
timedatectl
Backup Node
To Backup a node to a single file maintaining permissions use:
sudo tar -cvpzf /nfsroot/pi4_pnode04_20200810.tar.gz /nfsroot/pnode04
To be continued later today. Sorry :), ran out of time.
CAMM Services Application Design
Brantford PC
Norfolk County PC
R.D. Cookson Disposal Ltd.
Web Site Design Brantford, Ontario
Web Site & Application Development Norfolk County, Ontario
Various front loader and roll off bin rentals in Simcoe, Ontario
3D Marketing and Exhibit Specialists, Greater Toronto Area
Robert Jones Marine Technical Services