Adding a New Disk
WARNING: BE CAREFUL! Some commands in this tutorial can cause permanent data loss. We recommend you backup your files before attempting to run these commands. You should also double check every command to make sure you have made no typos or mistakes. A single typo could destroy your data forever!
NOTE: This guide is no substitute for reading the OpenBSD FAQ. In particular, you should read the section on Disk Setup.
NOTE: Before performing the upgrades, you will want to announce the changes to your users and teammates.
On OpenBSD, there are two disk drivers: wd and sd. wd is an IDE-like disk, and sd is a SCSI-like disk (including USB disks). To list all your available disks:
$ dmesg | grep -E '(sd[0-9]|wd[0-9])' sd0 at scsibus2 targ 0 lun 0: <BUYVM, SLAB, 2.5+> serial.BUYVM_SLAB_VOLUME-4047 sd0: 524288MB, 512 bytes/sector, 1073741824 sectors, thin sd1 at scsibus3 targ 0 lun 0: <VirtIO, Block Device, > sd1: 40960MB, 512 bytes/sector, 83886080 sectors
In this example, our server has two SCSI-like disks (a BuyVM slab and a VirtIO block device). In this case, the VirtIO block device is the hard disk the operating system is installed on, and the slab is extra storage we purchased. The disks are numbered by the order in which they are detected at boot time. The first SCSI-like disk will be sd0, the second sd1, and so forth.
The first disk sd0 can store around 500GB of data, the second disk sd1 can store around 40GB.
We will now add the new space from sd0 by creating a new partition for the disk, formatting it, and then mounting it to the filesystem.
WARNING: Double check your commands and make certain you are editing the correct disk. You must substitute the disk in the example for the one you actually want to format. Typing the wrong disk by accident could destroy all your data forever.
Creating a New Partition
The first task is to use fdisk to create fdisk partitions (Master Boot Record partitions). Now, the easy way is to run this one command:
$ doas fdisk -iy sd0
This automatically creates an fdisk partition that spans the entire disk. That said, you can (and should!) learn how to edit this interactively, to fine tune your options.
$ doas fdisk -e sd0 Enter 'help' for information
Type help to get help, manual to read the fdisk man page, and print to view the partitions:
sd0: 1> help
help Command help list
manual Show entire OpenBSD man page for fdisk
reinit Re-initialize loaded MBR (to defaults)
setpid Set the identifier of a given table entry
disk Edit current drive stats
edit Edit given table entry
flag Flag given table entry as bootable
update Update machine code in loaded MBR
select Select extended partition table entry MBR
swap Swap two partition entries
print Print loaded MBR partition table
write Write loaded MBR to disk
exit Exit edit of current MBR, without saving changes
quit Quit edit of current MBR, saving current changes
abort Abort program without saving current changes
sd0: 1> print
Disk: sd0 geometry: 66837/255/63 [1073741824 Sectors]
Offset: 0 Signature: 0x0
Starting Ending LBA Info:
#: id C H S - C H S [ start: size ]
-------------------------------------------------------------------------------
0: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
1: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
2: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
3: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
This disk sd0 is completely blank, so it is probably safe to edit the fdisk partitions and format the disk.
WARNING: If the disk already has existing partitions, double check to make sure you are not wiping out data!
First, let's reinit. This should automatically create an fdisk partition on the last partition, partition 3, for OpenBSD and use all available space. It will also initialize this as the boot block.
sd0: 1> reinit
Disk: sd0 geometry: 66837/255/63 [1073741824 Sectors]
Offset: 0 Signature: 0xAA55
Starting Ending LBA Info:
#: id C H S - C H S [ start: size ]
-------------------------------------------------------------------------------
0: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
1: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
2: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
*3: A6 0 1 2 - 66836 254 63 [ 64: 1073736341 ] OpenBSD
Use 'write' to update disk.
Notice that the Signature changes from 0x0 to 0xAA55, and partition #3 is now set to id A6 (the OpenBSD partition type).
This can also be done manually by using the edit command on partition 3:
sd0: 1> edit 3
Starting Ending LBA Info:
#: id C H S - C H S [ start: size ]
-------------------------------------------------------------------------------
3: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
Partition id ('0' to disable) [01 - FF]: [0] (? for help) A6
Do you wish to edit in CHS mode? [n] y
BIOS Starting cylinder [0 - 66836]: [0]
BIOS Starting head [0 - 254]: [1]
BIOS Starting sector [1 - 63]: [2]
BIOS Ending cylinder [0 - 66836]: [0] 66836
BIOS Ending head [0 - 254]: [0] 254
BIOS Ending sector [1 - 63]: [1] 63
We need to edit in Cylinder-Head-Sector (CHS) mode. To use the full space of the disk, you will want to choose the lowest numbers in the range for the starting cylinder, head, and sector; and the highest possible numbers for the ending cylinder, head, and sector.
We now write to disk:
sd0*: 1> write Writing MBR at offset 0.
We'll print G to see everything in gigabytes:
sd0: 1> print G
Disk: sd0 geometry: 66837/255/63 [1073741824 Sectors]
Offset: 0 Signature: 0xAA55
Starting Ending LBA Info:
#: id C H S - C H S [ start: size ]
-------------------------------------------------------------------------------
0: 00 0 0 0 - 0 0 0 [ 0: 0G] unused
1: 00 0 0 0 - 0 0 0 [ 0: 0G] unused
2: 00 0 0 0 - 0 0 0 [ 0: 0G] unused
*3: A6 0 1 2 - 66836 254 63 [ 64: 512G] OpenBSD
If everything looks good,
sd0: 1> quit
Disklabel
Next, we need to use disklabel to manage OpenBSD filesystem partitions (these disklabel partitions exist inside of an fdisk partition).
We should leave the first logical track unused, so the first partition should start at block 64.
Each fdisk partition can be sliced into OpenBSD disklabel partitions with letters as labels. The a partition is your root partition, b is your swap, and c is the entire disk.
If you already have the operating system installed on another disk (like we do here), you just need 1 single partition: the a partition with the same size as the disk.
$ doas disklabel -E sd0 Label editor (enter '?' for help at any prompt) sd0> ? Available commands: ? | h - show help n [part] - set mount point A - auto partition all space p [unit] - print partitions a [part] - add partition q - quit & save changes b - set OpenBSD boundaries R [part] - resize auto allocated partition c [part] - change partition size r - display free space D - reset label to default s [path] - save label to file d [part] - delete partition U - undo all changes e - edit drive parameters u - undo last change g [d|u] - [d]isk or [u]ser geometry w - write label to disk i - modify disklabel UID X - toggle expert mode l [unit] - print disk label header x - exit & lose changes M - disklabel(8) man page z - delete all partitions m [part] - modify partition Suffixes can be used to indicate units other than sectors: 'b' (bytes), 'k' (kilobytes), 'm' (megabytes), 'g' (gigabytes) 't' (terabytes) 'c' (cylinders), '%' (% of total disk), '&' (% of free space). Values in non-sector units are truncated to the nearest cylinder boundary.
To print the partitions in gigabytes:
sd0> p G OpenBSD area: 64-1073736405; size: 512.0G; free: 512.0G # size offset fstype [fsize bsize cpg] c: 512.0G 0 unused
We want to add the a partition. We'll use defaults for offset, size, and FS type:
sd0> a a offset: [64] size: [1073736341] FS type: [4.2BSD] sd0*> p G OpenBSD area: 64-1073736405; size: 512.0G; free: 0.0G # size offset fstype [fsize bsize cpg] a: 512.0G 64 4.2BSD 4096 32768 1 c: 512.0G 0 unused
Then write the changes to disk and quit:
sd0*> w sd0> q No label changes.
Next, you'll want to format the disk:
$ doas newfs sd0a /dev/rsd0a: 524285.3MB in 1073736320 sectors of 512 bytes 644 cylinder groups of 814.44MB, 26062 blocks, 52224 inodes each super-block backups (for fsck -b #) at: ...
We now have sd0a formatted, but we need to mount it. Let's look at where our system's partitions are currently mounted:
$ mount /dev/sd1a on / type ffs (local) /dev/sd1k on /home type ffs (local, nodev, nosuid) /dev/sd1d on /tmp type ffs (local, nodev, nosuid) /dev/sd1f on /usr type ffs (local, nodev) /dev/sd1g on /usr/X11R6 type ffs (local, nodev) /dev/sd1h on /usr/local type ffs (local, nodev, wxallowed) /dev/sd1j on /usr/obj type ffs (local, nodev, nosuid) /dev/sd1i on /usr/src type ffs (local, nodev, nosuid) /dev/sd1e on /var type ffs (local, nodev, nosuid)
We can also see the size of these partitions:
$ df -h Filesystem Size Used Avail Capacity Mounted on /dev/sd1a 615M 232M 352M 40% / /dev/sd1k 3.7G 2.1G 1.4G 59% /home /dev/sd1d 863M 28.0K 820M 0% /tmp /dev/sd1f 2.3G 1.6G 600M 74% /usr /dev/sd1g 648M 238M 378M 39% /usr/X11R6 /dev/sd1h 2.3G 2.2G 50.2M 98% /usr/local /dev/sd1j 5.2G 2.0K 4.9G 0% /usr/obj /dev/sd1i 1.4G 2.0K 1.3G 0% /usr/src /dev/sd1e 1.3G 318M 913M 26% /var
For this example, /home is too small for our server, so we're going to copy all the data from sd1k to sd0, then mount sd0a as the new /home.
$ doas su # mount /dev/sd0a /mnt # cd /mnt
Then we dump and restore to clone the partition:
# dump -0 -a -f - /home | restore -rf -
You will want to replace /home with whatever partition you want to back up on the new disk.
Alternatively, you can copy the files manually by running:
# cp -R /home/* /home/.* /mnt/
Inspect the files in the partition to make sure they copied successfully:
# ls -a /mnt/
Then unmount the partition:
# umount /mnt/ # ^D
(Type ctrl+d or ^D to exit)
The next steps will take the system offline, so before performing any more steps, you will want to announce to your users and teammates that they should save all files and prepare for downtime.
Now we need to automatically mount this partition at bootup. Let's take a look at the current /etc/fstab:
37deb3c1c30cbfbf.b none swap sw 37deb3c1c30cbfbf.a / ffs rw 1 1 37deb3c1c30cbfbf.k /home ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.d /tmp ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.f /usr ffs rw,nodev 1 2 37deb3c1c30cbfbf.g /usr/X11R6 ffs rw,nodev 1 2 37deb3c1c30cbfbf.h /usr/local ffs rw,wxallowed,nodev 1 2 37deb3c1c30cbfbf.j /usr/obj ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.i /usr/src ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.e /var ffs rw,nodev,nosuid 1 2
The string 37deb3c1c30cbfbf is called the disklabel unique identifier (DUID). Each disk partition has its own unique DUID. Let's find the DUID of our new disk partition:
$ sysctl hw.disknames hw.disknames=cd0:,sd0:f433d9e11879420f,sd1:37deb3c1c30cbfbf,fd0:
The DUID for sd0 is f433d9e11879420f. So, we need to replace /home's partition 37deb3c1c30cbfbf.k with f433d9e11879420f.a (since we are trying to mount sd0a):
37deb3c1c30cbfbf.b none swap sw 37deb3c1c30cbfbf.a / ffs rw 1 1 f433d9e11879420f.a /home ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.d /tmp ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.f /usr ffs rw,nodev 1 2 37deb3c1c30cbfbf.g /usr/X11R6 ffs rw,nodev 1 2 37deb3c1c30cbfbf.h /usr/local ffs rw,wxallowed,nodev 1 2 37deb3c1c30cbfbf.j /usr/obj ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.i /usr/src ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.e /var ffs rw,nodev,nosuid 1 2
Then, reboot:
$ doas shutdown -r now
If everything was done correctly, the system should automatically mount sd0a as /home upon bootup. The old partition, /dev/sd1k, can still be mounted if you need it:
$ doas mount /dev/sd1k /mnt/
Double check to make sure the files on /home exactly match those on /mnt from sd1k. If they do not, you will need to clone the disk again using dump/restore or cp.
Once you are certain the files have been copied properly, you can use unmount the partition:
$ doas umount /mnt
Then you can clear out the free space to grow an existing partition? or you can create a new partition with it and mount it elsewhere.
Troubleshooting:
Uh oh, my system won't reboot!
This can happen if you foul up /etc/fstab. For example, suppose I had made this error in /etc/fstab:
37deb3c1c30cbfbf.b none swap sw 37deb3c1c30cbfbf.a / ffs rw 1 1 f433d9e11879420f.b /home ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.d /tmp ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.f /usr ffs rw,nodev 1 2 37deb3c1c30cbfbf.g /usr/X11R6 ffs rw,nodev 1 2 37deb3c1c30cbfbf.h /usr/local ffs rw,wxallowed,nodev 1 2 37deb3c1c30cbfbf.j /usr/obj ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.i /usr/src ffs rw,nodev,nosuid 1 2 37deb3c1c30cbfbf.e /var ffs rw,nodev,nosuid 1 2
For the third line, instead of putting .a, I put .b by accident. When I reboot the system by running doas shutdown -r now, I can no longer ssh into the system. When I log in via serial console (cu for vmm or VNC for BuyVM)
root on sd0a (a0e97b069eed0743.a) swap on sd0b dump on sd0b
Automatic boot in progress: starting file system checks.
/dev/sd0a (a0e97b069eed0743.a): file system is clean; not checking
Can't open d40a0cfcd4551ac7.b: No such file or directory
CAN'T CHECK FILE SYSTEM.
d40a0cfcd4551ac7.b: UNEXPECTED INCONSISTENCY; RUN fsck_ffs MANUALLY.
/dev/sd0k (a0e97b069eed0743.k): file system is clean; not checking
/dev/sd0d (a0e97b069eed0743.d): file system is clean; not checking
/dev/sd0f (a0e97b069eed0743.f): file system is clean; not checking
/dev/sd0g (a0e97b069eed0743.g): file system is clean; not checking
/dev/sd0h (a0e97b069eed0743.h): file system is clean; not checking
/dev/sd0j (a0e97b069eed0743.j): file system is clean; not checking
/dev/sd0i (a0e97b069eed0743.i): file system is clean; not checking
/dev/sd0e (a0e97b069eed0743.e): file system is clean; not checking
THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY:
ffs: d40a0cfcd4551ac7.b (/home)
Automatic file system check failed; help!
Enter pathname of shell or RETURN for sh:
In order to boot up, you'll need to fix fstab. So, press the enter key to enter into the shell. Then, to view your /etc/fstab, type:
# cat /etc/fstab
Check how your partitions are mounted:
# mount root_device on / type ffs (local, read-only)
You'll notice that only / is mounted (so you can't use programs from /usr or view files in /home). This means you'll lack basic utilities like less and text editors like vi and mg. You'll also be unable to edit and save /etc/fstab, since / is mounted read-only. So first, let's fix that.
# mount -rw / # mount -rw /dev/sd0f /usr
Our fstab says a0e97b069eed0743.f is used for /usr so we mount /dev/sd0f to /usr.
Also, TERM=vt220 will allow us to use text editors like vi to edit (you could, of course, use ed).
# export TERM=vt220 # vi /etc/fstab
Replace f433d9e11879420f.b with f433d9e11879420f.a, save and quit, then reboot the system:
# shutdown -r now