View source for Storage management

== Introduction ==

Save large files like code, binary files, movies, etc. The storage media are things like HDDs, SSDs, magnetic tape.

Storage is not memory, where memory caches memory for quick access, things like any type of RAM: SRAM, SDRAM, VRAM, etc.

Swap storage is fake memory that is actually storage, but pretends to be memory. You need a partition for this, and so you run
{{RootCmd|typemkswap /dev/sda2}} 
to make the partition a swap file, then run
{{RootCmd|swapon /dev/sda2}}
to enable swap. Run <code>top</code> or <code>htop</code> to see your swap and memory usage.

== Looking at things ==
* [https://linux.die.net/man/8/lsblk lsblk] - "list block devices", shows partitions
** <code>lsblk -f</code> gives same info as <code>blkid</code>
* [https://linux.die.net/man/8/blkid blkid] - "block device ids", shows storage UUIDs
* [https://linux.die.net/man/8/fdisk fdisk] - list all your storage media using <code>fdisk</code>
* [https://linux.die.net/man/1/df df -h] - shows you the free space on each drive.
{{Tip|When trying to save space, use <code>du -sh ./*</code> to look at folder sizes and work your way down to delete the things that aren't needed.}}

== Mounting systems ==
=== [https://linux.die.net/man/8/findmnt findmnt] ===
Finds all the mounts in your system.

Anything in [https://www.thegeekdiary.com/understanding-the-sysfs-file-system-in-linux/ /sys] or [https://www.geeksforgeeks.org/proc-file-system-linux/ /proc] are kernel stuff. The kernel opens up files for you to look at. I don't know anything else about it! Look it up yourself, cuz I don't know what it means. It's not important to working with storage systems.

/dev is your stuff... mostly. Some system stuff that you don't use: [https://unix.stackexchange.com/questions/93531/what-is-stored-in-dev-pts-files-and-can-we-open-them /dev/pts], [http://www.csl.mtu.edu/cs3331.ck/www/Home.html /dev/shm], [https://users.pja.edu.pl/~jms/qnx/help/watcom/clibref/mq_overview.html /dev/mqueue].

=== Mounting commands ===
You're working with storage media, so every single command will always be run as root.

<code>/dev/sdX</code> are your drives! <code>/dev/sda</code> is your first, <code>/dev/sdb</code> is your second, etc.

Mount a drive to look at:
{{RootCmd|mount /dev/sdb1 /mnt}}
Unmount it when you're done:
{{RootCmd|umount /mnt}}
If you plug in a USB and you want to access it, use <code>lsblk -f</code>, then <code>mount /dev/sdXN /mnt</code> or wherever you want the USB files. Then, <code>umount /mnt</code> to safely remove.
{{Note|If you're testing this on [https://www.virtualbox.org/ VirtualBox], your virtual drives have no format, so just run 
{{RootCmd|mkfs.ext4 /dev/sdX}} for every drive and you can mount them.}}

== Bare-metal drives ==
We use [https://linux.die.net/man/8/fdisk fdisk].
{{RootCmd|fdisk /dev/sda}}
{| class="wikitable" style="margin:auto"
|+ [https://linux.die.net/man/8/fdisk fdisk] commands
|-
! Command !! What it does
|-
| g || sets disk as GUID Partition Table (GPT). This supports UEFI and up to 8 ZB (it goes GB, TB, PB, EB, ZB.. so a lot). 
|-
| o || Sets disk as a Master Boot Record MBR. This supports legacy boot, NO UEFI, and up to 2 TB size. Easier to set up than GPT. 
|-
| n || New partition.
|-
| p || Primary partition.
|-
| t || Sets the file type. Key ones are: (MBR) 82=Linux swap, 83=Linux, fd=RAID, 8e=LVM. (GPT) 19=Linux swap, 20=Linux, 29=Linux RAID, 30=Linux LVM, 1=EFI. The default on fdisk is usually Linux.
|-
| w || Write your changes.
|}

=== Demo setup ===
Follow [https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Disks#Partitioning_the_disk_with_GPT_for_UEFI this] setup to test. 
<syntaxhighlight lang="text">
Device        Start      End  Sectors  Size Type
/dev/sda1      2048   526335   524288  256M EFI System
/dev/sda2    526336  2623487  2097152    1G Linux swap
/dev/sda3   2623488 19400703 16777216    8G Linux filesystem
</syntaxhighlight>

{{Tip|I'd recommend playing with this in [https://www.virtualbox.org/ VirtualBox] because it is very difficult to change these on your filesystem once you have data on it. In fact, }}
{{Warning|You destroy all data when running these commands, so if you're doing this on your system just move everything off the drives before running these commands.}}

== Logical volume management (LVM) ==
LVMs abstract away drives quite a bit.
<syntaxhighlight lang="text">
                   /---------------\ /-----\ /--------\
Logical volumes    | movies         | code  |  home   |
                   \---------------/ \-----/ \--------/

                   /----------------------------------\
Volume groups      |               lvm                |
                   \----------------------------------/

                   /----------\ /---------\ /----------\
Physical volumes   | /dev/sda1 | /dev/sdb1 | /dev/sdc1 |
                   \----------/ \---------/ \----------/
</syntaxhighlight>

The main commands are:
{| class="wikitable" style="margin:auto"
|+ LVM commands
|-
! Command !! What it does
|-
| pvcreate || Add a drive to the physical volumes.
|-
| pvdisplay || View physical volumes in the LVM. 
|-
| vgcreate || Create a logical group with specified physical volumes.
|-
| vgextend || Add physical volumes to the volume group.
|-
| vgdisplay || View logical groups in the LVM. 
|-
| lvcreate || Create a logical volume within the volume group.
|-
| lvextend || Expand a logical volume within the volume group.
|-
| lvreduce || Decrease a logical volume within the volume group.
|-
| lvdisplay || View all logical volumes. 
|}

I'd highly, highly recommend following along with [https://www.howtoforge.com/linux_lvm this tutorial] within an LVM. You will be very comfortable with all of these commands after this.

{{Warning|You destroy all data when running these commands, so if you're doing this on your system just move everything off the drives before running these commands.}}

=== Demo setup ===

Let us assume we have <code>/dev/sda3</code>, <code>/dev/sdb1/</code>, <code>/dev/sdc1</code>, 3 drives ready to be added to an LVM group. We will name our volume group <code>lvm</code> because why not? This will create a folder, <code>/dev/lvm</code> that contains our logical volumes. 

First, add your drives to the physical volume list:
{{RootCmd|pvcreate /dev/sda3 /dev/sdb1 /dev/sdc1}}
Then create a volume group with them all:
{{RootCmd|vgcreate lvm /dev/sda3 /dev/sdb1 /dev/sdc1}}
Say you want your home directory in a different volume: 
{{RootCmd|lvcreate -L 30G lvm -n lv_home}}
And then use the rest of the logical volume:
{{RootCmd|lvcreate -l 100%FREE lvm -n lv_root}}
You got a new hard drive? Cool. It's <code>/dev/sdd1</code>. Add it to the volume group:
{{RootCmd|vgextend lvm /dev/sdd1}}
You ran out of storage in a volume group? Resize it:
{{RootCmd|lvextend -L 35G /dev/lvm/lv_home}}

== RAID ==
[https://wiki.gentoo.org/wiki/User:SwifT/Complete_Handbook/Software_RAID RAID] is for data protection. This is not a <em>backup</em>. It is just for protection. Instead of one drive dying and you lose all your data, you can use a RAID system so you have protection against it dying.

Instead of a table of the different RAID systems, here's a [https://www.cru-inc.com/table-raid-levels/ list of them]. I'm going to assume we use RAID 1 or RAID 10. 

RAID is done all under [https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm mdadm]. It's straightforward comapred to LVM.
{{RootCmd|mdadm --create /dev/md0 --level&#x3d;1 --raid-devices&#x3d;2 /dev/sdb1 /dev/sdc1}}
{{RootCmd|mdadm --create /dev/md0 --level&#x3d;10 --raid-devices&#x3d;2 /dev/sdb1 /dev/sdc1}}

== ZFS ==
You can use LVM and RAID by manually making sure you don't go over 50% or whatever RAID you want requires, then using mdadm on your logical volumes. Or just use ZFS to create a pool. Even less work.
{{RootCmd|zpool create pool /dev/sdb /dev/sdc}}
RAID 1:
{{RootCmd|zpool create pool mirror /dev/sdb /dev/sdc}}
RAID 10:
{{RootCmd|zpool create pool mirror /dev/sdb /dev/sdc mirror /dev/sdd /dev/sde}}
Check out what you did:
{{RootCmd|zpool status}}
Here is [https://mirrors.lug.mtu.edu/ our Linux mirrors] configuration:
<syntaxhighlight lang="text">
  pool: lug
 state: ONLINE
config:

    NAME                       STATE     READ WRITE CKSUM
    lug                        ONLINE       0     0     0
      mirror-0                 ONLINE       0     0     0
        label/lug-HGST-4TB-01  ONLINE       0     0     0
        label/lug-HGST-4TB-02  ONLINE       0     0     0
      mirror-1                 ONLINE       0     0     0
        label/lug-HGST-4TB-03  ONLINE       0     0     0
        label/lug-HGST-4TB-04  ONLINE       0     0     0
      mirror-2                 ONLINE       0     0     0
        label/lug-HGST-4TB-05  ONLINE       0     0     0
        label/lug-HGST-4TB-06  ONLINE       0     0     0
      mirror-3                 ONLINE       0     0     0
        label/lug-HGST-4TB-07  ONLINE       0     0     0
        label/lug-HGST-4TB-08  ONLINE       0     0     0
      mirror-4                 ONLINE       0     0     0
        label/lug-HGST-4TB-09  ONLINE       0     0     0
        label/lug-HGST-4TB-10  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0B in 00:01:47 with 0 errors on Sun Sep 26 16:42:05 2021
config:

    NAME        STATE     READ WRITE CKSUM
    zroot       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        da10p3  ONLINE       0     0     0
        da11p3  ONLINE       0     0     0

errors: No known data errors
</syntaxhighlight>

== fstab ==
Automatically mount your file systems at start up and tells you which file system corresponds to which drive, etc.

<code><fs> <mountpoint> <type> <opts> <dump/pass>.</code>

[https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/System#Creating_the_fstab_file It is recommended] you use UUIDs for non-LVM non-ZFS. [https://xan.manning.io/2017/05/29/best-practice-for-mounting-an-lvm-logical-volume-with-etc-fstab.html It is recommended] you use the device mapper symlink for LVM or ZFS. I am not smart enough to know the benefits for either.

Options are seen in "Filesystem independent mount options" and "Filesystem dependent mount options" in the [https://linux.die.net/man/8/mount mount(8)] man page. I'd just use <code>defaults</code>. Use <code>_netdev</code> for a network drive (cough David cough).

The numbers at the end are not too necessary, but read the [https://linux.die.net/man/5/fstab man page] on the 5th and 6th fields if you really want to know. [https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/System#Creating_the_fstab_file Gentoo] recommends having / at 0 1 and everything else as 0 2.

Here's my computer's setup:
<syntaxhighlight lang="text">
/dev/lvm/lv_root                          /     ext4 defaults         0 1 
/dev/lvm/lv_home                          /home ext4 defaults         0 2 
UUID=46ef4b37-b2bc-4a3a-b821-604c9fb64787 /boot ext4 defaults,noatime 0 2 
UUID=61518d0a-1a81-4936-85e0-6bb7e1d69155 none  swap sw               0 0
</syntaxhighlight>