Kenneth: /* Notes */

2019-02-22T16:20:27Z

Notes

New page

== mdadm ==

"mdadm is a Linux utility by Neil Brown that is used to manage RAID devices, previously known as mdctl. Besides managing, it can create, delete, or monitor Linux software RAIDs. Available under version 2 or later of the GNU General Public License, mdadm is free software. Mdadm derives its name from the “md” (multiple disk) device nodes it manages." [http://en.wikipedia.org/wiki/Mdadm]

== Commands ==

Show all md devices status:
cat /proc/mdstat

Show md device details:
mdadm --detail /dev/md0

Create RAID 0 (stripe):
mdadm --create --verbose /dev/md0 --level=raid0 --raid-devices=3 /dev/hd[cde]1

Create RAID 5 (stripe with parity):
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[cde]1
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 --spare-devices=1 /dev/sd[cdef]1

Add partition to md device:
mdadm /dev/md0 --add /dev/sdb1

Fail and remove partition of md device:
mdadm /dev/md0 --fail /dev/sdb1
mdadm /dev/md0 --remove /dev/sdb1

Start drive without resync:
mdadm -S /dev/md0
mdadm -A --assume-clean /dev/md0

Create /etc/mdadm.conf (high to low detail, any will work) [https://serverfault.com/questions/170817/software-mdadm-raid-5-inactive-md0-showing]
mdadm --detail --scan --verbose > /etc/mdadm.conf
mdadm --detail --scan > /etc/mdadm.conf
mdadm --examine --scan > /etc/mdadm.conf

# ubuntu uses /etc/mdadm/mdadm.conf
sudo mdadm --examine --scan --config=/etc/mdadm/mdadm.conf
sudo mdadm --examine --scan --config=/etc/mdadm/mdadm.conf >> /etc/mdadm/mdadm.conf
sudo mdadm -A /dev/md0

Stop array:
mdadm --stop /dev/md0

Start (assemble) array:
mdadm --assemble /dev/md0

Add disk to RAID 5:
unmount /dev/md1
mdadm --add /dev/md1 /dev/sdb3
mdadm --grow --raid-devices=4 /dev/md1

# wait for rebuild...
watch -dc cat /proc/mdstat
e2fsck -f /dev/md3
resize2fs /dev/md3

fdisk to 'Linux raid':
type 'fd'

Device Boot Start End Blocks Id System
/dev/sdb1 2048 1953525167 976761560 fd Linux raid autodetect

== Common mdadm commands ==

Generate mdadm.conf
cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.save
/usr/share/mdadm/mkconf --generate > /etc/mdadm/mdadm.conf

Create RAID
mdadm --create /dev/md2 --raid-devices=3 --spare-devices=0 --level=5 --run /dev/sd[cde]1

Remove disk from RAID
mdadm --fail /dev/md0 /dev/sda1
mdadm --remove /dev/md0 /dev/sda1

Copy the partition structure (when replacing a failed drive)
sfdisk -d /dev/sda | sfdisk /dev/sdb
mdadm --zero-superblock /dev/sdb

Add a disk to a RAID array (to replace a removed failed drive)
mdadm --add /dev/md0 /dev/sdf1

Check RAID status
cat /proc/mdstat
mdadm --detail /dev/md0

Reassemble a group of RAID disks
#This works to move an assembly from one physical machine to another.
mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1

Steps to emulate mdrun (which has been depreciated)
# haven't tested this. Use with care
mdadm --examine --scan --config=partitions > /tmp/mdadm.conf
mdadm --assemble --scan --config=/tmp/mdadm.conf

Add a disk to an existing RAID and resize the filesystem
mdadm --add /dev/md0 /dev/sdg1
mdadm --grow /dev/md0 -n 5
e2fsck -f /dev/md0
resize2fs /dev/md0
e2fsck -f /dev/md0

Replace all disks in an array with larger drives and resize
# For each drive in the existing array
mdadm --fail /dev/md0 /dev/sda1
mdadm --remove /dev/md0 /dev/sda1
# physically replace the drive
mdadm --add /dev/md0 /dev/sda1
# now, wait until md0 is rebuilt.
# this can literally take days

# All drives have been replaced and sync'd, but they still use the original size.
# Issue the following command to use all available space:
mdadm --grow /dev/md0 --size=max

# Do not forget to resized the file system which sits on the raid set:
# for ext2/3/4
e2fsck -f /dev/md0 && resize2fs /dev/md0 && e2fsck -f /dev/md0
# for lvm pv
pvresize /dev/md0
# for ntfs
ntfsresize /dev/md0
# note, most likely ntfs is NOT exported as a single partition. In the case
# of a Xen hvm machine, it is a "disk device" so you will need to resize the
# partition itself, then resize ntfs.

Stop and remove the RAID device
mdadm --stop /dev/md0
mdadm --remove /dev/md0

Destroy an existing array
mdadm --manage /dev/md2 --fail /dev/sd[cde]1
mdadm --manage /dev/md2 --remove /dev/sd[cde]1
mdadm --manage /dev/md2 --stop
mdadm --zero-superblock /dev/sd[cde]1

Speed up a sync (after drive replacement)
cat /proc/sys/dev/raid/speed_limit_max
200000

Rename an existing array
mdadm --stop /dev/md127
mdadm -A /dev/md0 -m127 --update=super-minor /dev/sd[bcd]

Source: Linux Server Tech FAQ - Common mdadm commands - http://wiki.linuxservertech.com/index.php?action=artikel&cat=7&id=11&artlang=en

== LVM ==

See [[LVM]]

== Larger than 2TB RAID ==

To get larger than 2TB you need GPTs

parted /dev/sdb
mklabel gpt
print free
mkpart primary 1M 4001GB
p
set 1 raid on
align-check
optimal
1
q

mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sd[bcd]1
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sd[bcde]1

mkfs.ext4 /dev/md0 -L /ci

cat /proc/mdstat

parted -a optimal /dev/sdf

Ref:
* server - How can I create a RAID array with >2TB disks? - Ask Ubuntu - https://askubuntu.com/questions/350266/how-can-i-create-a-raid-array-with-2tb-disks
* A guide to mdadm - Linux Raid Wiki - https://raid.wiki.kernel.org/index.php/A_guide_to_mdadm
* Using parted to create a RAID primary partition — Lucid Solutions - https://plone.lucidsolutions.co.nz/linux/io/using-parted-to-create-a-raid-primary-partition

== Handle Drive Failure ==

Fail device:
mdadm /dev/md0 --fail /dev/sdb1 # -f
mdadm: set /dev/sdb1 faulty in /dev/md0

Remove failed device:
mdadm /dev/md0 --remove /dev/sdb1

Verify failed device:
mdadm --detail /dev/md0

Add device to md:
mdadm /dev/md0 --add /dev/sdb1 # -a

Verify rebuild:
mdadm --detail /dev/md0
cat /proc/mdstat

References:
* Handling a Drive Failure - http://www.gagme.com/greg/linux/raid-lvm.php#failure

=== Force Partial Failure Recovery ===

inactive array (missing 2 devices due to io failure)
<pre>
md4 : inactive sdf3[5] sde3[4] sdd3[3] sdc3[2]
5780957696 blocks
</pre>

/etc/mdadm.conf
ARRAY /dev/md4 level=raid5 num-devices=6 UUID=f0ce1e02:2cd38a68:6ffb704c:fe6e32b0

Stop array:
mdadm --stop /dev/md4

Forcefully Rebuild Array:
mdadm -A --force /dev/md4 /dev/sd[acdef]3
# notice 'b' is missing as it is dead dead
# but 'a' is only partially dead (random io errors)

References:
* linux - How to get an inactive RAID device working again? - Super User - http://superuser.com/questions/117824/how-to-get-an-inactive-raid-device-working-again

== Bring online auto-read-only ==

<pre>
cat /proc/mdstat
Personalities : [raid1]
md1 : active (auto-read-only) raid1 sde1[1] sdd1[0]
</pre>

Bring online:

mdadm --readwrite /dev/md1

Ref:
* raid - New md array is auto-read-only and has resync=PENDING - Unix & Linux Stack Exchange - https://unix.stackexchange.com/questions/101072/new-md-array-is-auto-read-only-and-has-resync-pending
* linux - How do I reactivate my MDADM RAID5 array? - Super User - https://superuser.com/questions/603481/how-do-i-reactivate-my-mdadm-raid5-array

== Notes ==

See [[mdadm/Notes]]

== keywords ==

[[Category:Linux]]

Linux/mdadm - Revision history

Kenneth: /* Notes */