http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 1 [Check] 20 ○ 30 ○ 40 ○ 50 ○ 60 ○ 70 ○ 80 ○ 90 ○ 100 ○ 110 ○ 120 ○ 130 ○ 140 ○ 150 ○ 160 ○ 170 ○ 180 ○ 190 ○ 200 ○ 210 ○ 220 ○ 230 ○ 240 ○ 250 ○ 260 ○ 270 ○ 280 ○ 290 ○ 300 ○ 310 ○ 320 ○ 330 ○ 340 ○ 350 ○ 360 ○ 370 ○ 380 ○ 390 ○ 400 ○
Ver.1.00
2016/11/15
【想定するサーバのスペックとネットワーク構成】 CPU: 2Core 以上 Memory: 2GB 以上 DVD ドライブ: 1台 HDD 1: 8GB 以上 （1st Disk, OS 用） HDD 2: 1GB 以上 （swap 用） HDD 3: 1GB 以上 （2nd Disk, データ用） HDD 4: 1GB 以上 （3rd Disk, データ用） HDD 5: 1GB 以上 （4th Disk, データ用） HDD 6: 1GB 以上 （5th Disk, データ用） NIC 1: 1Gbps 以上、サービス用セグメント （IBM NIC 2: 1Gbps 以上、インターコネクト用セグメント（IBM NIC 3: 1Gbps 以上、サービス用セグメント （IBM NIC 4: 1Gbps 以上、インターコネクト用セグメント（IBM (eth1,eth3)bond1 .2 iscsitgt01a.example.com
Bluemix(SoftLayer) Bluemix(SoftLayer) Bluemix(SoftLayer) Bluemix(SoftLayer)
Swap Disk
Disk Disk Disk Disk
LVM(vg0) DRBD LVM(vg1)
LVM(vg0) DRBD LVM(vg1)
iSCSI lun0,1,2,3 Snapshot .57 (eth0,eth2)bond0
Private Public Private Public
VLAN）へ接続 VLAN）へ接続 VLAN）へ接続 VLAN）へ接続
bond1(eth1,eth3) .3 iscsitgt01s.example.com
1st Disk: OS
2nd 3rd 4th 5th
では では では では
[インターコネクト・セグメント] 192.168.1.0/24 MTU: 9000
1st Disk: OS
/dev/sda
Swap Disk
/dev/sdb
2nd 3rd 4th 5th
/dev/sdc /dev/sdd /dev/sde /dev/sdf
iSCSI lun0,1,2,3 Snapshot
.58 VIP bond0(eth0,eth2) ←---.59 -------------------→
Disk Disk Disk Disk
Router .1
InitiatorName: Initiator01 Initiator02 Initiator03 Initiator04 NTP1: 10.0.77.54 NTP2: NTP3: NTP4: DNS1: 10.0.80.11 DNS2: 10.0.80.12 [サービス・セグメント] 10.110.88.0/26 MTU: 9000
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 2 410 420 430 440 450 460 470 480 490 500 510 520 530 540 550 560 570 580 590 600 610 620 630 640 650 660 670
○ ○
※ 当文書内で緑色にした部分は、環境に合わせて読み替えたり、カスタマイズ（名前を変えたり、実行するしないを選択） する部分を表します。ただし、日付や注目していない UUID 等は除きます。 ※ IBM Bluemix(SoftLayer) のベアメタルサーバで NIC を冗長化した場合、 「NIC 1 (eth0)」と「NIC 3 (eth2)」、「NIC 2 (eth1)」と「NIC 4 (eth3)」が LAG で束ねられています。 ※ 本手順書では、LAG の設定がない前提としますが、LAG 対応させるための設定方法は注記しておきます。 ※ IBM Bluemix(SoftLayer) の仮想サーバは、ベアメタルサーバと比較して、主に以下の相違点があります。 ・「NIC 3」と「NIC 4」を追加できません。 ・MTU は 1500 までしかサポートされません。 ・ローカルストレージのデバイス名が異なります。
○
【共有ストレージの構成】
○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○
/dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 vg0 /dev/vg0/lv-drbd0 /dev/drbd0 vg1 /dev/vg1/lv-lun0 /dev/vg1/lv-lun1 /dev/vg1/lv-lun2 /dev/vg1/lv-lun3
LVM 物理ボリューム LVM 物理ボリューム LVM 物理ボリューム LVM 物理ボリューム LVM ボリュームグループ LVM 論理ボリューム（DRBD 用ブロックデバイスとして使用） DRBD リソース（LVM 物理ボリュームとして使用） DRBD 上のボリュームグループ DRBD 上の LVM 論理ボリューム（lun0 としてエクスポート） DRBD 上の LVM 論理ボリューム（lun1 としてエクスポート） DRBD 上の LVM 論理ボリューム（lun2 としてエクスポート） DRBD 上の LVM 論理ボリューム（lun3 としてエクスポート）
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 3 680 690 700 710 720 730 740 750 760 770 780 790 800 810 820 830 840 850 860 870 880 890 900 910 920 930 940 950 960 970 980 990 1000 1010 1020 1030 1040 1050 1060 1070
○
【OS のインストールと初期設定】
○
インストーラを DVD ドライブにセットし、サーバを起動します。
a,s ○ a,s ○ a,s a,s a,s
# V834394-01.iso (Oracle Linux 7.3) インストーラの起動メニューが表示されたら60秒以内に「Tab」キーを押下します。 # Tab 起動オプションを以下のように編集し、「Enter」キーを押下します。 # vmlinuz … rd.live.check quiet # ↓ # vmlinuz … net.ifnames=0 biosdevname=0 selinux=0 vconsole.keymap=jp106 ※ 英語キーボードとして認識されている状態なので、「=」を入力するには「^」を押下します。
○ a,s ○ a,s
○ a,s
anaconda の「Welcome」画面が出てきたら「Ctrl + Alt + F3」キーを押下し、シェルに移行します。 # Ctrl + Alt + F3 [anaconda root@localhost /]# HDD の情報を確認します。 fdisk -l | grep ^Disk | sort Disk /dev/mapper/live-base: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/mapper/live-rw: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sda: 17.2 GB, 17179869184 bytes, 33554432 sectors Disk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors Disk /dev/sdd: 107.4 GB, 107374182400 bytes, 209715200 sectors Disk /dev/sde: 107.4 GB, 107374182400 bytes, 209715200 sectors Disk /dev/sdf: 107.4 GB, 107374182400 bytes, 209715200 sectors パーティションを作成します。 fdisk -H 64 -S 32 /dev/sda Welcome to fdisk (util-linux 2.23.2).
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 4 1080 1090 1100 1110 1120 1130 1140 1150 1160 1170 1180 1190 1200 1210 1220 1230 1240 1250 1260 1270 1280 1290 1300 1310 1320 1330 1340 1350 1360 1370 1380 1390 1400 1410 1420 1430 1440 1450 1460 1470
Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x2a058c02. a,s
#Command o (m for help): o Building a new DOS disklabel with disk identifier 0xc9c2368a.
a,s
#Command n (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended #Select [Enter] (default p): [Enter] Using default response p #Partition [Enter] number (1-4, default 1): [Enter] #First [Enter] sector (2048-33554431, default 2048): [Enter] Using default value 2048 #Last +500M sector, +sectors or +size{K,M,G} (2048-33554431, default 33554431): +500M Partition 1 of type Linux and of size 500 MiB is set
a,s a,s a,s a,s a,s
#Command a (m for help): a Selected partition 1
a,s
#Command n (m for help): n Partition type: p primary (1 primary, 0 extended, 3 free) e extended #Select [Enter] (default p): [Enter] Using default response p #Partition [Enter] number (2-4, default 2): [Enter] #First [Enter] sector (1026048-33554431, default 1026048): [Enter] Using default value 1026048 #Last [Enter] sector, +sectors or +size{K,M,G} (1026048-33554431, default 33554431): [Enter] Using default value 33554431 Partition 2 of type Linux and of size 15.5 GiB is set
a,s a,s a,s a,s
a,s
#Command p (m for help): p Disk /dev/sda: 17.2 GB, 17179869184 bytes, 33554432 sectors Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 5 1480 1490 1500 1510 1520 1530 1540 1550 1560 1570 1580 1590 1600 1610 1620 1630 1640 1650 1660 1670 1680 1690 1700 1710 1720 1730 1740 1750 1760 1770 1780 1790 1800 1810 1820 1830 1840 1850 1860 1870
Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0xc9c2368a Device Boot /dev/sda1 * /dev/sda2 a,s
Start 2048 1026048
End 1026047 33554431
Blocks 512000 16264192
Id System 83 Linux 83 Linux
#Command w (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
a,s
fdisk -H 64 -S 32 /dev/sdb Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Device does not contain a recognized partition table Building a new DOS disklabel with disk identifier 0x2a058c02.
a,s
#Command o (m for help): o Building a new DOS disklabel with disk identifier 0xb3afd860.
a,s
#Command n (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended #Select [Enter] (default p): [Enter] Using default response p #Partition [Enter] number (1-4, default 1): [Enter] #First [Enter] sector (2048-2097151, default 2048): [Enter] Using default value 2048 #Last [Enter] sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): [Enter] Using default value 2097151 Partition 1 of type Linux and of size 1023 MiB is set
a,s a,s a,s a,s
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 6 1880 1890 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 2020 2030 2040 2050 2060 2070 2080 2090 2100 2110 2120 2130 2140 2150 2160 2170 2180 2190 2200 2210 2220 2230 2240 2250 2260 2270
a,s a,s a,s
#Command t (m for help): t Selected partition 1 #Hex 82code (type L to list all codes): 82 Changed type of partition 'Linux' to 'Linux swap / Solaris' #Command p (m for help): p Disk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0xb3afd860 Device Boot /dev/sdb1
a,s
Start 2048
End 2097151
Blocks 1047552
Id System 82 Linux swap / Solaris
#Command w (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
a,s
○ a,s
fdisk -l | grep /dev/ | sort /dev/sda1 * 2048 1026047 512000 83 Linux /dev/sda2 1026048 33554431 16264192 83 Linux /dev/sdb1 2048 2097151 1047552 82 Linux swap / Solaris Disk /dev/mapper/live-base: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/mapper/live-rw: 2147 MB, 2147483648 bytes, 4194304 sectors Disk /dev/sda: 17.2 GB, 17179869184 bytes, 33554432 sectors Disk /dev/sdb: 107.4 GB, 107374182400 bytes, 209715200 sectors Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors Disk /dev/sdd: 107.4 GB, 107374182400 bytes, 209715200 sectors Disk /dev/sde: 107.4 GB, 107374182400 bytes, 209715200 sectors Disk /dev/sdf: 107.4 GB, 107374182400 bytes, 209715200 sectors 「Ctrl + Alt + F6」キーを押下し、anaconda の「welcome」画面に戻ります。 # Ctrl + Alt + F6 Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 7 2280 2290 2300 2310 2320 2330 2340 2350 2360 2370 2380 2390 2400 2410 2420 2430 2440 2450 2460 2470 2480 2490 2500 2510 2520 2530 2540 2550 2560 2570 2580 2590 2600 2610 2620 2630 2640 2650 2660 2670
○ a,s ○ a,s ○ a,s ○ a,s a,s a,s ○ a,s ○ a,s ○ a,s ○ a,s ○
「English (United States)」が選択されていることを確認し、「Continue」を選択します。 # Continue 「DATE & TIME」を選択し、「Asia / Tokyo」を選択します。 # DATE & TIME: Asia / Tokyo 「KEYBOARD」を選択し、「Japanese (OADG 109A)」のみ選ばれているように選択します。 # KEYBOARD: Japanese (OADG 109A) 「INSTALLATION DESTINATION」を選択し、以下のように設定します。 デバイス名 FS MountPoint ラベル # /dev/sda1 xfs /boot /boot # /dev/sda2 xfs / / # /dev/sdb1 swap swap 「Begin Install」を選択します。 # Begin Install 「ROOT PASSWORD」を選択し、パスワードを設定します。 # ROOT PASSWORD: ******** 「Reboot」ボタンが表示されるのを待ち、「Reboot」を選択します。 # Reboot 再起動処理中に Eject されたインストーラをDVDドライブから取り外します。 # Eject DVD 再起動完了後、コンソールにてログインします。 Oracle Linux Server 7.3 Kernel 4.1.12-61.1.18.el7uek.x86_64 on an x86_64 Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 8 2680 2690 a,s 2700 a,s 2710 2720 2730 ○ 2740 2750 a,s 2760 2770 2780 2790 2800 2810 2820 2830 2840 2850 2860 2870 2880 2890 2900 ○ 2910 2920 2930 2940 ○ 2950 2960 a 2970 2980 s 2990 3000 3010 3020 ○ 3030 3040 3050 3060 ○ 3070
#localhost root login: root #Password: **************** [root@localhost ~] # MAC アドレスを確認します。 ip addr show 1: lo:
mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc mq state UP link/ether 00:0c:29:0b:ab:ce brd ff:ff:ff:ff:ff:ff 3: eth1: mtu 1500 qdisc mq state UP link/ether 00:0c:29:0b:ab:b0 brd ff:ff:ff:ff:ff:ff 4: eth2: mtu 1500 qdisc mq state UP link/ether 00:0c:29:0b:ab:ba brd ff:ff:ff:ff:ff:ff 5: eth3: mtu 1500 qdisc mq state UP link/ether 00:0c:29:0b:ab:c4 brd ff:ff:ff:ff:ff:ff
qlen 1000 qlen 1000 qlen 1000 qlen 1000
MAC アドレスをもとに、 LAN ケーブルの結線(組み合わせ)を設計通りに修正します。 ※ この後の手順で、NIC デバイス名の方を入れ替えても構いません。 IP アドレスを一時的に設定します。 ip addr add 10.110.88.57/26 dev eth0 ip addr add 10.110.88.58/26 dev eth0 ※ デバイス名が意図するものとずれている場合、デバイス名は適宜変更する必要があります。 必要に応じて、デフォルトゲートウェイを一時的に設定します。 ip route add default via 10.110.88.1 root にて、ssh でログインします。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 9 3080 a 3090 3100 3110 a 3120 3130 a 3140 3150 3160 s 3170 3180 3190 s 3200 3210 s 3220 3230 3240 ○ 3250 3260 a,s 3270 3280 3290 3300 3310 3320 3330 3340 3350 3360 3370 a,s 3380 3390 3400 3410 3420 a,s 3430 3440 3450 3460 3470
ssh [email protected] The authenticity of host '10.110.88.57 (10.110.88.57)' can't be established. ECDSA key fingerprint is 95:bc:49:71:b2:a3:dd:ab:63:ad:35:e4:fe:4d:fc:82. #Are yesyou sure you want to continue connecting (yes/no)? yes #Warning: yes Permanently added '10.110.88.57' (ECDSA) to the list of known hosts. #[email protected]'s ******** password: ******** Last login: Sat Oct 29 18:33:24 2016 ssh [email protected] The authenticity of host '10.110.88.58 (10.110.88.58)' can't be established. ECDSA key fingerprint is 8f:f6:81:0f:44:e1:83:d5:0a:9d:3f:90:7c:3e:93:73. #Are yesyou sure you want to continue connecting (yes/no)? yes #Warning: yes Permanently added '10.110.88.58' (ECDSA) to the list of known hosts. #[email protected]'s ******** password: ******** Last login: Sat Oct 29 18:33:24 2016 ストレージの情報を確認します。 fdisk -l | grep /dev/ | sort /dev/sda1 * 2048 1026047 512000 83 Linux /dev/sda2 1026048 33554431 16264192 83 Linux /dev/sdb1 2048 2097151 1047552 82 Linux swap / Solaris Disk /dev/sda: 17.2 GB, 17179869184 bytes, 33554432 sectors Disk /dev/sdb: 1073 MB, 1073741824 bytes, 2097152 sectors Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors Disk /dev/sdd: 107.4 GB, 107374182400 bytes, 209715200 sectors Disk /dev/sde: 107.4 GB, 107374182400 bytes, 209715200 sectors Disk /dev/sdf: 107.4 GB, 107374182400 bytes, 209715200 sectors blkid /dev/sda1: LABEL="/boot" UUID="0d524bba-e554-4e4c-8594-4293131808af" TYPE="xfs" /dev/sda2: LABEL="/" UUID="7fe16a0a-7b75-4e49-bef7-3b01194313fc" TYPE="xfs" /dev/sdb1: LABEL="swap" UUID="9d2b758f-05a3-4aa2-b19a-23684a062a65" TYPE="swap" cat /etc/fstab # # /etc/fstab # Created by anaconda on Sat Oct 29 18:33:24 2016 # # Accessible filesystems, by reference, are maintained under '/dev/disk' Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 10 3480 3490 3500 3510 3520 3530 3540 3550 3560 3570 3580 3590 3600 3610 3620 3630 3640 3650 3660 3670 3680 3690 3700 3710 3720 3730 3740 3750 3760 3770 3780 3790 3800 3810 3820 3830 3840 3850 3860 3870
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info # UUID=7fe16a0a-7b75-4e49-bef7-3b01194313fc / xfs defaults UUID=0d524bba-e554-4e4c-8594-4293131808af /boot xfs defaults UUID=9d2b758f-05a3-4aa2-b19a-23684a062a65 swap swap defaults ○
キーボード、ロケールの情報を確認します。
a,s
cat /etc/vconsole.conf KEYMAP="jp-OADG109A" FONT="latarcyrheb-sun16"
a,s
cat /etc/locale.conf LANG="en_US.UTF-8"
a,s
localectl status System Locale: VC Keymap: X11 Layout: X11 Variant:
○
0 0 0 0 0 0
LANG=en_US.UTF-8 jp-OADG109A jp OADG109A
タイムゾーンの情報を確認します。
a,s
cat /etc/adjtime 0.0 0 0.0 0 UTC
a,s
hwclock --debug hwclock from util-linux 2.23.2 Using /dev interface to clock. Last drift adjustment done at 0 seconds after 1969 Last calibration done at 0 seconds after 1969 Hardware clock is on UTC time Assuming hardware clock is kept in UTC time. Waiting for clock tick... ...got clock tick Time read from Hardware Clock: 2016/10/29 09:33:24 Hw clock time : 2016/10/29 09:33:24 = 1478911709 seconds since 1969 Sat 29 Oct 2016 18:33:24 AM JST -0.520512 seconds Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 11 3880 3890 3900 3910 3920 3930 3940 3950 3960 3970 3980 3990 4000 4010 4020 4030 4040 4050 4060 4070 4080 4090 4100 4110 4120 4130 4140 4150 4160 4170 4180 4190 4200 4210 4220 4230 4240 4250 4260 4270
a,s
ls -l /etc/localtime lrwxrwxrwx 1 root root 32 Oct 29 18:33 /etc/localtime -> ../usr/share/zoneinfo/Asia/Tokyo
a,s
timedatectl status Local time: Sat 2016-10-29 18:33:24 JST Universal time: Sat 2016-10-29 09:33:24 UTC RTC time: Sat 2016-10-29 09:33:24 Time zone: Asia/Tokyo (JST, +0900) NTP enabled: n/a NTP synchronized: no RTC in local TZ: no DST active: n/a
○ a,s
SELinux を無効化します。 sed -i -e 's/^SELINUX=.*$/SELINUX=disabled/' /etc/sysconfig/selinux ※ カーネルパラメータで無効化していますが、運用上紛らわしいので設定ファイルも変更します。
○
管理者用一般ユーザを作成します。
a,s
sed -i -e 's/^CREATE_MAIL_SPOOL=.*$/CREATE_MAIL_SPOOL=no/' /etc/default/useradd
a,s a,s a,s
groupadd -g 1000 admin useradd -g admin -G wheel -u 1000 admin echo 'password' | passwd --stdin admin Changing password for user admin. passwd: all authentication tokens updated successfully.
a,s
id admin uid=1000(admin) gid=1000(admin) groups=1000(admin),10(wheel)
○ a,s ○ a
wheel グループのユーザがパスワードなしで sudo コマンドを使えるように設定します。 echo '%wheel ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/wheel 管理者用一般ユーザにて、ssh でログインします。 ssh [email protected] Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 12 4280 4290 4300 4310 4320 4330 4340 4350 4360 4370 4380 4390 4400 4410 4420 4430 4440 4450 4460 4470 4480 4490 4500 4510 4520 4530 4540 4550 4560 4570 4580 4590 4600 4610 4620 4630 4640 4650 4660 4670
a
#[email protected]'s ******** password: ********
s s
ssh [email protected] #[email protected]'s ******** password: ********
○ a,s a,s ○ a,s a,s ○
wheel グループのユーザのみが su コマンドを使えるように設定します。 sudo sed -i -e '/^#auth.*required.*pam_wheel.so use_uid$/ s/#//' /etc/pam.d/su echo "SU_WHEEL_ONLY yes" | sudo tee -a /etc/login.defs root アカウントでのパスワード認証による ssh 接続を禁止します。 sudo sed -i -e 's/^#PermitRootLogin .*$/PermitRootLogin without-password/' /etc/ssh/sshd_config sudo systemctl restart sshd 参照・監視用一般ユーザを作成します。
a,s a,s a,s
sudo groupadd -g 1001 monitor sudo useradd -g monitor -u 1001 monitor echo 'password' | sudo passwd --stdin monitor Changing password for user monitor. passwd: all authentication tokens updated successfully.
a,s
id monitor uid=1001(monitor) gid=1001(monitor) groups=1001(monitor)
○ a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
NIC のデバイス名をバス情報に基づいて固定します。 sudo cp /dev/null /etc/udev/rules.d/70-persistent-net.rules NUM=0 while : do ip addr show eth$NUM > /dev/null 2>&1 || break BUS=$(ethtool -i eth$NUM | grep bus-info | awk '{print $2}') cat << EOF | sudo tee -a /etc/udev/rules.d/70-persistent-net.rules SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="$BUS", ATTR{type}=="1", NAME="eth$NUM" EOF NUM=$((NUM+1)) done SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:04:00.0", ATTR{type}=="1", NAME="eth0" Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 13 4680 4690 4700 4710 4720 4730 4740 4750 4760 4770 4780 4790 4800 4810 4820 4830 4840 4850 4860 4870 4880 4890 4900 4910 4920 4930 4940 4950 4960 4970 4980 4990 5000 5010 5020 5030 5040 5050 5060 5070
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:0b:00.0", ATTR{type}=="1", NAME="eth1" SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:13:00.0", ATTR{type}=="1", NAME="eth2" SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", KERNELS=="0000:1b:00.0", ATTR{type}=="1", NAME="eth3" ※ このファイルを編集して、NIC デバイス名を入れ替えても構いません。 ※ KERNELS=="<バス情報>" を ATTR{address}=="" に入れ替えても構いません。 ○ a,s a,s a,s a,s
OS 起動時のカーネルパラメータを変更します。 sudo sed -i -e '/^GRUB_CMDLINE_LINUX=/ s/ *biosdevname=[^ "]*//' /etc/default/grub sudo sed -i -e '/^GRUB_CMDLINE_LINUX=/ s/ *net\.ifnames=[^ "]*//' /etc/default/grub sudo sed -i -e '/^GRUB_CMDLINE_LINUX=/ s/rhgb quiet/ipv6.disable=1/' /etc/default/grub sudo grub2-mkconfig -o /boot/grub2/grub.cfg Generating grub configuration file ... Found linux image: /boot/vmlinuz-4.1.12-61.1.18.el7uek.x86_64 Found initrd image: /boot/initramfs-4.1.12-61.1.18.el7uek.x86_64.img Found linux image: /boot/vmlinuz-3.10.0-514.el7.x86_64 Found initrd image: /boot/initramfs-3.10.0-514.el7.x86_64.img Found linux image: /boot/vmlinuz-0-rescue-4305b3f1881f49358c6f848704b93e32 Found initrd image: /boot/initramfs-0-rescue-4305b3f1881f49358c6f848704b93e32.img done ※ 「net.ifnames=0」「biosdevname=0」があると、前項の udev 設定が機能しません。 ※ IPv6 を無効化しています。
○ a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
NIC を設定します。 BOND0_UUID=$(uuidgen) BOND0_BONDING_OPTS="resend_igmp=1 updelay=0 use_carrier=1 miimon=100 downdelay=0 xmit_hash_policy=0" BOND0_BONDING_OPTS="$BOND0_BONDING_OPTS primary_reselect=0 fail_over_mac=0 arp_validate=0" BOND0_BONDING_OPTS="$BOND0_BONDING_OPTS mode=active-backup primary=eth0" ※ LAGの場合は「mode=802.3ad」 BOND0_BONDING_OPTS="$BOND0_BONDING_OPTS lacp_rate=0 arp_interval=0 ad_select=0" BOND1_UUID=$(uuidgen) BOND1_BONDING_OPTS="resend_igmp=1 updelay=0 use_carrier=1 miimon=100 downdelay=0 xmit_hash_policy=0" BOND1_BONDING_OPTS="$BOND1_BONDING_OPTS primary_reselect=0 fail_over_mac=0 arp_validate=0" BOND1_BONDING_OPTS="$BOND1_BONDING_OPTS mode=active-backup primary=eth1" ※ LAGの場合は「mode=802.3ad」 BOND1_BONDING_OPTS="$BOND1_BONDING_OPTS lacp_rate=0 arp_interval=0 ad_select=0" cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-bond0 Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 14 5080 5090 5100 5110 5120 5130 5140 5150 5160 5170 5180 5190 5200 5210 5220 5230 5240 5250 5260 5270 5280 5290 5300 5310 5320 5330 5340 5350 5360 5370 5380 5390 5400 5410 5420 5430 5440 5450 5460 5470
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
DEVICE=bond0 NAME=bond0 TYPE=Bond UUID=$BOND0_UUID BONDING_OPTS="$BOND0_BONDING_OPTS" BONDING_MASTER=yes ONBOOT=yes BOOTPROTO=none DEFROUTE=yes PEERDNS=no PEERROUTES=no IPV4_FAILURE_FATAL=yes IPV6INIT=no IPV6_AUTOCONF=no IPV6_DEFROUTE=no IPV6_PEERDNS=no IPV6_PEERROUTES=no IPV6_FAILURE_FATAL=no EOF cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-bond1 DEVICE=bond1 NAME=bond1 TYPE=Bond UUID=$BOND1_UUID BONDING_OPTS="$BOND1_BONDING_OPTS" BONDING_MASTER=yes ONBOOT=yes BOOTPROTO=none DEFROUTE=no PEERDNS=no PEERROUTES=no IPV4_FAILURE_FATAL=yes IPV6INIT=no IPV6_AUTOCONF=no IPV6_DEFROUTE=no IPV6_PEERDNS=no IPV6_PEERROUTES=no IPV6_FAILURE_FATAL=no EOF Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 15 5480 5490 5500 5510 5520 5530 5540 5550 5560 5570 5580 5590 5600 5610 5620 5630 5640 5650 5660 5670 5680 5690 5700 5710 5720 5730 5740 5750 5760 5770 5780 5790 5800 5810 5820 5830 5840 5850 5860 5870
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 NAME=eth0 TYPE=Ethernet UUID=$(uuidgen) MASTER=$BOND0_UUID SLAVE=yes ONBOOT=yes MTU=9000 EOF cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth2 DEVICE=eth2 NAME=eth2 TYPE=Ethernet UUID=$(uuidgen) MASTER=$BOND0_UUID SLAVE=yes ONBOOT=yes MTU=9000 EOF cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth1 DEVICE=eth1 NAME=eth1 TYPE=Ethernet UUID=$(uuidgen) MASTER=$BOND1_UUID SLAVE=yes ONBOOT=yes MTU=9000 EOF cat << EOF | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth3 DEVICE=eth3 NAME=eth3 TYPE=Ethernet UUID=$(uuidgen) MASTER=$BOND1_UUID Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 16 5880 5890 5900 5910 5920 5930 5940 5950 5960 5970 5980 5990 6000 6010 6020 6030 6040 6050 6060 6070 6080 6090 6100 6110 6120 6130 6140 6150 6160 6170 6180 6190 6200 6210 6220 6230 6240 6250 6260 6270
a,s a,s a,s a,s a,s a a a a a a a a a a a a a a a a a s s s s s s s s s s s s s s s s ○
SLAVE=yes ONBOOT=yes MTU=9000 EOF # for Active cat << EOF | sudo tee -a /etc/sysconfig/network-scripts/ifcfg-bond0 IPADDR=10.110.88.57 PREFIX=26 GATEWAY=10.110.88.1 DNS1=10.0.80.11 DNS2=10.0.80.12 DOMAIN=example.com MTU=9000 EOF cat << EOF | sudo tee -a /etc/sysconfig/network-scripts/ifcfg-bond1 IPADDR=192.168.1.2 PREFIX=24 MTU=9000 EOF # for Stand-by cat << EOF | sudo tee -a /etc/sysconfig/network-scripts/ifcfg-bond0 IPADDR=10.110.88.58 PREFIX=26 GATEWAY=10.110.88.1 DNS1=10.0.80.11 DNS2=10.0.80.12 DOMAIN=example.com MTU=9000 EOF cat << EOF | sudo tee -a /etc/sysconfig/network-scripts/ifcfg-bond1 IPADDR=192.168.1.3 PREFIX=24 MTU=9000 EOF NIC オフロード機能を無効化します。 Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 17 6280 6290 6300 6310 6320 6330 6340 6350 6360 6370 6380 6390 6400 6410 6420 6430 6440 6450 6460 6470 6480 6490 6500 6510 6520 6530 6540 6550 6560 6570 6580 6590 6600 6610 6620 6630 6640 6650 6660 6670
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
cat << 'EOF' | sudo tee /etc/NetworkManager/dispatcher.d/00-ethertool #!/bin/sh if [ "$2" == "up" ]; then if [ "${1:0:3}" == "eth" ]; then ethtool -K $1 \ rx off \ tx off \ sg off \ tso off \ ufo off \ gso off \ gro off \ lro off \ rxvlan off \ txvlan off \ ntuple off \ rxhash off \ highdma off \ rx-vlan-filter off \ tx-gso-robust off \ tx-fcoe-segmentation off \ fcoe-mtu off \ tx-nocache-copy off \ loopback off \ rx-fcs off \ rx-all off #ethtool -K $1 vlan-challenged off tx-lockless off netns-local off ethtool -G $1 rx 4096 tx 4096 rx-jumbo 2048 fi fi EOF sudo chmod 755 /etc/NetworkManager/dispatcher.d/00-ethertool ※ 「VMXNET 3」でのみ動作確認しました。 ※ NIC のリンク速度、duplex モードを設定したい場合はこのスクリプトに組み込みます。
○ a,s
hosts を設定します。 cat << 'EOF' | sudo tee /etc/hosts Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 18 6680 6690 6700 6710 6720 6730 6740 6750 6760 6770 6780 6790 6800 6810 6820 6830 6840 6850 6860 6870 6880 6890 6900 6910 6920 6930 6940 6950 6960 6970 6980 6990 7000 7010 7020 7030 7040 7050 7060 7070
a,s a,s a,s a,s a,s a,s a,s a,s ○
127.0.0.1 ::1 10.110.88.57 10.110.88.58 10.110.88.59 192.168.1.2 192.168.1.3 EOF
localhost localhost.localdomain localhost4 localhost4.localdomain4 localhost localhost.localdomain localhost6 localhost6.localdomain6 iscsitgt01a.example.com iscsitgt01a iscsitgt01s.example.com iscsitgt01s iscsitgt01.example.com iscsitgt01 iscsitgt01a-ic.example.com iscsitgt01a-ic iscsitgt01s-ic.example.com iscsitgt01s-ic
hostname を設定します。
a
sudo hostnamectl set-hostname iscsitgt01a.example.com
s
sudo hostnamectl set-hostname iscsitgt01s.example.com
○ a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
yum リポジトリを設定します。 cat << 'EOF' | sudo tee /etc/yum.repos.d/media.repo [media] name=media baseurl=file:///mnt gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle gpgcheck=1 enabled=0 [media-mysql] name=media-mysql baseurl=file:///mnt/addons/Mysql gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle gpgcheck=1 enabled=0 [media-ha] name=media-ha baseurl=file:///mnt/addons/HighAvailability gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle gpgcheck=1 enabled=0 [media-rs] Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 19 7080 7090 7100 7110 7120 7130 7140 7150 7160 7170 7180 7190 7200 7210 7220 7230 7240 7250 7260 7270 7280 7290 7300 7310 7320 7330 7340 7350 7360 7370 7380 7390 7400 7410 7420 7430 7440 7450 7460 7470
a,s a,s a,s a,s a,s a,s
name=media-rs baseurl=file:///mnt/addons/ResilientStorage gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle gpgcheck=1 enabled=0 EOF ※ インストール・メディアを利用可能にします。
a,s
sudo sed -i -e 's/^/#/' /etc/yum.repos.d/public-yum-ol7.repo ※ インターネット接続していないとエラーとなるリポジトリを無効化します。
○ a,s ○ a,s ○ a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
以下のインストーラを DVD ドライブにセットします。 # V834394-01.iso (Oracle Linux 7.3) インストーラをマウントします。 sudo mount /dev/cdrom /mnt mount: /dev/sr0 is write-protected, mounting read-only どのような環境でも共通して導入しておいた方がよいと思われる標準パッケージをインストールします。 sudo yum -y --disablerepo=\* --enablerepo=media,media-mysql install \ @development \ @base \ OpenIPMI \ aide \ dos2unix \ dropwatch \ dstat \ expect \ filebench \ freeipmi-bmc-watchdog \ freeipmi-ipmidetectd \ ftp \ haproxy \ hdparm \ iotop \ Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 20 7480 7490 7500 7510 7520 7530 7540 7550 7560 7570 7580 7590 7600 7610 7620 7630 7640 7650 7660 7670 7680 7690 7700 7710 7720 7730 7740 7750 7760 7770 7780 7790 7800 7810 7820 7830 7840 7850 7860 7870
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s ○ a,s ○ a,s ○ a,s a,s
ipmitool \ iptables-services \ iptraf-ng \ iptstate \ ipvsadm \ iscsi-initiator-utils \ keepalived \ kernel-uek-devel \ latrace \ lftp \ logwatch \ lrzsz \ ltrace \ nmap \ openssl-devel \ oprofile \ pax \ perf \ prelink \ screen \ sg3_utils \ snapper \ telnet \ tmpwatch \ trace-cmd \ tree \ x86info インストーラをアンマウントします。 sudo umount /mnt インストーラをDVDドライブから取り外します。 # Eject DVD NTP を設定します。 cat << 'EOF' | sudo tee /etc/chrony.conf server 10.0.77.54 iburst Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 21 7880 7890 7900 7910 7920 7930 7940 7950 7960 7970 7980 7990 8000 8010 8020 8030 8040 8050 8060 8070 8080 8090 8100 8110 8120 8130 8140 8150 8160 8170 8180 8190 8200 8210 8220 8230 8240 8250 8260 8270
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
# server ***.***.***.*** # server ***.***.***.*** # server ***.***.***.*** EOF # sudo sed -i -e '/^#/d'
iburst iburst iburst /etc/chrony.conf
cat << 'EOF' | sudo tee -a /etc/chrony.conf # Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). # Ignore stratum in source selection. stratumweight 0 # Record the rate at which the system clock gains/losses time. driftfile /var/lib/chrony/drift # Enable kernel RTC synchronization. rtcsync # In first three updates step the system clock instead of slew # if the adjustment is larger than 10 seconds. makestep 10 3 # Allow NTP client access from local network. #allow 192.168/16 # Listen for commands only on localhost. bindcmdaddress 127.0.0.1 #bindcmdaddress ::1 # Serve time even if not synchronized to any NTP server. #local stratum 10 keyfile /etc/chrony.keys # Specify the key used as password for chronyc. commandkey 1 # Generate command key if missing. Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 22 8280 8290 8300 8310 8320 8330 8340 8350 8360 8370 8380 8390 8400 8410 8420 8430 8440 8450 8460 8470 8480 8490 8500 8510 8520 8530 8540 8550 8560 8570 8580 8590 8600 8610 8620 8630 8640 8650 8660 8670
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s ○ a,s a,s a,s a,s
generatecommandkey # Disable logging of client accesses. noclientlog # Send a message to syslog if a clock adjustment is larger than 0.5 seconds. logchange 0.5 logdir /var/log/chrony #log measurements statistics tracking EOF cat << 'EOF' | sudo tee /etc/sysconfig/chronyd OPTIONS="-4" EOF 不要なサービスを無効化します。 sudo sudo sudo sudo
systemctl systemctl systemctl systemctl
disable disable disable disable
dmraid-activation.service firewalld.service mdmonitor.service postfix.service
※ 仮想環境の場合は、「smartd.service」も無効化します。RAID コントローラが対応していない場合も無効化します。 ○ a,s ○
必要なサービスを有効化します。 sudo systemctl enable psacct.service iSCSI イニシエータ関連サービスの自動起動を無効化します。
a,s
sudo systemctl disable iscsi.service Removed symlink /etc/systemd/system/sysinit.target.wants/iscsi.service.
a,s
sudo systemctl disable iscsid.socket Removed symlink /etc/systemd/system/sockets.target.wants/iscsid.socket.
a,s
sudo systemctl disable iscsiuio.socket Removed symlink /etc/systemd/system/sockets.target.wants/iscsiuio.socket.
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 23 8680 8690 8700 8710 8720 8730 8740 8750 8760 8770 8780 8790 8800 8810 8820 8830 8840 8850 8860 8870 8880 8890 8900 8910 8920 8930 8940 8950 8960 8970 8980 8990 9000 9010 9020 9030 9040 9050 9060 9070
○
IPv6 無効化に伴う不具合を解消するための設定変更を行います。
a,s a,s
sudo sed -i -e 's/^#AddressFamily .*$/AddressFamily inet/' /etc/ssh/sshd_config sudo sed -i -e 's/^inet_interfaces .*$/inet_interfaces = 127.0.0.1/' /etc/postfix/main.cf
a,s
sudo sed -i -e 's/^udp6/#udp6/' -e 's/^tcp6/#tcp6/' /etc/netconfig
○ a,s ○
再起動します。 sudo reboot 管理者用一般ユーザにて、ssh でログインします。
a a
ssh [email protected] #[email protected]'s ******** password: ********
s s
ssh [email protected] #[email protected]'s ******** password: ********
○ a,s
カーネル起動パラメータを確認します。 cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.1.12-61.1.18.el7uek.x86_64 root=UUID=657f59aa-f627-4096-9970-9238b234ef00 ro crashkernel=auto selinux=0 ipv6.disable=1
※ 「crashkernel」の値は、搭載メモリサイズに応じて自動的に固定値へ変更される場合があります。 ○
kdump の設定を確認します。
a,s
systemctl is-enabled kdump.service enabled
a,s
sudo kdumpctl status Kdump is operational
○ a,s
SELinux の設定を確認します。 grep -v ^# /etc/sysconfig/selinux SELINUX=disabled SELINUXTYPE=targeted
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 24 9080 a,s 9090 9100 9110 ○ 9120 9130 a,s 9140 9150 9160 9170 9180 9190 9200 9210 9220 9230 9240 9250 9260 9270 9280 9290 9300 9310 9320 9330 9340 a,s 9350 9360 9370 9380 9390 9400 9410 9420 9430 9440 9450 9460 9470
getenforce Disabled ネットワーク設定を確認します。 ip addr show 1: lo: mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo 2: eth0: mtu 9000 qdisc mq master link/ether 00:0c:29:0b:ab:c4 brd ff:ff:ff:ff:ff:ff 3: eth1: mtu 9000 qdisc mq master link/ether 00:0c:29:0b:ab:ce brd ff:ff:ff:ff:ff:ff 4: eth2: mtu 9000 qdisc mq master link/ether 00:0c:29:0b:ab:c4 brd ff:ff:ff:ff:ff:ff 5: eth3: mtu 9000 qdisc mq master link/ether 00:0c:29:0b:ab:ce brd ff:ff:ff:ff:ff:ff 6: bond0: mtu 9000 qdisc noqueue link/ether 00:0c:29:0b:ab:c4 brd ff:ff:ff:ff:ff:ff inet 10.110.88.57/26 brd 10.110.88.63 scope global bond0 7: bond1: mtu 9000 qdisc noqueue link/ether 00:0c:29:0b:ab:ce brd ff:ff:ff:ff:ff:ff inet 192.168.1.2/24 brd 192.168.0.255 scope global bond1
bond0 state UP qlen 1000 bond1 state UP qlen 1000 bond0 state UP qlen 1000 bond1 state UP qlen 1000 state UNKNOWN state UNKNOWN
※ IPv6 のリンクローカルアドレスが存在しないことも確認します。 cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: fault-tolerance (active-backup) Primary Slave: eth0 (primary_reselect always) Currently Active Slave: eth0 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth0 MII Status: up Speed: 10000 Mbps Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 25 9480 9490 9500 9510 9520 9530 9540 9550 9560 9570 9580 9590 9600 9610 a,s 9620 9630 9640 9650 9660 9670 9680 9690 9700 9710 9720 9730 9740 9750 9760 9770 9780 9790 9800 9810 9820 9830 9840 9850 9860 9870
Duplex: full Link Failure Count: 0 Permanent HW addr: 00:0c:29:0b:ab:b0 Slave queue ID: 0 Slave Interface: eth2 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:0c:29:0b:ab:c4 Slave queue ID: 0 cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: fault-tolerance (active-backup) Primary Slave: eth1 (primary_reselect always) Currently Active Slave: eth1 MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 Slave Interface: eth1 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:0c:29:0b:ab:ba Slave queue ID: 0 Slave Interface: eth3 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:0c:29:0b:ab:ce Slave queue ID: 0
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 26 9880 9890 9900 9910 9920 9930 9940 9950 9960 9970 9980 9990 10000 10010 10020 10030 10040 10050 10060 10070 10080 10090 10100 10110 10120 10130 10140 10150 10160 10170 10180 10190 10200 10210 10220 10230 10240 10250 10260 10270
※ ボンディング設定時は、このコマンドで個々の NIC の MAC アドレスを確認できます。 ※ LAG の場合、以下のように表示されます。 cat /proc/net/bonding/bond1 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: slow Min links: 0 Aggregator selection policy (ad_select): stable Active Aggregator Info: Aggregator ID: 1 Number of ports: 1 Actor Key: 13 Partner Key: 1 Partner Mac Address: 00:00:00:00:00:00 Slave Interface: eth1 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:0c:29:8b:ff:e8 Slave queue ID: 0 Aggregator ID: 1 Actor Churn State: none Partner Churn State: churned Actor Churned Count: 0 Partner Churned Count: 1 details actor lacp pdu: system priority: 0 port key: 13 port priority: 255 Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 27 10280 10290 10300 10310 10320 10330 10340 10350 10360 10370 10380 10390 10400 10410 10420 10430 10440 10450 10460 10470 10480 10490 10500 10510 10520 10530 10540 10550 10560 10570 10580 10590 10600 10610 10620 10630 10640 10650 10660 10670
port number: 1 port state: 205 details partner lacp pdu: system priority: 65535 oper key: 1 port priority: 255 port number: 1 port state: 3 Slave Interface: eth3 MII Status: up Speed: 10000 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:0c:29:8b:ff:fc Slave queue ID: 0 Aggregator ID: 2 Actor Churn State: churned Partner Churn State: churned Actor Churned Count: 1 Partner Churned Count: 1 details actor lacp pdu: system priority: 0 port key: 13 port priority: 255 port number: 2 port state: 197 details partner lacp pdu: system priority: 65535 oper key: 1 port priority: 255 port number: 1 port state: 3 a
ip route show default via 10.110.88.1 dev bond0 proto static metric 300 10.110.88.0/26 dev bond0 proto kernel scope link src 10.110.88.57 metric 300 192.168.1.0/24 dev bond1 proto kernel scope link src 192.168.0.2 metric 300
s
ip route show Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 28 10680 10690 10700 10710 10720 a,s 10730 10740 10750 10760 10770 10780 ○ 10790 10800 a 10810 10820 10830 10840 10850 10860 10870 10880 10890 10900 10910 10920 s 10930 10940 10950 10960 10970 10980 10990 11000 11010 11020 11030 11040 ○ 11050 11060 a,s 11070
default via 10.110.88.1 dev bond0 proto static metric 300 10.110.88.0/26 dev bond0 proto kernel scope link src 10.110.88.58 metric 300 192.168.1.0/24 dev bond1 proto kernel scope link src 192.168.0.3 metric 300 cat /etc/resolv.conf # Generated by NetworkManager search example.com nameserver 10.0.80.11 nameserver 10.0.80.12 hostname 設定を確認します。 hostnamectl status Static hostname: Icon name: Chassis: Machine ID: Boot ID: Virtualization: Operating System: CPE OS Name: Kernel: Architecture:
iscsitgt01a.example.com computer-vm vm d7806eba789047baa165a57149c83843 b5b36a3403dd403aad4656d2f7f9e7aa vmware Oracle Linux Server 7.3 cpe:/o:oracle:linux:7:2:server Linux 4.1.12-61.1.18.el7uek.x86_64 x86-64
hostnamectl status Static hostname: Icon name: Chassis: Machine ID: Boot ID: Virtualization: Operating System: CPE OS Name: Kernel: Architecture:
iscsitgt01s.example.com computer-vm vm b325c1c5d682439a91a65f7cfc317b20 a419d4d1ef00452f93da10a227365aca vmware Oracle Linux Server 7.3 cpe:/o:oracle:linux:7:2:server Linux 4.1.12-61.1.18.el7uek.x86_64 x86-64
NIC のオフロード設定を確認します。 ethtool -k eth0 Features for eth0: Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 29 11080 11090 11100 11110 11120 11130 11140 11150 11160 11170 11180 11190 11200 11210 11220 11230 11240 11250 11260 11270 11280 11290 11300 11310 11320 11330 11340 11350 11360 11370 11380 11390 11400 11410 11420 11430 11440 11450 11460 11470
rx-checksumming: off tx-checksumming: off tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: off tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off rx-vlan-offload: off tx-vlan-offload: off ntuple-filters: off [fixed] receive-hashing: off highdma: off rx-vlan-filter: on [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 30 11480 11490 11500 11510 11520 11530 11540 11550 11560 11570 11580 11590 11600 11610 11620 11630 11640 11650 11660 11670 11680 11690 11700 11710 11720 11730 11740 11750 11760 11770 11780 11790 11800 11810 11820 11830 11840 11850 11860 11870
rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] busy-poll: off [fixed] hw-switch-offload: off [fixed] a,s a,s a,s
ethtool -k eth1 ethtool -k eth2 ethtool -k eth3
a,s
ethtool -g eth0 Ring parameters for eth0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 2048 TX: 4096 Current hardware settings: RX: 4032 RX Mini: 0 RX Jumbo: 2048 TX: 4096 ※ 搭載メモリサイズに応じて結果が異なります。
a,s a,s a,s ○ a,s
ethtool -g eth1 ethtool -g eth2 ethtool -g eth3 NTP の状態を確認します。 systemctl status chronyd.service -l ● chronyd.service - NTP client/server Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2016-10-29 18:33:24 JST; 42min left Process: 601 ExecStartPost=/usr/libexec/chrony-helper update-daemon (code=exited, status=0/SUCCESS) Process: 576 ExecStart=/usr/sbin/chronyd $OPTIONS (code=exited, status=0/SUCCESS) Main PID: 583 (chronyd) CGroup: /system.slice/chronyd.service ┗583 /usr/sbin/chronyd -4
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 31 11880 11890 11900 11910 11920 11930 11940 a,s 11950 11960 11970 11980 11990 12000 a,s 12010 12020 12030 12040 12050 12060 12070 12080 12090 12100 ○ 12110 12120 a,s 12130 12140 12150 12160 12170 12180 12190 12200 12210 12220 12230 12240 12250 12260 12270
Oct Oct Oct Oct Oct
29 29 29 29 29
18:33:24 18:33:24 18:33:24 18:33:24 18:33:24
iscsitgt01a.example.com iscsitgt01a.example.com iscsitgt01a.example.com iscsitgt01a.example.com iscsitgt01a.example.com
systemd[1]: Starting NTP client/server... chronyd[584]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH) chronyd[584]: Generated key 1 systemd[1]: Started NTP client/server. chronyd[584]: Selected source 10.0.77.54
chronyc sources 210 Number of sources = 1 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* 10.0.77.54 1 10 377 217 -177us[ -161us] +/- 4360us timedatectl status Local time: Sat 2016-10-29 18:33:24 JST Universal time: Sat 2016-10-29 09:33:24 UTC RTC time: Sat 2016-10-29 09:33:24 Time zone: Asia/Tokyo (JST, +0900) NTP enabled: yes NTP synchronized: yes RTC in local TZ: no DST active: n/a 自動起動するサービスを確認します。 systemctl list-unit-files | grep enabled | NetworkManager-dispatcher.service NetworkManager.service abrt-ccpp.service abrt-oops.service abrt-vmcore.service abrt-xorg.service abrtd.service atd.service auditd.service [email protected] chronyd.service crond.service dbus-org.freedesktop.NetworkManager.service dbus-org.freedesktop.nm-dispatcher.service default.target
LANG=C sort enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 32 12280 12290 12300 12310 12320 12330 12340 12350 12360 12370 12380 12390 12400 12410 12420 12430 12440 12450 12460 12470 12480 12490 12500 12510 12520 12530 12540 12550 ○ 12560 12570 a,s 12580 12590
dm-event.socket [email protected] irqbalance.service kdump.service libstoragemgmt.service lvm2-lvmetad.socket lvm2-lvmpolld.socket lvm2-monitor.service microcode.service psacct.service multi-user.target remote-fs.target rngd.service rpcbind.socket rsyslog.service runlevel2.target runlevel3.target runlevel4.target smartd.service sshd.service sysstat.service systemd-readahead-collect.service systemd-readahead-drop.service systemd-readahead-replay.service tuned.service vmtoolsd.service
enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled enabled
※ LVM を利用しない場合、不要 ※ LVM を利用しない場合、不要 ※ LVM を利用しない場合、不要
※ nfs, iSCSI イニシエータを利用しない場合、不要 ※ nfs を利用しない場合、不要
※ 仮想環境、RAID コントローラ未対応の場合、不要
※ ESXi で動かす場合のみ必要
設定ファイルをバックアップします。 sudo cp -a /etc{,~}
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 33 12600 12610 12620 12630 12640 12650 12660 12670 12680 12690 12700 12710 12720 12730 12740 12750 12760 12770 12780 12790 12800 12810 12820 12830 12840 12850 12860 12870 12880 12890 12900 12910 12920 12930 12940 12950 12960 12970 12980 12990
○
【iSCSI Target クラスタのインストールと初期設定】
○
以下のインストーラを DVD ドライブにセットします。
a,s ○ a,s ○ a,s a,s a,s a,s a,s a,s ○ o ○
# V834394-01.iso (Oracle Linux 7.3) インストーラをマウントします。 sudo mount /dev/cdrom /mnt mount: /dev/sr0 is write-protected, mounting read-only 当該 OS で必要となる標準パッケージをインストールします。 sudo yum -y --disablerepo=\* --enablerepo=media,media-mysql,media-ha install \ fence-agents-ipmilan \ omping \ pcs \ rubygem-abrt \ targetcli インターネットと接続可能な端末で以下のコマンドを実行する等して、必要なパッケージを収集します。 curl -O http://elrepo.org/linux/elrepo/el7/x86_64/RPMS/drbd84-utils-8.9.6-1.el7.elrepo.x86_64.rpm
収集したパッケージをホームディレクトリにコピーし、確認します。
a,s
scp xxxx@yyy:drbd84-utils-8.9.6-1.el7.elrepo.x86_64.rpm .
a,s
ls -l *.rpm -rw-rw-r-- 1 admin admin
a,s ○ a,s ○
410308 Oct 29 18:33 drbd84-utils-8.9.6-1.el7.elrepo.x86_64.rpm
file *.rpm drbd84-utils-8.9.6-1.el7.elrepo.x86_64.rpm:
RPM v3.0 bin i386/x86_64 drbd84-utils-8.9.6-1.el7.elrepo
drbd 管理ツールをインストールします。Oracle 社サポート外のパッケージです。 sudo yum -y --disablerepo=\* --enablerepo=media localinstall drbd84-utils-*.rpm インストーラをアンマウントします。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 34 13000 13010 13020 13030 13040 13050 13060 13070 13080 13090 13100 13110 13120 13130 13140 13150 13160 13170 13180 13190 13200 13210 13220 13230 13240 13250 13260 13270 13280 13290 13300 13310 13320 13330 13340 13350 13360 13370 13380 13390
a,s ○ a,s ○
sudo umount /mnt インストーラをDVDドライブから外します。 # Eject DVD 追加インストールしたパッケージの設定をバックアップします。
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s ○
sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo sudo
cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp cp
-a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a -a
/etc{,~}/bash_completion.d/drbdadm /etc{,~}/corosync /etc{,~}/dbus-1/system.d/corosync-signals.conf /etc{,~}/drbd.conf /etc{,~}/drbd.d /etc{,~}/ha.d /etc{,~}/libreport/events.d/ruby_event.conf /etc{,~}/logrotate.d/corosync /etc{,~}/logrotate.d/pacemaker /etc{,~}/logrotate.d/pcsd /etc{,~}/pam.d/pcsd /etc{,~}/sysconfig/corosync /etc{,~}/sysconfig/corosync-notifyd /etc{,~}/sysconfig/crm_mon /etc{,~}/sysconfig/ipmievd /etc{,~}/sysconfig/pacemaker /etc{,~}/sysconfig/pcsd /etc{,~}/target /etc{,~}/xen /etc/passwd /etc~/passwd_$(date +%Y%m%d_%H%M%S) /etc/passwd- /etc~/passwd-_$(date +%Y%m%d_%H%M%S) /etc/shadow /etc~/shadow_$(date +%Y%m%d_%H%M%S) /etc/shadow- /etc~/shadow-_$(date +%Y%m%d_%H%M%S) /etc/group /etc~/group_$(date +%Y%m%d_%H%M%S) /etc/group/etc~/group-_$(date +%Y%m%d_%H%M%S) /etc/gshadow /etc~/gshadow_$(date +%Y%m%d_%H%M%S) /etc/gshadow- /etc~/gshadow-_$(date +%Y%m%d_%H%M%S)
a,s a,s
cat << 'EOF' | sudo tee /etc/sysctl.d/tgt.conf net.core.netdev_max_backlog = 250000
カーネルパラメータを設定します。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 35 13400 13410 13420 13430 13440 13450 13460 13470 13480 13490 13500 13510 13520 13530 13540 13550 13560 13570 13580 13590 13600 13610 13620 13630 13640 13650 13660 13670 13680 13690 13700 13710 13720 13730 13740 13750 13760 13770 13780 13790
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s ○ a,s ○
net.core.optmem_max = 16777216 net.core.rmem_default = 16777216 net.core.rmem_max = 16777216 net.core.wmem_default = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_mem = 39363 209944 314904 net.ipv4.tcp_rmem = 8192 87380 16777216 net.ipv4.tcp_wmem = 8192 65536 16777216 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_sack = 0 net.ipv4.tcp_timestamps = 0 EOF 再起動します。 sudo reboot 管理者用一般ユーザにて、ssh でログインします。
a a
ssh [email protected] #[email protected]'s ******** password: ********
s s
ssh [email protected] #[email protected]'s ******** password: ********
○ a,s
カーネルパラメータを確認します。 sysctl -a 2> /dev/null | egrep 'net\.core\..*mem|net\.core\.netdev_m|net\.ipv4\.tcp_.*mem|net\.ipv4\.tcp_no_|net\.ipv4\.tcp_sa|net\.ipv4\.tcp_ti'
net.core.netdev_max_backlog = 250000 net.core.optmem_max = 16777216 net.core.rmem_default = 16777216 net.core.rmem_max = 16777216 net.core.wmem_default = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_mem = 39363 209944 314904 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_rmem = 8192 87380 16777216 net.ipv4.tcp_sack = 0 net.ipv4.tcp_timestamps = 0 net.ipv4.tcp_wmem = 8192 65536 16777216 Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 36 13800 13810 13820 13830 13840 13850 13860 13870 13880 13890 13900 13910 13920 13930 13940 13950 13960 13970 13980 13990 14000 14010 14020 14030 14040 14050 14060 14070 14080 14090 14100 14110 14120 14130 14140 14150 14160 14170 14180 14190
○ a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s ○ a,s a,s a,s a,s a,s a,s a,s a,s a,s
LVM の設定を変更します。 sudo sed -i -e 's/obtain_device_list_from_udev =.*$/obtain_device_list_from_udev = 0/' /etc/lvm/lvm.conf sudo sed -i -e 's/use_blkid_wiping =.*$/use_blkid_wiping = 0/' /etc/lvm/lvm.conf sudo sed -i -e 's/use_lvmetad =.*$/use_lvmetad = 0/' /etc/lvm/lvm.conf sudo sed -i -e 's/use_lvmpolld =.*$/use_lvmpolld = 0/' /etc/lvm/lvm.conf sudo sed -i -e 's/write_cache_state =.*$/write_cache_state = 0/' /etc/lvm/lvm.conf sudo patch --ignore-whitespace /etc/lvm/lvm.conf << 'EOF' diff -upr /etc/lvm/lvm.conf /etc/lvm/lvm.conf.new --- /etc/lvm/lvm.conf 2015-11-21 12:01:29.000000000 +0900 +++ /etc/lvm/lvm.conf.new 2016-09-21 07:52:14.164259555 +0900 @@ -139,6 +139,7 @@ devices { # # This configuration option has an automatic default value. # filter = [ "a|.*/|" ] +filter = ["r|vg.*|", "a|sd.*|", "a|drbd.*|", "r|.*|"] # Configuration option devices/global_filter. # Limit the block devices that are used by LVM system components. EOF sudo sudo sudo sudo sudo sudo
systemctl stop lvm2-lvmetad.socket systemctl stop lvm2-lvmpolld.socket systemctl disable lvm2-lvmetad.socket systemctl disable lvm2-lvmpolld.socket rm -f /etc/lvm/cache/.cache cp -a /etc/lvm/lvm.conf /etc~/lvm/lvm.conf_$(date +%Y%m%d_%H%M%S)
LVM の設定変更を初期化 RAM ディスクに反映します。 for i in /boot/initramfs-* do KVER=$(echo $i | sed -n 's%/boot/initramfs-\(.*\).img%\1%p') if echo $KVER | grep -q -v rescue; then if echo $KVER | grep -q -v kdump; then sudo dracut --force /boot/initramfs-$KVER.img $KVER; fi fi done
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 37 14200 14210 14220 14230 14240 14250 14260 14270 14280 14290 14300 14310 14320 14330 14340 14350 14360 14370 14380 14390 14400 14410 14420 14430 14440 14450 14460 14470 14480 14490 14500 14510 14520 14530 14540 14550 14560 14570 14580 14590
○ a,s a,s a,s a,s a,s
LVM 物理ボリューム用パーティションを作成します。 echo Yes | sudo parted /dev/sdc mklabel gpt echo Yes | sudo parted /dev/sdd mklabel gpt echo Yes | sudo parted /dev/sde mklabel gpt echo Yes | sudo parted /dev/sdf mklabel gpt sudo parted -l Model: VMware Virtual disk (scsi) Disk /dev/sda: 17.2GB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number 1 2
Start 1049kB 525MB
End 525MB 17.2GB
mkpart mkpart mkpart mkpart
primary primary primary primary
1MiB 1MiB 1MiB 1MiB
100% 100% 100% 100%
set set set set
1 1 1 1
lvm lvm lvm lvm
on on on on
Size Type File system Flags 524MB primary xfs boot 16.7GB primary xfs
Model: VMware Virtual disk (scsi) Disk /dev/sdb: 1074MB Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags: Number 1
Start 1049kB
End 1074MB
Size Type File system Flags 1073MB primary linux-swap(v1)
Model: VMware Virtual disk (scsi) Disk /dev/sdc: 107GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number 1
Start 1049kB
End 107GB
Size 107GB
File system Name Flags primary lvm
Model: VMware Virtual disk (scsi) Disk /dev/sdd: 107GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 38 14600 14610 14620 14630 14640 14650 14660 14670 14680 14690 14700 14710 14720 14730 14740 14750 14760 14770 14780 14790 14800 14810 14820 14830 14840 14850 14860 14870 14880 14890 14900 14910 14920 14930 14940 14950 14960 14970 14980 14990
Number 1
Start 1049kB
End 107GB
Size 107GB
File system Name Flags primary lvm
Model: VMware Virtual disk (scsi) Disk /dev/sde: 107GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number 1
Start 1049kB
End 107GB
Size 107GB
File system Name Flags primary lvm
Model: VMware Virtual disk (scsi) Disk /dev/sdf: 107GB Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number 1 ○
Start 1049kB
End 107GB
Size 107GB
File system Name Flags primary lvm
LVM 物理ボリュームを作成します。
a,s
sudo pvcreate /dev/sdc1 Physical volume "/dev/sdc1" successfully created
a,s
sudo pvcreate /dev/sdd1 Physical volume "/dev/sdd1" successfully created
a,s
sudo pvcreate /dev/sde1 Physical volume "/dev/sde1" successfully created
a,s
sudo pvcreate /dev/sdf1 Physical volume "/dev/sdf1" successfully created
○ a,s
LVM ボリュームグループを作成します。 sudo vgcreate -s 4M vg0 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 Volume group "vg0" successfully created Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 39 15000 15010 15020 15030 15040 15050 15060 15070 15080 15090 15100 15110 15120 15130 15140 15150 15160 15170 15180 15190 15200 15210 15220 15230 15240 15250 15260 15270 15280 15290 15300 15310 15320 15330 15340 15350 15360 15370 15380 15390
○ a,s ○ a,s
LVM 論理ボリュームを作成します。 sudo lvcreate --name lv-drbd0 --extents 90%FREE vg0 Logical volume "lv-drbd0" created. LVM の状態を確認します。 sudo pvs PV /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
VG vg0 vg0 vg0 vg0
Fmt lvm2 lvm2 lvm2 lvm2
Attr a-a-a-a--
PSize PFree 100.00g 0 100.00g 0 100.00g 0 100.00g 40.00g
a,s
sudo vgs VG #PV #LV #SN Attr VSize VFree vg0 4 1 0 wz--n- 399.98g 40.00g
a,s
sudo lvs LV VG lv-drbd0 vg0
○ a,s
Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert -wi-a----- 359.98g
DRBD の設定ファイルを作成します。 cat /etc/drbd.conf # You can find an example in
/usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf"; include "drbd.d/*.res"; a,s a,s a,s a,s a,s a,s a,s a,s a,s
cat << 'EOF' | sudo tee /etc/drbd.d/global_common.conf global { usage-count no; } common { handlers { pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo 1 > /proc/sys/kernel/sysrq; echo b > /proc/sysrq-trigger; reboot -f";
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo 1 > /proc/sys/kernel/sysrq; echo o > /proc/sysrq-trigger; halt -f";
fence-peer "/usr/lib/drbd/crm-fence-peer.sh"; Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 40 15400 15410 15420 15430 15440 15450 15460 15470 15480 15490 15500 15510 15520 15530 15540 15550 15560 15570 15580 15590 15600 15610 15620 15630 15640 15650 15660 15670 15680 15690 15700 15710 15720 15730 15740 15750 15760 15770 15780 15790
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 4"; after-resync-target "/usr/lib/drbd/unsnapshot-resync-target-lvm.sh; /usr/lib/drbd/crm-unfence-peer.sh"; } startup { #wfc# wfc-timeout 10; #wfc# degr-wfc-timeout 10; #wfc# outdated-wfc-timeout 10; } disk { on-io-error detach; fencing resource-only; al-extents 6433; c-plan-ahead 20; c-delay-target 100; c-fill-target 0; c-max-rate 100M; c-min-rate 1M; } net { protocol C; max-buffers 128k; sndbuf-size 0; rcvbuf-size 0; cram-hmac-alg sha1; shared-secret "password"; congestion-fill 100M; congestion-extents 2000; csums-alg md5; verify-alg md5; use-rle yes; } } EOF sudo cp -a /etc/drbd.d/global_common.conf /etc~/drbd.d/global_common.conf_$(date +%Y%m%d_%H%M%S) cat << 'EOF' | sudo tee /etc/drbd.d/r0.res resource r0 { volume 0 { device /dev/drbd0; disk /dev/vg0/lv-drbd0; Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 41 15800 15810 15820 15830 15840 15850 15860 15870 15880 15890 15900 15910 15920 15930 15940 15950 15960 15970 15980 15990 16000 16010 16020 16030 16040 16050 16060 16070 16080 16090 16100 16110 16120 16130 16140 16150 16160 16170 16180 16190
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s ○ a,s
○ a,s
○ a,s ○ a,s
meta-disk internal; } on iscsitgt01a.example.com { address 192.168.1.2:7788; } on iscsitgt01s.example.com { address 192.168.1.3:7788; } } EOF DRBD リソースを初期化します。 sudo drbdadm create-md r0 initializing activity log NOT initializing bitmap Writing meta data... New drbd meta data block successfully created. targetcli から exit する際に自動的に設定を save する挙動を無効化します。 sudo targetcli set global auto_save_on_exit=false Warning: Could not load preferences file /root/.targetcli/prefs.bin. Parameter auto_save_on_exit is now 'false'. targetcli から target を追加する際に自動的に portal が作成される挙動を無効化します。 sudo targetcli set global auto_add_default_portal=false Parameter auto_add_default_portal is now 'false'. targetcli コマンドのデフォルト設定を確認します。 sudo targetcli get global GLOBAL CONFIG GROUP =================== auto_add_default_portal=false ---------------------------If true, adds a portal listening on all IPs to new targets. auto_add_mapped_luns=true Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 42 16200 16210 16220 16230 16240 16250 16260 16270 16280 16290 16300 16310 16320 16330 16340 16350 16360 16370 16380 16390 16400 16410 16420 16430 16440 16450 16460 16470 16480 16490 16500 16510 16520 16530 16540 16550 16560 16570 16580 16590
------------------------If true, automatically create node ACLs mapped LUNs after creating a new target LUN or a new node ACL auto_cd_after_create=false -------------------------If true, changes current path to newly created objects. auto_enable_tpgt=true --------------------If true, automatically enables TPGTs upon creation. auto_save_on_exit=false ---------------------If true, saves configuration on exit. color_command=cyan -----------------Color to use for command completions. color_default=none -----------------Default text display color. color_keyword=cyan -----------------Color to use for keyword completions. color_mode=true --------------Console color display mode. color_parameter=magenta ----------------------Color to use for parameter completions. color_path=magenta -----------------Color to use for path completions export_backstore_name_as_model=true Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 43 16600 16610 16620 16630 16640 16650 16660 16670 16680 16690 16700 16710 16720 16730 16740 16750 16760 16770 16780 16790 16800 16810 16820 16830 16840 16850 16860 16870 16880 16890 16900 16910 16920 16930 16940 16950 ○ 16960 16970 a,s 16980 a,s 16990 a,s
----------------------------------If true, the backstore name is used for the scsi inquiry model name. logfile=/root/.targetcli/log.txt -------------------------------Logfile to use. loglevel_console=info --------------------Log level for messages going to the console. loglevel_file=debug ------------------Log level for messages going to the log file. prompt_length=30 ---------------Max length of the shell prompt path, 0 for infinite. tree_max_depth=0 ---------------Maximum depth of displayed node tree. tree_round_nodes=true --------------------Tree node display style. tree_show_root=true ------------------Whether or not to display tree root. tree_status_mode=true --------------------Whether or not to display status in tree. LIO のリソース・エージェントを作成します。 cat << 'EOF_LIO' | sudo tee /usr/lib/ocf/resource.d/heartbeat/LIO #!/bin/bash # Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 44 17000 17010 17020 17030 17040 17050 17060 17070 17080 17090 17100 17110 17120 17130 17140 17150 17160 17170 17180 17190 17200 17210 17220 17230 17240 17250 17260 17270 17280 17290 17300 17310 17320 17330 17340 17350 17360 17370 17380 17390
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
# # # # # # # # # # # # # # # # # # # # # # # # # #
LIO OCF RA. manages iSCSI target LIO. (c) 2009-2010 Florian Haas, Dejan Muhamedagic, and Linux-HA contributors modified by Katsuaki Hamada ([email protected]), 23 Oct 2016 This program is free software; you can redistribute it and/or modify it under the terms of version 2 of the GNU General Public License as published by the Free Software Foundation. This program is distributed in the hope that it would be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Further, this software is distributed without any warranty that it is free of the rightful claim of any third person regarding infringement or the like. Any license provided herein, whether implied or otherwise, applies only to this software file. Patent licenses, if any, provided herein do not apply to combinations of this program with other software, or any other product whatsoever. You should have received a copy of the GNU General Public License along with this program; if not, write the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.
####################################################################### # Initialization: : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat} . ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs # Lockfile, used for selecting a target ID LOCKFILE=${HA_RSCTMP}/target.lock ####################################################################### meta_data() { cat < Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 45 17400 17410 17420 17430 17440 17450 17460 17470 17480 17490 17500 17510 17520 17530 17540 17550 17560 17570 17580 17590 17600 17610 17620 17630 17640 17650 17660 17670 17680 17690 17700 17710 17720 17730 17740 17750 17760 17770 17780 17790
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
0.9 Manages iSCSI target LIO. An iSCSI target is a collection of SCSI Logical Units (LUs) exported via a daemon that speaks the iSCSI protocol. iSCSI target export agent The target iSCSI Qualified Name (IQN). Should follow the conventional iqn.yyyy-mm.[:identifier] syntax. iSCSI target IQN END } ####################################################################### LIO_usage() { cat <
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 46 17800 17810 17820 17830 17840 17850 17860 17870 17880 17890 17900 17910 17920 17930 17940 17950 17960 17970 17980 17990 18000 18010 18020 18030 18040 18050 18060 18070 18080 18090 18100 18110 18120 18130 18140 18150 18160 18170 18180 18190
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
} LIO_start() { LIO_monitor [ $? = $OCF_SUCCESS ] && return $OCF_SUCCESS /usr/bin/targetctl restore LIO_monitor } LIO_stop() { LIO_monitor [ $? -eq $OCF_NOT_RUNNING ] || /usr/bin/targetctl clear return $OCF_SUCCESS } LIO_monitor() { # if we have no configfs entry for the target, it's definitely stopped [ -d /sys/kernel/config/target/iscsi/${OCF_RESKEY_iqn} ] || return $OCF_NOT_RUNNING # if the target is there, but its TPG is not enabled, then we also consider it stopped [ $(cat /sys/kernel/config/target/iscsi/${OCF_RESKEY_iqn}/tpgt_1/enable) -eq 1 ] || return $OCF_NOT_RUNNING return $OCF_SUCCESS } LIO_validate() { if ! ocf_is_probe; then # Do we have all required binaries? check_binary targetctl fi return $OCF_SUCCESS } case $1 in meta-data) meta_data; exit $OCF_SUCCESS;; usage|help) LIO_usage; exit $OCF_SUCCESS;; esac # Everything except usage and meta-data must pass the validate test LIO_validate Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 47 18200 18210 18220 18230 18240 18250 18260 18270 18280 18290 18300 18310 18320 18330 18340 18350 18360 18370 18380 18390 18400 18410 18420 18430 18440 18450 18460 18470 18480 18490 18500 18510 18520 18530 18540 18550 18560 18570 18580 18590
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s ○ a,s a,s
case $__OCF_ACTION in start) LIO_start;; stop) LIO_stop;; monitor|status) LIO_monitor;; reload) ocf_log err "Reloading..."; LIO_start;; validate-all) ;; *) LIO_usage; exit $OCF_ERR_UNIMPLEMENTED;; esac rc=$? ocf_log debug "${OCF_RESOURCE_INSTANCE} $__OCF_ACTION : $rc" exit $rc EOF_LIO sudo chmod 755 /usr/lib/ocf/resource.d/heartbeat/LIO VIP に関するリソース・エージェント (IPaddr2) の名前を変更します。 sed -e 's/IPaddr2/VIP/g' /usr/lib/ocf/resource.d/heartbeat/IPaddr2 | sudo tee /usr/lib/ocf/resource.d/heartbeat/VIP > /dev/null
sudo chmod 755 /usr/lib/ocf/resource.d/heartbeat/VIP ※ LVM, LIO, VIP リソース・エージェント名文字数を統一し、「sudo pcs status」等の実行結果を見やすくします。 ※ 問い合わせを行う時は、IPaddr2 を上記のコマンドで変更している点を伝えないと話が通じないものと思われます。
○
pcs の利用環境を整えます。
a,s
echo 'password' | sudo passwd --stdin hacluster Changing password for user hacluster. passwd: all authentication tokens updated successfully.
a,s a,s
sudo cp -a /etc/shadow sudo cp -a /etc/shadow-
a,s
sudo usermod -a -G haclient admin
a,s
id admin uid=1000(admin) gid=1000(admin) groups=1000(admin),10(wheel),189(haclient)
a,s
sudo usermod -a -G haclient monitor
a,s
id monitor
/etc~/shadow_$(date +%Y%m%d_%H%M%S) /etc~/shadow-_$(date +%Y%m%d_%H%M%S)
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 48 18600 18610 18620 18630 18640 18650 18660 18670 18680 18690 18700 18710 18720 18730 18740 18750 18760 18770 18780 18790 18800 18810 18820 18830 18840 18850 18860 18870 18880 18890 18900 18910 18920 18930 18940 18950 18960 18970 18980 18990
uid=1001(monitor) gid=1001(monitor) groups=1001(monitor),189(haclient) a,s a,s
sudo cp -a /etc/group sudo cp -a /etc/group-
a,s a,s
sudo systemctl start pcsd sudo systemctl enable pcsd Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service.
○ a,s a,s a,s a,s
/etc~/group_$(date +%Y%m%d_%H%M%S) /etc~/group-_$(date +%Y%m%d_%H%M%S)
Corosync のサービス設定を変更します。 sed -e 's/^#Restart=on-failure.*$/Restart=on-failure/' \ -e 's/^#RestartSec=.*$/RestartSec=70/' \ -e 's%^#ExecStartPre=/sbin/modprobe softdog soft_margin=.*$%ExecStartPre=/sbin/modprobe softdog soft_margin=6%' \
/usr/lib/systemd/system/corosync.service | sudo tee /etc/systemd/system/corosync.service [Unit] Description=Corosync Cluster Engine ConditionKernelCommandLine=!nocluster Requires=network-online.target After=network-online.target [Service] ExecStart=/usr/share/corosync/corosync start ExecStop=/usr/share/corosync/corosync stop Type=forking # The following config is for corosync with enabled watchdog service. # # When corosync watchdog service is being enabled and using with # pacemaker.service, and if you want to exert the watchdog when a # corosync process is terminated abnormally, # uncomment the line of the following Restart= and RestartSec=. Restart=on-failure # Specify a period longer than soft_margin as RestartSec. RestartSec=70 # rewrite according to environment. ExecStartPre=/sbin/modprobe softdog soft_margin=6 [Install] WantedBy=multi-user.target Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 49 19000 19010 19020 19030 19040 a,s 19050 19060 19070 19080 19090 19100 19110 19120 19130 19140 19150 19160 ○ 19170 19180 a,s 19190 a,s 19200 19210 19220 19230 19240 19250 19260 19270 19280 19290 19300 19310 19320 19330 19340 19350 19360 19370 19380 19390
※ カーネル内のソフトウェア watchdog 機能を有効化します。 ※ Corosync プロセス障害検知時間を6秒以内とします。 cat /etc/sysconfig/corosync # Corosync init script configuration file # COROSYNC_INIT_TIMEOUT specifies number of seconds to wait for corosync # initialization (default is one minute). COROSYNC_INIT_TIMEOUT=60 # COROSYNC_OPTIONS specifies options passed to corosync command # (default is no options). # See "man corosync" for detailed descriptions of the options. COROSYNC_OPTIONS="" Pacemaker のサービス設定を変更します。 sed -e "s%^# ExecStopPost=/bin/sh -c 'pidof crmd || killall -TERM corosync'$%ExecStopPost=/bin/sh -c 'pidof crmd || killall -TERM corosync'%" \
/usr/lib/systemd/system/pacemaker.service | sudo tee /etc/systemd/system/pacemaker.service [Unit] Description=Pacemaker High Availability Cluster Manager After=dbus.service After=basic.target After=syslog.service After=network.target After=corosync.service Requires=dbus.service Requires=basic.target Requires=corosync.service # if you use crm_mon, uncomment the line below. # Wants=crm_mon.service [Install] WantedBy=multi-user.target [Service] Type=simple Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 50 19400 19410 19420 19430 19440 19450 19460 19470 19480 19490 19500 19510 19520 19530 19540 19550 19560 19570 19580 19590 19600 19610 19620 19630 19640 19650 19660 19670 19680 19690 19700 19710 19720 19730 19740 19750 19760 19770 19780 19790
KillMode=process NotifyAccess=main EnvironmentFile=-/etc/sysconfig/pacemaker EnvironmentFile=-/etc/sysconfig/sbd SuccessExitStatus=100 ExecStart=/usr/sbin/pacemakerd -f # If pacemakerd doesn't stop, its probably waiting on a cluster # resource. Sending -KILL will just get the node fenced SendSIGKILL=no # # # # # # # #
If we ever hit the StartLimitInterval/StartLimitBurst limit and the admin wants to stop the cluster while pacemakerd is not running, it might be a good idea to enable the ExecStopPost directive below. Although the node will likely end up being fenced as a result so its not on by default ExecStopPost=/usr/bin/killall -TERM crmd attrd fenced cib pengine lrmd
# If you want Corosync to stop whenever Pacemaker is stopped, # uncomment the next line too: # ExecStopPost=/bin/sh -c 'pidof crmd || killall -TERM corosync' # Uncomment this for older versions of systemd that didn't support # TimeoutStopSec # TimeoutSec=30min # Pacemaker can only exit after all managed services have shut down # A HA database could conceivably take even longer than this TimeoutStopSec=30min TimeoutStartSec=60s # Restart options include: no, on-success, on-failure, on-abort or always Restart=on-failure # crm_perror() writes directly to stderr, so ignore it here # to avoid double-logging with the wrong format Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 51 19800 19810 19820 19830 19840 19850 19860 19870 a,s 19880 a,s 19890 a,s 19900 19910 19920 19930 19940 19950 19960 19970 19980 19990 20000 20010 20020 20030 20040 20050 20060 20070 20080 20090 20100 20110 20120 20130 20140 20150 20160 20170 20180 20190
StandardError=null # if you use crm_mon, uncomment the line below. # ExecStopPost=/bin/sh -c 'systemctl status crm_mon >/dev/null && systemctl stop crm_mon' ※ Pacemaker サービス停止時に Corosync サービスを停止します。 sudo sed -i -e 's/^# PCMK_fail_fast=.*$/PCMK_fail_fast=yes/' /etc/sysconfig/pacemaker sudo cp -a /etc/sysconfig/pacemaker /etc~/sysconfig/pacemaker_$(date +%Y%m%d_%H%M%S) cat /etc/sysconfig/pacemaker # For non-systemd based systems, prefix export to each enabled line # Turn on special handling for CMAN clusters in the init script # Without this, fenced (and by inference, cman) cannot reliably be made to shut down # PCMK_STACK=cman #==#==# Variables that control logging # # # #
Enable debug logging globally or per-subsystem Multiple subsystems may me listed separated by commas eg. PCMK_debug=crmd,pengine PCMK_debug=yes|no|crmd|pengine|cib|stonith-ng|attrd|pacemakerd
# # # #
Send INFO (and higher) messages to the named log file Additional messages may also appear here depending on any configured debug and trace settings By default Pacemaker will inherit the logfile specified in corosync.conf PCMK_logfile=/var/log/pacemaker.log
# # # #
Specify an alternate syslog target for NOTICE (and higher) messages Use 'none' to disable - not recommended The default value is 'daemon' PCMK_logfacility=none|daemon|user|local0|local1|local2|local3|local4|local5|local6|local7
# # # #
Send all messages up-to-and-including the configured priority to syslog A value of 'info' will be far too verbose for most installations and 'debug' is almost certain to send you blind The default value is 'notice' PCMK_logpriority=emerg|alert|crit|error|warning|notice|info|debug
# Log all messages from a comma-separated list of functions # PCMK_trace_functions=function1,function2,function3 Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 52 20200 20210 20220 20230 20240 20250 20260 20270 20280 20290 20300 20310 20320 20330 20340 20350 20360 20370 20380 20390 20400 20410 20420 20430 20440 20450 20460 20470 20480 20490 20500 20510 20520 20530 20540 20550 20560 20570 20580 20590
# Log all messages from a comma-separated list of files (no path) # Supports wildcards eg. PCMK_trace_files=prefix*.c # PCMK_trace_files=file.c,other.h # Log all messages matching comma-separated list of formats # PCMK_trace_formats="Sent delete %d" # Log all messages from a comma-separated list of tags # PCMK_trace_tags=tag1,tag2 # Dump the blackbox whenever the message at function and line is printed # eg. PCMK_trace_blackbox=te_graph_trigger:223,unpack_clone:81 # PCMK_trace_blackbox=fn:line,fn2:line2,... # # # # # # # # # #
Enable blackbox logging globally or per-subsystem The blackbox contains a rolling buffer of all logs (including info+debug+trace) and is written after a crash, assertion failure and/or when SIGTRAP is received The blackbox recorder can also be enabled for Pacemaker daemons at runtime by sending SIGUSR1 (or SIGTRAP), and disabled by sending SIGUSR2 Multiple subsystems may me listed separated by commas eg. PCMK_blackbox=crmd,pengine PCMK_blackbox=yes|no|crmd|pengine|cib|stonith-ng|attrd|pacemakerd
#==#==# Advanced use only # Enable this for compatibility with older corosync (prior to 2.0) # based clusters which used the nodes uname as its uuid also # PCMK_uname_is_uuid=no # Specify an alternate location for RNG schemas and XSL transforms # Mostly only useful for developer testing # PCMK_schema_directory=/some/path # Enable this for rebooting this machine at the time of process (subsystem) failure PCMK_fail_fast=yes #==#==# Pacemaker Remote Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 53 20600 20610 20620 20630 20640 20650 20660 20670 20680 20690 20700 20710 20720 20730 20740 20750 20760 20770 20780 20790 20800 20810 20820 20830 20840 20850 20860 20870 20880 20890 20900 20910 20920 20930 20940 20950 20960 20970 20980 20990
# # # # #
Use a custom directory for finding the authkey. PCMK_authkey_location=/etc/pacemaker/authkey Specify a custom port for Pacemaker Remote connections PCMK_remote_port=3121
#==#==# IPC # Force use of a particular class of IPC connection # PCMK_ipc_type=shared-mem|socket|posix|sysv # Specify an IPC buffer size in bytes # Useful when connecting to really big clusters that exceed the default 20k buffer # PCMK_ipc_buffer=20480 #==#==# Profiling and memory leak testing # # # # # # # # #
Variables for running child daemons under valgrind and/or checking for memory problems G_SLICE=always-malloc MALLOC_PERTURB_=221 # or 0 MALLOC_CHECK_=3 # or 0,1,2 PCMK_valgrind_enabled=yes PCMK_valgrind_enabled=cib,crmd PCMK_callgrind_enabled=yes PCMK_callgrind_enabled=cib,crmd VALGRIND_OPTS="--leak-check=full --trace-children=no --num-callers=25 --log-file=/var/lib/pacemaker/valgrind-%p --suppressions=/usr/share/pacemaker/tests/valgrind-pcmk.suppressions --gen-suppressions=all" ※ Pacemaker の内部プロセス障害をノード障害として扱うようにします。
○ a,s a,s a,s a,s a,s a,s a,s a,s
Pacemaker のリソース設定スクリプトを作成します。 cat << 'EOF' #!/bin/bash pcs property pcs property pcs property pcs property pcs property pcs property
| sudo tee /etc/ha.d/crm.sh set set set set set set
batch-limit=30 cluster-delay=60 cluster-recheck-interval=15min crmd-finalization-timeout=30min crmd-integration-timeout=3min crmd-transition-delay=0s Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 54 21000 21010 21020 21030 21040 21050 21060 21070 21080 21090 21100 21110 21120 21130 21140 21150 21160 21170 21180 21190 21200 21210 21220 21230 21240 21250 21260 21270 21280 21290 21300 21310 21320 21330 21340 21350 21360 21370 21380 21390
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs pcs
property property property property property property property property property property property property property property property property property property property property property property property property property property property property property property property
set set set set set set set set set set set set set set set set set set set set set set set set set set set set set set set
dc-deadtime=20s default-action-timeout=20 election-timeout=2min enable-acl=true --force enable-startup-probes=true is-managed-default=true load-threshold=80% maintenance-mode=false migration-limit=-1 no-quorum-policy=ignore node-action-limit=0 node-health-green=0 node-health-red=-INFINITY node-health-strategy=none node-health-yellow=0 notification-agent=/dev/null pe-error-series-max=100 pe-input-series-max=100 pe-warn-series-max=100 placement-strategy=default remove-after-stop=false shutdown-escalation=20min start-failure-is-fatal=true startup-fencing=true stonith-action=reboot stonith-enabled=false stonith-timeout=60 stop-all-resources=false stop-orphan-actions=true stop-orphan-resources=true symmetric-cluster=true
pcs resource defaults resource-stickiness=200 migration-threshold=2 pcs acl role create write-access description="Full access" write xpath /cib pcs acl role create read-only description="Read access to cluster" read xpath /cib pcs acl user create admin write-access pcs acl user create monitor read-only
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 55 21400 21410 21420 21430 21440 21450 21460 21470 21480 21490 21500 21510 21520 21530 21540 21550 21560 21570 21580 21590 21600 21610 21620 21630 21640 21650 21660 21670 21680 21690 21700 21710 21720 21730 21740 21750 21760 21770 21780 21790
a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s a,s
pcs resource create p_drbd_r0 ocf:linbit:drbd \ params drbd_resource=r0 \ op start timeout=240 \ op monitor interval=10 role=Master timeout=20 \ op monitor interval=20 role=Slave timeout=20 \ op notify timeout=90 \ op stop timeout=100 \ op promote timeout=90 \ op demote timeout=90 pcs resource master ms_drbd_r0 p_drbd_r0 \ meta master-max=1 master-node-max=1 clone-max=2 \ clone-node-max=1 notify=true target-role=Started \ is-managed=true pcs resource create p_lvm ocf:heartbeat:LVM \ params volgrpname=vg1 \ op start timeout=30 \ op monitor interval=5 timeout=10 \ op stop timeout=30 pcs resource create p_lio ocf:heartbeat:LIO \ params iqn=iqn.2016-09.com.example:iscsitgt01 \ op start timeout=10 \ op monitor interval=5 timeout=5 \ op stop timeout=10 pcs resource create p_vip ocf:heartbeat:VIP \ params ip=10.110.88.59 cidr_netmask=26 nic=bond0 iflabel=1 arp_interval=200 arp_count=5 \ op start timeout=20 \ op monitor interval=5 timeout=10 \ op stop timeout=20 pcs resource group add g_tgt p_lvm p_lio p_vip pcs constraint location add lc_tgt g_tgt iscsitgt01a.example.com 100 pcs constraint colocation add g_tgt \ ms_drbd_r0 INFINITY with-rsc-role=Master
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 56 21800 21810 21820 21830 21840 21850 21860 21870 21880 21890 21900 21910 21920 21930 21940 21950 21960 21970 21980 21990 22000 22010 22020 22030 22040 22050 22060 22070 22080 22090 22100 22110 22120 22130 22140 22150 22160 22170 22180 22190
a,s a,s a,s a,s
pcs constraint order promote ms_drbd_r0 then start p_lvm EOF sudo chmod 755 /etc/ha.d/crm.sh sudo cp -a /etc{,~}/ha.d/crm.sh ※ ここからの作業は、Active 機と Stand-by 機が連動して動作していく前提の操作となります。
○ a,s
Active 機と Stand-by 機の間の疎通を確認します。 ping -c 1 -M do -s 8972 10.110.88.57 || echo Error PING 10.110.88.57 (10.110.88.57) 8972(9000) bytes of data. 8980 bytes from 10.110.88.57: icmp_seq=1 ttl=64 time=0.136 ms --- 10.110.88.57 ping statistics --1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.136/0.136/0.136/0.000 ms
a,s
traceroute -F 10.110.88.57 8972 traceroute to 10.110.88.57 (10.110.88.57), 30 hops max, 8972 byte packets 1 iscsitgt01a.example.com (10.110.88.57) 0.303 ms 0.265 ms 0.256 ms
a,s a,s
ping -c 1 -M do -s 8972 10.110.88.58 || echo Error traceroute -F 10.110.88.58 8972
a,s a,s
ping -c 1 -M do -s 8972 192.168.1.2 || echo Error traceroute -F 192.168.1.2 8972
a,s a,s
ping -c 1 -M do -s 8972 192.168.1.3 || echo Error traceroute -F 192.168.1.3 8972
○ a a a a a
Active 機で ssh 鍵を作成し、Stand-by 機にコピーします。 ssh-keygen -q -f ~/.ssh/id_rsa -N "" mv -f ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys scp -pr .ssh/ iscsitgt01s: The authenticity of host 'iscsitgt01s (10.110.88.58)' can't be established. ECDSA key fingerprint is cf:3a:39:91:fc:c9:ac:5c:4e:16:38:72:97:88:28:b2. #Are yesyou sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'iscsitgt01s,10.110.88.58' (ECDSA) to the list of known hosts. #admin@iscsitgt01s's ******** password: ******** Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 57 22200 22210 22220 22230 22240 ○ 22250 22260 a,s 22270 a,s 22280 22290 ○ 22300 22310 a,s 22320 22330 ○ 22340 22350 s 22360 22370 22380 22390 22400 22410 22420 22430 ○ 22440 22450 a 22460 22470 ○ 22480 22490 22500 22510 22520 22530 22540 22550 22560 22570 22580 22590
id_rsa authorized_keys known_hosts
100% 1679 100% 411 100% 186
1.6KB/s 0.4KB/s 0.2KB/s
00:00 00:00 00:00
Active 機と Stand-by 機で、ssh 鍵を root アカウント用にコピーします。 sudo cp -a .ssh/ /root/ sudo chown -R root:root /root/.ssh Active 機と Stand-by 機でほぼ同時に DRBD サービスを起動します。 sudo systemctl start drbd.service Stand-by 機で DRBD の状態をワッチします。 watch cat /proc/drbd Every 2.0s: cat /proc/drbd
Sat Oct 29 18:33:24 2016
version: 8.4.5 (api:1/proto:86-101) srcversion: 1AEFF755B8BD61B81A0AF27 0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:3759976 Active 機で DRBD の初期同期を開始します。 sudo drbdadm primary --force all Stand-by 機で DRBD の状態を確認します。 Every 2.0s: cat /proc/drbd
Sat Oct 29 18:33:24 2016
version: 8.4.5 (api:1/proto:86-101) srcversion: 1AEFF755B8BD61B81A0AF27 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r----ns:0 nr:0 dw:0 dr:1244 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:3758732 [>....................] sync'ed: 0.2% (3758732/3759976)K finish: 0:39:09 speed: 1,244 (1,244) want: 10,400 K/sec ※ この状態でも、Active 機側で作業を続行できます。今回は、初期同期の完了を待つことにします。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 58 22600 22610 22620 22630 22640 22650 22660 22670 22680 22690 22700 22710 22720 22730 22740 22750 22760 22770 22780 22790 22800 22810 22820 22830 22840 22850 22860 22870 22880 22890 22900 22910 22920 22930 22940 22950 22960 22970 22980 22990
Every 2.0s: cat /proc/drbd
Sat Oct 29 18:33:24 2016
version: 8.4.5 (api:1/proto:86-101) srcversion: 1AEFF755B8BD61B81A0AF27 0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r----ns:0 nr:0 dw:0 dr:3759976 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 ※ 「自機/対向機」がともに「UpToDate/UpToDate」となっているのが正常な状態です。 ○
Active 機で DRBD デバイス上に LVM を構成します。
a
sudo pvcreate /dev/drbd0 Physical volume "/dev/drbd0" successfully created
a
sudo vgcreate -s 4M vg1 /dev/drbd0 Volume group "vg1" successfully created
a
sudo lvcreate --name lv-lun0 --extents 20%VG vg1 Logical volume "lv-lun0" created.
a
sudo lvcreate --name lv-lun1 --extents 20%VG vg1 Logical volume "lv-lun1" created.
a
sudo lvcreate --name lv-lun2 --extents 20%VG vg1 Logical volume "lv-lun2" created.
a
sudo lvcreate --name lv-lun3 --extents 20%VG vg1 Logical volume "lv-lun3" created.
a
sudo pvs PV /dev/drbd0 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
a
VG Fmt Attr vg1 lvm2 a-vg0 lvm2 a-vg0 lvm2 a-vg0 lvm2 a-vg0 lvm2 a--
PSize PFree 359.97g 72.00g 100.00g 0 100.00g 0 100.00g 0 100.00g 40.00g
sudo vgs VG #PV #LV #SN Attr VSize VFree vg0 4 1 0 wz--n- 399.98g 40.00g Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 59 23000 23010 23020 23030 23040 23050 23060 23070 23080 23090 23100 23110 23120 23130 23140 23150 23160 23170 23180 23190 23200 23210 23220 23230 23240 23250 23260 23270 23280 23290 23300 23310 23320 23330 23340 23350 23360 23370 23380 23390
vg1 a
○ a
○
1
4
sudo lvs LV lv-drbd0 lv-lun0 lv-lun1 lv-lun2 lv-lun3
VG vg0 vg1 vg1 vg1 vg1
0 wz--n- 359.97g 72.00g Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert -wi-ao---- 359.98g -wi-a----- 71.99g -wi-a----- 71.99g -wi-a----- 71.99g -wi-a----- 71.99g
Active 機で、targetcli から状態を確認します。 sudo targetcli ls / o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- block .............................................. [Storage Objects: 0] | o- fileio ............................................. [Storage Objects: 0] | o- pscsi .............................................. [Storage Objects: 0] | o- ramdisk ............................................ [Storage Objects: 0] o- iscsi ........................................................ [Targets: 0] o- loopback ..................................................... [Targets: 0] Active 機で、IQN を定義します。
a
sudo targetcli /iscsi create iqn.2016-09.com.example:iscsitgt01 Created target iqn.2016-09.com.example:iscsitgt01. Created TPG 1.
a
sudo targetcli ls / o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- block .............................................. [Storage Objects: 0] | o- fileio ............................................. [Storage Objects: 0] | o- pscsi .............................................. [Storage Objects: 0] | o- ramdisk ............................................ [Storage Objects: 0] o- iscsi ........................................................ [Targets: 1] | o- iqn.2016-09.com.example:iscsitgt01 ............................ [TPGs: 1] | o- tpg1 ........................................... [no-gen-acls, no-auth] | o- acls ...................................................... [ACLs: 0] | o- luns ...................................................... [LUNs: 0] Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 60 23400 23410 23420 23430 23440 23450 23460 23470 23480 23490 23500 23510 23520 23530 23540 23550 23560 23570 23580 23590 23600 23610 23620 23630 23640 23650 23660 23670 23680 23690 23700 23710 23720 23730 23740 23750 23760 23770 23780 23790
| o- portals ................................................ [Portals: 0] o- loopback ..................................................... [Targets: 0] ○
Active 機で、ACL（アクセス許可リスト）にイニシエータ名を登録します。必要に応じて CHAP 認証情報も紐付けします。
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls create iqn.2016-09.com.example:initiator01 Created Node ACL for iqn.2016-09.com.example:initiator01
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator01 set auth userid=iscsiuser01
Parameter userid is now 'iscsiuser01'. a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator01 set auth password='password-user01'
Parameter password is now 'password-user01'. a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls create iqn.2016-09.com.example:initiator02 Created Node ACL for iqn.2016-09.com.example:initiator02
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator02 set auth userid=iscsiuser02
Parameter userid is now 'iscsiuser02'. a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator02 set auth password='password-user02'
Parameter password is now 'password-user02'. a a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls create iqn.2016-09.com.example:initiator03 Created Node ACL for iqn.2016-09.com.example:initiator03 sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator03 set auth userid=iscsiuser03
Parameter userid is now 'iscsiuser03'. a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator03 set auth password='password-user03'
Parameter password is now 'password-user03'. a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls create iqn.2016-09.com.example:initiator04 Created Node ACL for iqn.2016-09.com.example:initiator04
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator04 set auth userid=iscsiuser04
Parameter userid is now 'iscsiuser04'. Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 61 23800 23810 23820 23830 23840 23850 23860 23870 23880 23890 23900 23910 23920 23930 23940 23950 23960 23970 23980 23990 24000 24010 24020 24030 24040 24050 24060 24070 24080 24090 24100 24110 24120 24130 24140 24150 24160 24170 24180 24190
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/acls/iqn.2016-09.com.example:initiator04 set auth password='password-user04'
Parameter password is now 'password-user04'. a
○
sudo targetcli ls / o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- block .............................................. [Storage Objects: 0] | o- fileio ............................................. [Storage Objects: 0] | o- pscsi .............................................. [Storage Objects: 0] | o- ramdisk ............................................ [Storage Objects: 0] o- iscsi ........................................................ [Targets: 1] | o- iqn.2016-09.com.example:iscsitgt01 ............................ [TPGs: 1] | o- tpg1 ........................................... [no-gen-acls, no-auth] | o- acls ...................................................... [ACLs: 4] | | o- iqn.2016-09.com.example:initiator01 .............. [Mapped LUNs: 0] | | o- iqn.2016-09.com.example:initiator02 .............. [Mapped LUNs: 0] | | o- iqn.2016-09.com.example:initiator03 .............. [Mapped LUNs: 0] | | o- iqn.2016-09.com.example:initiator04 .............. [Mapped LUNs: 0] | o- luns ...................................................... [LUNs: 0] | o- portals ................................................ [Portals: 0] o- loopback ..................................................... [Targets: 0] Active 機で、バックエンド・デバイスを指定し、IQN に紐付けます。
a
sudo targetcli /backstores/block create name=lun0 dev=/dev/vg1/lv-lun0 Created block storage object lun0 using /dev/vg1/lv-lun0.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/luns create /backstores/block/lun0 Created LUN 0. Created LUN 0->0 mapping in node ACL iqn.2016-09.com.example:initiator04 Created LUN 0->0 mapping in node ACL iqn.2016-09.com.example:initiator03 Created LUN 0->0 mapping in node ACL iqn.2016-09.com.example:initiator02 Created LUN 0->0 mapping in node ACL iqn.2016-09.com.example:initiator01
a
sudo targetcli /backstores/block create name=lun1 dev=/dev/vg1/lv-lun1 Created block storage object lun1 using /dev/vg1/lv-lun1.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/luns create /backstores/block/lun1 Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 62 24200 24210 24220 24230 24240 24250 24260 24270 24280 24290 24300 24310 24320 24330 24340 24350 24360 24370 24380 24390 24400 24410 24420 24430 24440 24450 24460 24470 24480 24490 24500 24510 24520 24530 24540 24550 24560 24570 24580 24590
Created Created Created Created Created
LUN LUN LUN LUN LUN
1. 1->1 1->1 1->1 1->1
mapping mapping mapping mapping
in in in in
node node node node
ACL ACL ACL ACL
iqn.2016-09.com.example:initiator04 iqn.2016-09.com.example:initiator03 iqn.2016-09.com.example:initiator02 iqn.2016-09.com.example:initiator01
a
sudo targetcli /backstores/block create name=lun2 dev=/dev/vg1/lv-lun2 Created block storage object lun2 using /dev/vg1/lv-lun2.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/luns create /backstores/block/lun2 Created LUN 2. Created LUN 2->2 mapping in node ACL iqn.2016-09.com.example:initiator04 Created LUN 2->2 mapping in node ACL iqn.2016-09.com.example:initiator03 Created LUN 2->2 mapping in node ACL iqn.2016-09.com.example:initiator02 Created LUN 2->2 mapping in node ACL iqn.2016-09.com.example:initiator01
a
sudo targetcli /backstores/block create name=lun3 dev=/dev/vg1/lv-lun3 Created block storage object lun3 using /dev/vg1/lv-lun3.
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/luns create /backstores/block/lun3 Created LUN 3. Created LUN 3->3 mapping in node ACL iqn.2016-09.com.example:initiator04 Created LUN 3->3 mapping in node ACL iqn.2016-09.com.example:initiator03 Created LUN 3->3 mapping in node ACL iqn.2016-09.com.example:initiator02 Created LUN 3->3 mapping in node ACL iqn.2016-09.com.example:initiator01
a
sudo targetcli ls / o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- block .............................................. [Storage Objects: 4] | | o- lun0 ............... [/dev/vg1/lv-lun0 (732.0MiB) write-thru activated] | | o- lun1 ............... [/dev/vg1/lv-lun1 (732.0MiB) write-thru activated] | | o- lun2 ............... [/dev/vg1/lv-lun2 (732.0MiB) write-thru activated] | | o- lun3 ............... [/dev/vg1/lv-lun3 (732.0MiB) write-thru activated] | o- fileio ............................................. [Storage Objects: 0] | o- pscsi .............................................. [Storage Objects: 0] | o- ramdisk ............................................ [Storage Objects: 0] o- iscsi ........................................................ [Targets: 1] Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 63 24600 24610 24620 24630 24640 24650 24660 24670 24680 24690 24700 24710 24720 24730 24740 24750 24760 24770 24780 24790 24800 24810 24820 24830 24840 24850 24860 24870 24880 24890 24900 24910 24920 24930 24940 24950 24960 24970 24980 24990
| o- iqn.2016-09.com.example:iscsitgt01 ............................ [TPGs: 1] | o- tpg1 ........................................... [no-gen-acls, no-auth] | o- acls ...................................................... [ACLs: 4] | | o- iqn.2016-09.com.example:initiator01 .............. [Mapped LUNs: 4] | | | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)] | | | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)] | | | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)] | | | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)] | | o- iqn.2016-09.com.example:initiator02 .............. [Mapped LUNs: 4] | | | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)] | | | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)] | | | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)] | | | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)] | | o- iqn.2016-09.com.example:initiator03 .............. [Mapped LUNs: 4] | | | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)] | | | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)] | | | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)] | | | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)] | | o- iqn.2016-09.com.example:initiator04 .............. [Mapped LUNs: 4] | | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)] | | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)] | | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)] | | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)] | o- luns ...................................................... [LUNs: 4] | | o- lun0 .............................. [block/lun0 (/dev/vg1/lv-lun0)] | | o- lun1 .............................. [block/lun1 (/dev/vg1/lv-lun1)] | | o- lun2 .............................. [block/lun2 (/dev/vg1/lv-lun2)] | | o- lun3 .............................. [block/lun3 (/dev/vg1/lv-lun3)] | o- portals ................................................ [Portals: 0] o- loopback ..................................................... [Targets: 0] ○
Active 機で、IQN に portal を作成します。
a
sudo targetcli /iscsi/iqn.2016-09.com.example:iscsitgt01/tpg1/portals create 10.110.88.59 3260 Using default IP port 3260 Created network portal 10.110.88.59:3260.
a
sudo targetcli ls / o- / ..................................................................... [...] o- backstores .......................................................... [...] Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 64 25000 25010 25020 25030 25040 25050 25060 25070 25080 25090 25100 25110 25120 25130 25140 25150 25160 25170 25180 25190 25200 25210 25220 25230 25240 25250 25260 25270 25280 25290 25300 25310 25320 25330 25340 25350 25360 25370 25380 25390
| o- block .............................................. [Storage Objects: 4] | | o- lun0 ............... [/dev/vg1/lv-lun0 (732.0MiB) write-thru activated] | | o- lun1 ............... [/dev/vg1/lv-lun1 (732.0MiB) write-thru activated] | | o- lun2 ............... [/dev/vg1/lv-lun2 (732.0MiB) write-thru activated] | | o- lun3 ............... [/dev/vg1/lv-lun3 (732.0MiB) write-thru activated] | o- fileio ............................................. [Storage Objects: 0] | o- pscsi .............................................. [Storage Objects: 0] | o- ramdisk ............................................ [Storage Objects: 0] o- iscsi ........................................................ [Targets: 1] | o- iqn.2016-09.com.example:iscsitgt01 ............................ [TPGs: 1] | o- tpg1 ........................................... [no-gen-acls, no-auth] | o- acls ...................................................... [ACLs: 4] | | o- iqn.2016-09.com.example:initiator01 .............. [Mapped LUNs: 4] | | | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)] | | | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)] | | | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)] | | | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)] | | o- iqn.2016-09.com.example:initiator02 .............. [Mapped LUNs: 4] | | | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)] | | | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)] | | | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)] | | | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)] | | o- iqn.2016-09.com.example:initiator03 .............. [Mapped LUNs: 4] | | | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)] | | | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)] | | | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)] | | | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)] | | o- iqn.2016-09.com.example:initiator04 .............. [Mapped LUNs: 4] | | o- mapped_lun0 .............................. [lun0 block/lun0 (rw)] | | o- mapped_lun1 .............................. [lun1 block/lun1 (rw)] | | o- mapped_lun2 .............................. [lun2 block/lun2 (rw)] | | o- mapped_lun3 .............................. [lun3 block/lun3 (rw)] | o- luns ...................................................... [LUNs: 4] | | o- lun0 .............................. [block/lun0 (/dev/vg1/lv-lun0)] | | o- lun1 .............................. [block/lun1 (/dev/vg1/lv-lun1)] | | o- lun2 .............................. [block/lun2 (/dev/vg1/lv-lun2)] | | o- lun3 .............................. [block/lun3 (/dev/vg1/lv-lun3)] | o- portals ................................................ [Portals: 1] | o- 10.110.88.59:3260 ............................................ [OK] o- loopback ..................................................... [Targets: 0] Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 65 25400 25410 25420 25430 25440 25450 25460 25470 25480 25490 25500 25510 25520 25530 25540 25550 25560 25570 25580 25590 25600 25610 25620 25630 25640 25650 25660 25670 25680 25690 25700 25710 25720 25730 25740 25750 25760 25770 25780 25790
a ○
ss -ant | grep LISTEN.\*3260 LISTEN 0 256 10.110.88.59:3260
*:*
Active 機で、設定ファイルに設定を保存し、Stand-by 機にコピーします。
a
sudo targetcli saveconfig Last 10 configs saved in /etc/target/backup. Configuration saved to /etc/target/saveconfig.json
a
sudo scp -p /etc/target/saveconfig.json iscsitgt01s:/etc/target/saveconfig.json saveconfig.json 100% 11KB 11.0KB/s 00:00
○
Active 機で、設定ファイルを確認します。
a
sudo cat /etc/target/saveconfig.json { "fabric_modules": [], "storage_objects": [ { "attributes": { "block_size": 512, "emulate_3pc": 1, "emulate_caw": 1, "emulate_dpo": 0, "emulate_fua_read": 0, "emulate_fua_write": 1, "emulate_model_alias": 1, "emulate_rest_reord": 0, "emulate_tas": 1, "emulate_tpu": 0, "emulate_tpws": 0, "emulate_ua_intlck_ctrl": 0, "emulate_write_cache": 0, "enforce_pr_isids": 1, "force_pr_aptpl": 0, "is_nonrot": 0, "max_unmap_block_desc_count": 0, "max_unmap_lba_count": 0, "max_write_same_len": 65535, Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 66 25800 25810 25820 25830 25840 25850 25860 25870 25880 25890 25900 25910 25920 25930 25940 25950 25960 25970 25980 25990 26000 26010 26020 26030 26040 26050 26060 26070 26080 26090 26100 26110 26120 26130 26140 26150 26160 26170 26180 26190
"optimal_sectors": 2048, "pi_prot_format": 0, "pi_prot_type": 0, "queue_depth": 128, "unmap_granularity": 0, "unmap_granularity_alignment": 0 }, "dev": "/dev/vg1/lv-lun3", "name": "lun3", "plugin": "block", "readonly": false, "write_back": false, "wwn": "33ef34a3-b07f-4876-baf8-e71fdc525e04" }, { "attributes": { "block_size": 512, "emulate_3pc": 1, "emulate_caw": 1, "emulate_dpo": 0, "emulate_fua_read": 0, "emulate_fua_write": 1, "emulate_model_alias": 1, "emulate_rest_reord": 0, "emulate_tas": 1, "emulate_tpu": 0, "emulate_tpws": 0, "emulate_ua_intlck_ctrl": 0, "emulate_write_cache": 0, "enforce_pr_isids": 1, "force_pr_aptpl": 0, "is_nonrot": 0, "max_unmap_block_desc_count": 0, "max_unmap_lba_count": 0, "max_write_same_len": 65535, "optimal_sectors": 2048, "pi_prot_format": 0, "pi_prot_type": 0, "queue_depth": 128, "unmap_granularity": 0, Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 67 26200 26210 26220 26230 26240 26250 26260 26270 26280 26290 26300 26310 26320 26330 26340 26350 26360 26370 26380 26390 26400 26410 26420 26430 26440 26450 26460 26470 26480 26490 26500 26510 26520 26530 26540 26550 26560 26570 26580 26590
"unmap_granularity_alignment": 0 }, "dev": "/dev/vg1/lv-lun2", "name": "lun2", "plugin": "block", "readonly": false, "write_back": false, "wwn": "7d7f8d13-78e6-4b0a-aacb-fee219526219" }, { "attributes": { "block_size": 512, "emulate_3pc": 1, "emulate_caw": 1, "emulate_dpo": 0, "emulate_fua_read": 0, "emulate_fua_write": 1, "emulate_model_alias": 1, "emulate_rest_reord": 0, "emulate_tas": 1, "emulate_tpu": 0, "emulate_tpws": 0, "emulate_ua_intlck_ctrl": 0, "emulate_write_cache": 0, "enforce_pr_isids": 1, "force_pr_aptpl": 0, "is_nonrot": 0, "max_unmap_block_desc_count": 0, "max_unmap_lba_count": 0, "max_write_same_len": 65535, "optimal_sectors": 2048, "pi_prot_format": 0, "pi_prot_type": 0, "queue_depth": 128, "unmap_granularity": 0, "unmap_granularity_alignment": 0 }, "dev": "/dev/vg1/lv-lun1", "name": "lun1", "plugin": "block", Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 68 26600 26610 26620 26630 26640 26650 26660 26670 26680 26690 26700 26710 26720 26730 26740 26750 26760 26770 26780 26790 26800 26810 26820 26830 26840 26850 26860 26870 26880 26890 26900 26910 26920 26930 26940 26950 26960 26970 26980 26990
"readonly": false, "write_back": false, "wwn": "43ab697b-6d1e-4d18-b7df-7dd1dd272965" }, { "attributes": { "block_size": 512, "emulate_3pc": 1, "emulate_caw": 1, "emulate_dpo": 0, "emulate_fua_read": 0, "emulate_fua_write": 1, "emulate_model_alias": 1, "emulate_rest_reord": 0, "emulate_tas": 1, "emulate_tpu": 0, "emulate_tpws": 0, "emulate_ua_intlck_ctrl": 0, "emulate_write_cache": 0, "enforce_pr_isids": 1, "force_pr_aptpl": 0, "is_nonrot": 0, "max_unmap_block_desc_count": 0, "max_unmap_lba_count": 0, "max_write_same_len": 65535, "optimal_sectors": 2048, "pi_prot_format": 0, "pi_prot_type": 0, "queue_depth": 128, "unmap_granularity": 0, "unmap_granularity_alignment": 0 }, "dev": "/dev/vg1/lv-lun0", "name": "lun0", "plugin": "block", "readonly": false, "write_back": false, "wwn": "37a29689-9f83-4f15-9624-471980ea8f45" } ], Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 69 27000 27010 27020 27030 27040 27050 27060 27070 27080 27090 27100 27110 27120 27130 27140 27150 27160 27170 27180 27190 27200 27210 27220 27230 27240 27250 27260 27270 27280 27290 27300 27310 27320 27330 27340 27350 27360 27370 27380 27390
"targets": [ { "fabric": "iscsi", "tpgs": [ { "attributes": { "authentication": 0, "cache_dynamic_acls": 0, "default_cmdsn_depth": 64, "default_erl": 0, "demo_mode_discovery": 1, "demo_mode_write_protect": 1, "fabric_prot_type": 0, "generate_node_acls": 0, "login_timeout": 15, "netif_timeout": 2, "prod_mode_write_protect": 0, "t10_pi": 0 }, "enable": true, "luns": [ { "index": 3, "storage_object": "/backstores/block/lun3" }, { "index": 2, "storage_object": "/backstores/block/lun2" }, { "index": 1, "storage_object": "/backstores/block/lun1" }, { "index": 0, "storage_object": "/backstores/block/lun0" } ], "node_acls": [ { Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 70 27400 27410 27420 27430 27440 27450 27460 27470 27480 27490 27500 27510 27520 27530 27540 27550 27560 27570 27580 27590 27600 27610 27620 27630 27640 27650 27660 27670 27680 27690 27700 27710 27720 27730 27740 27750 27760 27770 27780 27790
"attributes": { "dataout_timeout": 3, "dataout_timeout_retries": 5, "default_erl": 0, "nopin_response_timeout": 30, "nopin_timeout": 15, "random_datain_pdu_offsets": 0, "random_datain_seq_offsets": 0, "random_r2t_offsets": 0 }, "chap_password": "password-user04", "chap_userid": "iscsiuser04", "mapped_luns": [ { "index": 3, "tpg_lun": 3, "write_protect": false }, { "index": 2, "tpg_lun": 2, "write_protect": false }, { "index": 1, "tpg_lun": 1, "write_protect": false }, { "index": 0, "tpg_lun": 0, "write_protect": false } ], "node_wwn": "iqn.2016-09.com.example:initiator04" }, { "attributes": { "dataout_timeout": 3, "dataout_timeout_retries": 5, Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 71 27800 27810 27820 27830 27840 27850 27860 27870 27880 27890 27900 27910 27920 27930 27940 27950 27960 27970 27980 27990 28000 28010 28020 28030 28040 28050 28060 28070 28080 28090 28100 28110 28120 28130 28140 28150 28160 28170 28180 28190
"default_erl": 0, "nopin_response_timeout": 30, "nopin_timeout": 15, "random_datain_pdu_offsets": 0, "random_datain_seq_offsets": 0, "random_r2t_offsets": 0 }, "chap_password": "password-user03", "chap_userid": "iscsiuser03", "mapped_luns": [ { "index": 3, "tpg_lun": 3, "write_protect": false }, { "index": 2, "tpg_lun": 2, "write_protect": false }, { "index": 1, "tpg_lun": 1, "write_protect": false }, { "index": 0, "tpg_lun": 0, "write_protect": false } ], "node_wwn": "iqn.2016-09.com.example:initiator03" }, { "attributes": { "dataout_timeout": 3, "dataout_timeout_retries": 5, "default_erl": 0, "nopin_response_timeout": 30, "nopin_timeout": 15, Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 72 28200 28210 28220 28230 28240 28250 28260 28270 28280 28290 28300 28310 28320 28330 28340 28350 28360 28370 28380 28390 28400 28410 28420 28430 28440 28450 28460 28470 28480 28490 28500 28510 28520 28530 28540 28550 28560 28570 28580 28590
"random_datain_pdu_offsets": 0, "random_datain_seq_offsets": 0, "random_r2t_offsets": 0 }, "chap_password": "password-user02", "chap_userid": "iscsiuser02", "mapped_luns": [ { "index": 3, "tpg_lun": 3, "write_protect": false }, { "index": 2, "tpg_lun": 2, "write_protect": false }, { "index": 1, "tpg_lun": 1, "write_protect": false }, { "index": 0, "tpg_lun": 0, "write_protect": false } ], "node_wwn": "iqn.2016-09.com.example:initiator02" }, { "attributes": { "dataout_timeout": 3, "dataout_timeout_retries": 5, "default_erl": 0, "nopin_response_timeout": 30, "nopin_timeout": 15, "random_datain_pdu_offsets": 0, "random_datain_seq_offsets": 0, "random_r2t_offsets": 0 Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 73 28600 28610 28620 28630 28640 28650 28660 28670 28680 28690 28700 28710 28720 28730 28740 28750 28760 28770 28780 28790 28800 28810 28820 28830 28840 28850 28860 28870 28880 28890 28900 28910 28920 28930 28940 28950 28960 28970 28980 28990
}, "chap_password": "password-user01", "chap_userid": "iscsiuser01", "mapped_luns": [ { "index": 3, "tpg_lun": 3, "write_protect": false }, { "index": 2, "tpg_lun": 2, "write_protect": false }, { "index": 1, "tpg_lun": 1, "write_protect": false }, { "index": 0, "tpg_lun": 0, "write_protect": false } ], "node_wwn": "iqn.2016-09.com.example:initiator01" } ], "parameters": { "AuthMethod": "CHAP,None", "DataDigest": "CRC32C,None", "DataPDUInOrder": "Yes", "DataSequenceInOrder": "Yes", "DefaultTime2Retain": "20", "DefaultTime2Wait": "2", "ErrorRecoveryLevel": "0", "FirstBurstLength": "65536", "HeaderDigest": "CRC32C,None", "IFMarkInt": "2048~65535", "IFMarker": "No", Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 74 29000 29010 29020 29030 29040 29050 29060 29070 29080 29090 29100 29110 29120 29130 29140 29150 29160 29170 29180 29190 29200 29210 29220 29230 29240 29250 29260 29270 29280 29290 29300 29310 29320 29330 29340 29350 29360 29370 29380 29390
"ImmediateData": "Yes", "InitialR2T": "Yes", "MaxBurstLength": "262144", "MaxConnections": "1", "MaxOutstandingR2T": "1", "MaxRecvDataSegmentLength": "8192", "MaxXmitDataSegmentLength": "262144", "OFMarkInt": "2048~65535", "OFMarker": "No", "TargetAlias": "LIO Target" }, "portals": [ { "ip_address": "10.110.88.59", "iser": false, "port": 3260 } ], "tag": 1 } ], "wwn": "iqn.2016-09.com.example:iscsitgt01" } ] } ○
Active 機で、LIO の設定をクリアします。
a
sudo targetctl clear
a
sudo targetcli ls / o- / ..................................................................... [...] o- backstores .......................................................... [...] | o- block .............................................. [Storage Objects: 0] | o- fileio ............................................. [Storage Objects: 0] | o- pscsi .............................................. [Storage Objects: 0] | o- ramdisk ............................................ [Storage Objects: 0] o- iscsi ........................................................ [Targets: 0] o- loopback ..................................................... [Targets: 0]
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 75 29400 ○ 29410 29420 a 29430 29440 29450 a 29460 29470 29480 29490 29500 29510 29520 29530 ○ 29540 29550 a 29560 29570 ○ 29580 29590 29600 29610 29620 29630 s 29640 29650 29660 ○ 29670 29680 a,s 29690 29700 a,s 29710 29720 29730 ○ 29740 29750 a 29760 a 29770 29780 29790
Active 機で、DRBD 上の LVM ボリュームグループを非活性化します。 sudo vgchange -a n vg1 0 logical volume(s) in volume group "vg1" now active sudo lvs LV lv-drbd0 lv-lun0 lv-lun1 lv-lun2 lv-lun3
VG vg0 vg1 vg1 vg1 vg1
Attr -wi-ao----wi-------wi-------wi-------wi-------
LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert 359.98g 71.99g 71.99g 71.99g 71.99g
Active 機で、DRBD リソースを secondary 化（デモート）します。 sudo drbdadm secondary all Stand-by 機で、DRBD の状態を確認し、「Ctrl + C」を押下してワッチを停止します。 #Every [Ctrl 2.0s: + C]cat /proc/drbd
Sat Oct 29 18:33:24 2016
#version: [Ctrl +8.4.5 C] (api:1/proto:86-101) srcversion: 1AEFF755B8BD61B81A0AF27 # 0: [Ctrl cs:Connected + C] ro:Secondary/Secondary ds:UpToDate/UpToDate C r----ns:0 nr:100 dw:100 dr:3759976 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 Active 機と Stand-by 機で、drbd.service を停止します。 sudo systemctl stop drbd.service cat /proc/drbd cat: /proc/drbd: No such file or directory Active 機で、Corosync の認証を設定し、起動します。 sudo pcs cluster auth iscsitgt01a.example.com iscsitgt01s.example.com 10.110.88.57 10.110.88.58 \ 192.168.1.2 192.168.1.3 -u hacluster -p 'password' --force iscsitgt01s.example.com: Authorized iscsitgt01a.example.com: Authorized 10.110.88.58: Authorized Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 76 29800 29810 29820 29830 29840 29850 29860 29870 29880 29890 29900 29910 29920 29930 29940 29950 29960 29970 29980 29990 30000 30010 30020 30030 30040 30050 30060 30070 30080 30090 30100 30110 30120 30130 30140 30150 30160 30170 30180 30190
192.168.1.2: Authorized 192.168.1.3: Authorized 10.110.88.57: Authorized a
sudo cat /var/lib/pcsd/tokens { "format_version": 2, "data_version": 4, "tokens": { "10.110.88.57": "77189e9e-3be0-40ce-b81e-3e5e6525e885", "10.110.88.58": "9e3f4ae9-b15e-49c9-b6ee-eb8c1b91783a", "192.168.1.2": "53da862f-ad22-445b-8887-add50d385736", "192.168.1.3": "4f78d9c6-34a4-4486-8ba1-e69f0d4e1257", "iscsitgt01a.example.com": "002cd1c0-2ab2-4a4c-a1a7-4bf14b61b822", "iscsitgt01s.example.com": "1a9981a9-04e6-461b-b904-c5df8b4c9815" } }
a a
sudo pcs cluster setup --name iscsitgt01 10.110.88.57,192.168.1.2 10.110.88.58,192.168.1.3 \ --transport=udp --rrpmode=passive -u hacluster -p 'password' --force Shutting down pacemaker/corosync services... Redirecting to /bin/systemctl stop pacemaker.service Redirecting to /bin/systemctl stop corosync.service Killing any remaining services... Removing all cluster configuration files... 10.110.88.57: Succeeded 10.110.88.58: Succeeded Synchronizing pcsd certificates on nodes 10.110.88.57, 10.110.88.58... 10.110.88.57: Success 10.110.88.58: Success Restaring pcsd on the nodes in order to reload the certificates... 10.110.88.57: Success 10.110.88.58: Success
a
cat /etc/corosync/corosync.conf totem { version: 2 secauth: off cluster_name: iscsitgt01 Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 77 30200 30210 30220 30230 30240 30250 30260 30270 30280 30290 30300 30310 30320 30330 30340 30350 30360 30370 30380 30390 30400 30410 30420 30430 30440 30450 30460 30470 30480 30490 30500 30510 30520 30530 30540 30550 30560 30570 30580 30590
transport: udp rrp_mode: passive } nodelist { node { ring0_addr: 10.110.88.57 ring1_addr: 192.168.1.2 nodeid: 1 } node { ring0_addr: 10.110.88.58 ring1_addr: 192.168.1.3 nodeid: 2 } } quorum { provider: corosync_votequorum two_node: 1 } logging { to_logfile: yes logfile: /var/log/cluster/corosync.log to_syslog: yes } a
sudo pcs cluster start --all 10.110.88.57: Starting Cluster... 10.110.88.58: Starting Cluster...
a
sudo pcs status corosync Membership information ---------------------Nodeid Votes Name 1 1 10.110.88.57 (local) 2 1 10.110.88.58
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 78 30600 30610 30620 30630 30640 30650 30660 30670 30680 30690 30700 30710 30720 30730 30740 30750 30760 30770 30780 30790 30800 30810 30820 30830 30840 30850 30860 30870 30880 30890 30900 30910 30920 30930 30940 30950 30960 30970 30980 30990
a
sudo pcs status Cluster name: iscsitgt01 WARNING: no stonith devices and stonith-enabled is not false WARNING: corosync and pacemaker node names do not match (IPs used in setup?) Last updated: Tue Oct 29 18:33:24 2016
Last change: Tue Oct 29 18:33:24 2016 by hacluster via crmd on iscsitgt01a.example.com
Stack: corosync Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum 2 nodes and 0 resources configured Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ] No resources Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled ※ 「Current DC」が表示されるまで、何回か実行します。20秒以上かかるものと思われます。 ※ 「Current DC」については、どちらが選ばれていてもあまり意味のある情報ではないので気にしないでください。 ※ 「WARNING」について、前者は後で対応します。後者は pcs のバグ（RRP mode 未対応）なので無視してください。 ○
Active 機と Stand-by 機で、Corosync の状態とプロセスを確認します。
a
sudo corosync-cfgtool -s Printing ring status. Local node ID 1 RING ID 0 id = 10.110.88.57 status = ring 0 active with no faults RING ID 1 id = 192.168.1.2 status = ring 1 active with no faults
s
sudo corosync-cfgtool -s Printing ring status. Local node ID 2 RING ID 0 id = 10.110.88.58 status = ring 0 active with no faults Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 79 31000 31010 31020 31030 31040 a,s 31050 31060 31070 31080 31090 31100 31110 31120 31130 31140 ○ 31150 31160 a 31170 31180 31190 ○ 31200 31210 a 31220 31230 31240 31250 31260 31270 31280 31290 31300 31310 31320 31330 31340 31350 31360 31370 31380 31390
RING ID 1 id status ps -ef | root root haclust+ root root haclust+ haclust+ haclust+
= 192.168.1.3 = ring 1 active with no faults
egrep '[c]orosync|[p]acemaker' 38502 1 0 19:49 ? 38517 1 0 19:49 ? 38518 38517 0 19:49 ? 38519 38517 0 19:49 ? 38520 38517 0 19:49 ? 38521 38517 0 19:49 ? 38522 38517 0 19:49 ? 38523 38517 0 19:49 ?
00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00 00:00:00
corosync /usr/sbin/pacemakerd -f /usr/libexec/pacemaker/cib /usr/libexec/pacemaker/stonithd /usr/libexec/pacemaker/lrmd /usr/libexec/pacemaker/attrd /usr/libexec/pacemaker/pengine /usr/libexec/pacemaker/crmd
Active 機で、クラスタにリソースを登録します。 sudo /etc/ha.d/crm.sh Adding ms_drbd_r0 p_lvm (kind: Mandatory) (Options: first-action=promote then-action=start) Active 機で、状態を確認します。 sudo pcs status Cluster name: iscsitgt01 WARNING: corosync and pacemaker node names do not match (IPs used in setup?) Last updated: Sat Oct 29 18:33:24 2016
Last change: Sat Oct 29 18:33:24 2016 by root via cibadmin on iscsitgt01s.example.com
Stack: corosync Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum 2 nodes and 5 resources configured Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ] Full list of resources: Master/Slave Set: ms_drbd_r0 [p_drbd_r0] p_drbd_r0 (ocf::linbit:drbd): FAILED iscsitgt01a.example.com (unmanaged) p_drbd_r0 (ocf::linbit:drbd): FAILED iscsitgt01s.example.com (unmanaged) Resource Group: g_tgt p_lvm (ocf::heartbeat:LVM): Stopped p_lio (ocf::heartbeat:LIO): Stopped p_vip (ocf::heartbeat:VIP): Stopped Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 80 31400 31410 31420 31430 31440 31450 31460 31470 31480 31490 31500 31510 31520 31530 31540 31550 31560 31570 31580 31590 31600 31610 31620 31630 31640 31650 31660 31670 31680 31690 31700 31710 31720 31730 31740 31750 31760 31770 31780 31790
Failed Actions: * p_drbd_r0_stop_0 on iscsitgt01a.example.com 'not configured' (6): call=6, status=complete, exitreason='none', last-rc-change='Sat Oct 29 18:33:24 2016', queued=0ms, exec=24ms * p_lvm_start_0 on iscsitgt01a.example.com 'unknown error' (1): call=11, status=complete, exitreason='Volume group [vg1] does not exist or contains error! Volume group "vg1" not found', last-rc-change='Sat Oct 29 18:33:24 2016', queued=0ms, exec=101ms * p_drbd_r0_stop_0 on iscsitgt01s.example.com 'not configured' (6): call=6, status=complete, exitreason='none', last-rc-change='Sat Oct 29 18:33:24 2016', queued=0ms, exec=23ms * p_lvm_start_0 on iscsitgt01s.example.com 'unknown error' (1): call=15, status=complete, exitreason='Volume group [vg1] does not exist or contains error! Volume group "vg1" not found', last-rc-change='Tue Oct 29 18:33:24 2016', queued=0ms, exec=190ms Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled ○ a ○ a
Active 機で、リソースのエラー情報をクリアします。 sudo pcs resource cleanup Waiting for 1 replies from the CRMd. OK Active 機で、状態を確認します。 sudo pcs status Cluster name: iscsitgt01 WARNING: corosync and pacemaker node names do not match (IPs used in setup?) Last updated: Sat Oct 29 18:33:24 2016
Last change: Sat Oct 29 18:33:24 2016 by hacluster via crmd on iscsitgt01a.example.com
Stack: corosync Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum 2 nodes and 5 resources configured Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ] Full list of resources: Master/Slave Set: ms_drbd_r0 [p_drbd_r0] Masters: [ iscsitgt01a.example.com ] Slaves: [ iscsitgt01s.example.com ] Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 81 31800 31810 31820 31830 31840 31850 31860 31870 31880 31890 31900 ○ 31910 31920 a,s 31930 31940 31950 31960 31970 31980 31990 32000 32010 32020 32030 32040 32050 32060 32070 32080 32090 32100 32110 32120 32130 32140 32150 32160 32170 32180 32190
Resource Group: g_tgt p_lvm (ocf::heartbeat:LVM): p_lio (ocf::heartbeat:LIO): p_vip (ocf::heartbeat:VIP):
Started iscsitgt01a.example.com Started iscsitgt01a.example.com Started iscsitgt01a.example.com
Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled Active 機と Stand-by 機で、設定情報を保存します。 sudo pcs config | sudo tee /etc/ha.d/crm.conf Cluster Name: iscsitgt01 Corosync Nodes: 10.110.88.57 10.110.88.58 Pacemaker Nodes: iscsitgt01a.example.com iscsitgt01s.example.com Resources: Master: ms_drbd_r0 Meta Attrs: master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true target-role=Started is-managed=true Resource: p_drbd_r0 (class=ocf provider=linbit type=drbd) Attributes: drbd_resource=r0 Operations: start interval=0s timeout=240 (p_drbd_r0-start-interval-0s) monitor interval=10 role=Master timeout=20 (p_drbd_r0-monitor-interval-10) monitor interval=20 role=Slave timeout=20 (p_drbd_r0-monitor-interval-20) notify interval=0s timeout=90 (p_drbd_r0-notify-interval-0s) stop interval=0s timeout=100 (p_drbd_r0-stop-interval-0s) promote interval=0s timeout=90 (p_drbd_r0-promote-interval-0s) demote interval=0s timeout=90 (p_drbd_r0-demote-interval-0s) Group: g_tgt Resource: p_lvm (class=ocf provider=heartbeat type=LVM) Attributes: volgrpname=vg1 Operations: start interval=0s timeout=30 (p_lvm-start-interval-0s) monitor interval=10 timeout=30 (p_lvm-monitor-interval-10) stop interval=0s timeout=30 (p_lvm-stop-interval-0s) Resource: p_lio (class=ocf provider=heartbeat type=LIO) Attributes: iqn=iqn.2016-09.com.example:iscsitgt01 Operations: start interval=0s timeout=10 (p_lio-start-interval-0s) Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 82 32200 32210 32220 32230 32240 32250 32260 32270 32280 32290 32300 32310 32320 32330 32340 32350 32360 32370 32380 32390 32400 32410 32420 32430 32440 32450 32460 32470 32480 32490 32500 32510 32520 32530 32540 32550 32560 32570 32580 32590
monitor interval=10 timeout=10 (p_lio-monitor-interval-10) stop interval=0s timeout=10 (p_lio-stop-interval-0s) Resource: p_vip (class=ocf provider=heartbeat type=VIP) Attributes: ip=10.110.88.59 cidr_netmask=26 nic=bond0 iflabel=1 arp_interval=200 arp_count=5 Operations: start interval=0s timeout=20 (p_vip-start-interval-0s) monitor interval=10 timeout=20 (p_vip-monitor-interval-10) stop interval=0s timeout=20 (p_vip-stop-interval-0s) Stonith Devices: Fencing Levels: Location Constraints: Resource: g_tgt Enabled on: iscsitgt01a.example.com (score:100) (id:lc_tgt) Ordering Constraints: promote ms_drbd_r0 then start p_lvm (kind:Mandatory) (id:order-ms_drbd_r0-p_lvm-mandatory) Colocation Constraints: g_tgt with ms_drbd_r0 (score:INFINITY) (with-rsc-role:Master) (id:colocation-g_tgt-ms_drbd_r0-INFINITY) Resources Defaults: resource-stickiness: 200 migration-threshold: 2 Operations Defaults: No defaults set Cluster Properties: batch-limit: 30 cluster-delay: 60 cluster-infrastructure: corosync cluster-name: iscsitgt01 cluster-recheck-interval: 15min crmd-finalization-timeout: 30min crmd-integration-timeout: 3min crmd-transition-delay: 0s dc-deadtime: 20s dc-version: 1.1.13-10.el7-44eb2dd default-action-timeout: 20 election-timeout: 2min enable-acl: true enable-startup-probes: true Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 83 32600 32610 32620 32630 32640 32650 32660 32670 32680 32690 32700 32710 32720 32730 32740 32750 32760 32770 32780 32790 32800 32810 32820 32830 32840 32850 32860 32870 32880 a,s 32890 32900 ○ 32910 32920 a 32930 32940 32950 32960 32970 32980 ○ 32990
have-watchdog: false is-managed-default: true load-threshold: 80% maintenance-mode: false migration-limit: -1 no-quorum-policy: ignore node-action-limit: 0 node-health-green: 0 node-health-red: -INFINITY node-health-strategy: none node-health-yellow: 0 notification-agent: /dev/null pe-error-series-max: 100 pe-input-series-max: 100 pe-warn-series-max: 100 placement-strategy: default remove-after-stop: false shutdown-escalation: 20min start-failure-is-fatal: true startup-fencing: true stonith-action: reboot stonith-enabled: false stonith-timeout: 60 stop-all-resources: false stop-orphan-actions: true stop-orphan-resources: true symmetric-cluster: true sudo cp -a /etc{,~}/ha.d/crm.conf Active 機で、スイッチオーバ（手動フェイルオーバ）させます。 sudo pcs resource move g_tgt Warning: Creating location constraint cli-ban-g_tgt-on-iscsitgt01a.example.com with a score of -INFINITY for resource g_tgt on node iscsitgt01a.example.com. This will prevent g_tgt from running on iscsitgt01a.example.com until the constraint is removed. This will be the case even if iscsitgt01a.example.com is the last node in the cluster. Active 機で、状態を確認します。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 84 33000 33010 33020 33030 33040 33050 33060 33070 33080 33090 33100 33110 33120 33130 33140 33150 33160 33170 33180 33190 33200 33210 33220 33230 33240 33250 33260 33270 33280 33290 33300 33310 33320 33330 33340 33350 33360 33370 33380 33390
a
sudo pcs status Cluster name: iscsitgt01 WARNING: corosync and pacemaker node names do not match (IPs used in setup?) Last updated: Sat Oct 29 18:33:24 2016
Last change: Sat Oct 29 18:33:24 2016 by root via crm_resource on iscsitgt01a.example.com
Stack: corosync Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum 2 nodes and 5 resources configured Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ] Full list of resources: Master/Slave Set: ms_drbd_r0 [p_drbd_r0] Masters: [ iscsitgt01s.example.com ] Slaves: [ iscsitgt01a.example.com ] Resource Group: g_tgt p_lvm (ocf::heartbeat:LVM): Started iscsitgt01s.example.com p_lio (ocf::heartbeat:LIO): Started iscsitgt01s.example.com p_vip (ocf::heartbeat:VIP): Started iscsitgt01s.example.com Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled ※ 「p_vip」のノードが変わるまで、何回か実行します。 ○ a
○ a a ○
Active 機で、設定変更を確認します。 diff <(grep -v last-lrm-refresh /etc/ha.d/crm.conf) <(sudo pcs config | grep -v last-lrm-refresh) 41a42 > Disabled on: iscsitgt01a.example.com (score:-INFINITY) (role: Started) (id:cli-ban-g_tgt-on-iscsitgt01a.example.com) Active 機で、設定変更を元に戻します。 sudo pcs resource clear g_tgt diff <(grep -v last-lrm-refresh /etc/ha.d/crm.conf) <(sudo pcs config | grep -v last-lrm-refresh) Active 機でリソースが起動した状態でない場合のみ、スイッチバック（フェイルバック）させます。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 85 33400 a 33410 33420 33430 33440 33450 33460 ○ 33470 33480 a 33490 33500 33510 33520 33530 33540 33550 33560 33570 33580 33590 33600 33610 33620 33630 33640 33650 33660 33670 33680 33690 33700 33710 33720 33730 ○ 33740 33750 a 33760 33770 ○ 33780 33790 a,s
sudo pcs resource move g_tgt; sleep 5; sudo pcs resource clear g_tgt Warning: Creating location constraint cli-ban-g_tgt-on-iscsitgt01s.example.com with a score of -INFINITY for resource g_tgt on node iscsitgt01s.example.com. This will prevent g_tgt from running on iscsitgt01s.example.com until the constraint is removed. This will be the case even if iscsitgt01s.example.com is the last node in the cluster. Active 機で、状態を確認します。 sudo pcs status Cluster name: iscsitgt01 WARNING: corosync and pacemaker node names do not match (IPs used in setup?) Last updated: Tue Oct 29 18:33:24 2016
Last change: Tue Oct 29 18:33:24 2016 by root via crm_resource on iscsitgt01a.example.com
Stack: corosync Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum 2 nodes and 6 resources configured Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ] Full list of resources: Master/Slave Set: ms_drbd_r0 [p_drbd_r0] Masters: [ iscsitgt01a.example.com ] Slaves: [ iscsitgt01s.example.com ] Resource Group: g_tgt p_lvm (ocf::heartbeat:LVM): Started iscsitgt01a.example.com p_lio (ocf::heartbeat:LIO): Started iscsitgt01a.example.com p_vip (ocf::heartbeat:VIP): Started iscsitgt01a.example.com Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled Active 機で、設定変更を確認します。 diff <(grep -v last-lrm-refresh /etc/ha.d/crm.conf) <(sudo pcs config | grep -v last-lrm-refresh) Active 機と Stand-by 機で、状態を記録します。 sudo pcs status | sudo tee /etc/ha.d/crm.status Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 86 33800 33810 33820 33830 33840 33850 33860 33870 33880 33890 33900 33910 33920 33930 33940 33950 33960 33970 33980 33990 34000 34010 34020 34030 34040 a,s 34050 34060 ○ 34070 34080 a 34090 34100 34110 34120 34130 34140 ○ 34150 34160 a,s 34170 34180 ○ 34190
Cluster name: iscsitgt01 WARNING: corosync and pacemaker node names do not match (IPs used in setup?) Last updated: Sat Oct 29 18:33:24 2016
Last change: Sat Oct 29 18:33:24 2016 by root via crm_resource on iscsitgt01a.example.com
Stack: corosync Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum 2 nodes and 6 resources configured Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ] Full list of resources: Master/Slave Set: ms_drbd_r0 [p_drbd_r0] Masters: [ iscsitgt01a.example.com ] Slaves: [ iscsitgt01s.example.com ] Resource Group: g_tgt p_lvm (ocf::heartbeat:LVM): Started iscsitgt01a.example.com p_lio (ocf::heartbeat:LIO): Started iscsitgt01a.example.com p_vip (ocf::heartbeat:VIP): Started iscsitgt01a.example.com Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/enabled sudo cp -a /etc{,~}/ha.d/crm.status Active 機で、クラスタを停止します。 sudo pcs cluster stop --all 10.110.88.57: Stopping Cluster 10.110.88.58: Stopping Cluster 10.110.88.58: Stopping Cluster 10.110.88.57: Stopping Cluster
(pacemaker)... (pacemaker)... (corosync)... (corosync)...
Active 機と Stand-by 機で、再起動します。 sudo reboot Active 機と Stand-by 機へ、管理者用一般ユーザにて、ssh でログインします。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 87 34200 34210 34220 34230 34240 34250 34260 34270 34280 34290 34300 34310 34320 34330 34340 34350 34360 34370 34380 34390 34400 34410 34420 34430 34440 34450 34460 34470 34480 34490 34500 34510 34520 34530 34540 34550 34560 34570 34580 34590
a a
ssh [email protected] #[email protected]'s ******** password: ********
s s
ssh [email protected] #[email protected]'s ******** password: ********
○ a
○ a
Active 機で、クラスタを起動します。 sudo pcs cluster start --all 10.110.88.57: Starting Cluster... 10.110.88.58: Starting Cluster... Active 機で、状態を確認します。 sudo pcs status Cluster name: iscsitgt01 WARNING: corosync and pacemaker node names do not match (IPs used in setup?) Last updated: Sat Oct 29 18:33:24 2016
Last change: Sat Oct 29 18:33:24 2016 by root via crm_resource on iscsitgt01a.example.com
Stack: corosync Current DC: iscsitgt01s.example.com (version 1.1.13-10.el7-44eb2dd) - partition with quorum 2 nodes and 6 resources configured Online: [ iscsitgt01a.example.com iscsitgt01s.example.com ] Full list of resources: Master/Slave Set: ms_drbd_r0 [p_drbd_r0] Masters: [ iscsitgt01a.example.com ] Slaves: [ iscsitgt01s.example.com ] Resource Group: g_tgt p_lvm (ocf::heartbeat:LVM): Started iscsitgt01a.example.com p_lio (ocf::heartbeat:LIO): Started iscsitgt01a.example.com p_vip (ocf::heartbeat:VIP): Started iscsitgt01a.example.com PCSD Status: iscsitgt01a.example.com (10.110.88.57): Online iscsitgt01s.example.com (10.110.88.58): Online Daemon Status: corosync: active/disabled Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 88 34600 34610 34620
pacemaker: active/disabled pcsd: active/enabled
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
http://dba-ha.blogspot.jp/ (メシの種 - DBAの落書き帳) LIO, DRBD, Pacemaker による冗長化 iSCSI Target 構築手順 - 89 -
当文書で紹介した構成で初期構築をご希望の方は、メール(mailto: [email protected])にてお問い合わせください。 リモートアクセスによる構築サービスを、弊社提示のヒアリング事項（当文書の太緑で記載した部分）以外のカスタマイズ無し前提で、 サーバ1台あたり10万円（税別）という超特価にてご提供中です。 1号機と2号機(Active 機と Stand-by 機)サーバの物理構成は同一構成との前提です。 カスタマイズやドキュメントの提供、個別訪問等は、別途ご相談となります。 クラスタの起動停止、スイッチオーバ(手動フェイルオーバ)以外の動作確認、結合試験、障害試験、性能試験等は、別途ご相談となります。 インストールメディアからインストールできない環境やコンソールにリモートアクセスできない環境でのインストールも別途ご相談となります。 監視設定（障害監視、リソース監視、セキュリティ監視等）については、別途ご相談となります。 UEFI 対応、ハードウェア固有のドライバや管理ソフト等のインストールについては、別途ご相談となります。 当該サーバには、消失したら困るデータは存在していない前提での作業となります。 既に動いている CentOS を置き換えるインストールの場合、ヒアリング事項を弊社で調査して提示することも可能です。 例えば、IBM Bluemix(SoftLayer) のベアメタルサーバの場合、CentOS7をあらかじめインストールした状態で弊社へお引き渡し頂ければ、 お客様の手間を省くことができます。 弊社での正式サポートは、お客様が当該サーバ用の Oracle Linux サブスクリプションを契約中か NRI OpenStandia に相談窓口をお持ちで、 代理で問い合わせを行うという前提を取らさせていただくことになります。費用はご相談ください。 サーバの調達・CE作業費用、OS 等のライセンス費用、サブスクリプション費用等はすべて別料金となります。 OS を Oracle Linux (UEK カーネル) に置き換えずに、RHEL・CentOS で構築することも可能ですが、制限事項についてご相談ください。 MySQL や PostgreSQL、Oracle の冗長構成構築サービスも鋭意開発中です。商品開発に関するリクエストがあればお知らせください。 検討させていただきます。 貴社のアプリケーション、サービス等を冗長化する共同開発も承ります。
Copyright © 2016 PC Office Co., Ltd. All Rights Reserved.
