9.4.3 Dynamically Releasing the SPARC <span class="nodiv">M12-2S</span> From the Physical Partition : Fujitsu Global

PPAR DR can be used in a building block configuration connecting two or more SPARC M12-2S units to a physical partition. By using PPAR DR, you can release the SPARC M12-2S from the physical partition without stopping Oracle Solaris on the logical domain.

This section describes the procedure for releasing the SPARC M12-2S by using PPAR DR in a building block configuration connecting two or more SPARC M12-2S units.

Note - The precautions for PPAR DR may vary depending on the versions of the XSCF firmware, Oracle Solaris, and Oracle VM for SPARC used and the version of the SRU applied. Before using PPAR DR, be sure to see the latest version of Fujitsu SPARC M12 Product Notes.

Note - To use PPAR DR, you need to configure the system with guaranteed availability according to "2.5 Dynamic Reconfiguration Conditions and Settings" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Domain Configuration Guide.

Note - The procedure described in this section assumes that -m unbind=resource is not used in the deleteboard command. In this case, before executing the deleteboard command, you need to remove resources assigned to logical domains, in addition to removing physical I/O devices, to ensure that enough BBs will be left for the number of CPUs and the memory size. For details, see "2.5.2 Considerations in System Operation for Dynamic Reconfiguration" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Domain Configuration Guide.

Table 9-11 lists the procedure for dynamically releasing the SPARC M12-2S from the physical partition.

Table 9-11 Procedure for Dynamically Releasing the SPARC M12-2S From the Physical Partition
Step	Operation Description	Execution Location	Command Used
1	Checking the physical partition configuration and the operation status of the target SPARC M12-2S	XSCF shell	XSCF>‌ showpcl -a XSCF>‌ showboards -p <‌ppar_id>‌
2	Logging in to the XSCF shell and confirming that the XSCF of the target SPARC M12-2S is in the standby state or is the slave XSCF. (*1)	XSCF shell	XSCF>‌ showhardconf
3	Checking the operation status of logical domains	primary domain	primary# ldm list-domain
4	Checking the connected virtual services in the control domain	primary domain	primary# ldm list-services
5	Checking the mounting locations of the physical LAN ports	Service domain	# dladm show-phys -L
6	Checking the virtual services and hardware resources assigned to logical domains	primary domain	primary# ldm list-bindings <‌domain-name>‌ primary# ldm list-socket
7	Checking the assignment status of I/O devices from the control domain in the physical partition to which the target SPARC M12-2S belongs	primary domain	primary# ldm list-io
8	Releasing the assignment of I/O devices of the target SPARC M12-2S from the control domain in the physical partition to which the target SPARC M12-2S belongs	primary domain	When releasing a virtual disk primary# ldm remove-vdisk <‌disk-name>‌ <‌domain-name>‌ When releasing a virtual network primary# ldm remove-vnet <‌if-name>‌ <‌domain-name>‌ primary# ldm remove-vsw <‌vswitch-name>‌ When releasing an I/O device primary# ldm remove-io <‌bus/device/vf-name>‌ <‌domain-name>‌
9	Confirming that the I/O devices are not assigned to logical domains, from the control domain in the physical partition to which the target SPARC M12-2S belongs	primary domain	primary# ldm list-io
10	Using the XSCF shell to release the target SPARC M12-2S from the physical partition	XSCF shell	For maintenance on the target SPARC M12-2S XSCF>‌ deleteboard -c disconnect <‌psb>‌ For changing the physical partition configuration XSCF>‌ deleteboard -c unassign <‌psb>‌
11	Checking the release process end status from the XSCF shell	XSCF shell	XSCF>‌ showresult
12	Checking the status of the released SPARC M12-2S from the XSCF shell	XSCF shell	XSCF>‌ showboards -va
1 If the XSCF of the SPARC M12-2S to be released is the master XSCF, switch it to the standby state with the switchscf command. For details, see the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 XSCF Reference Manual* of the XCP version used.

The following example executes the commands for releasing BB#01 in the 2BB configuration. To restore the system after maintenance on the SPARC M12-2S, write down the status checked in steps 1 to 7.

First, log in to the XSCF shell, and check the hardware configuration information for the SPARC M12-2S to be released.

Check the PSB configuration and operation status of the physical partition.
The following example assumes that the user logs in to the master XSCF, checks the PSB configuration, LSB number, and operation status of PPAR ID#00, and then checks the operation status of the PSB. Here, it is assumed that BB#01 (PSB#01-0) will be released.

XSCF> showpcl -p 0
PPAR-ID LSB PSB Status
00 Running (*1)
00 00-0 (*2)
01 01-0 (*3)

XSCF> showboards -p 0
PSB PPAR-ID(LSB) Assignment Pwr Conn Conf Test Fault

---- ------------ ----------- ---- ---- ---- ------- --------

00-0 00(00) Assigned y y y Passed Normal (*3)
01-0 00(01) Assigned y y y Passed Normal (*3)

*1 PPAR ID#00 is running.
*2 LSB#00 is defined for PSB#00-0 of PPAR ID#00, and LSB#01 is defined for PSB#01-0.
*3 PSB#00-0 and PSB#01-0 are assigned to PPAR ID#00, and both are running.

Check the status of the XSCF of the SPARC M12-2S to be released.
The following example assumes that the user logs in to the master XSCF and confirms that the XSCF of the SPARC M12-2S to be released is in the standby state or is the slave XSCF.

XSCF> showhardconf
SPARC M12-2S;
+ Serial:PZ51652009; Operator_Panel_Switch:Locked;
+ System_Power:On; System_Phase:Cabinet Power On;
Partition#0 PPAR_Status:Running;
BB#00 Status:Normal; Role:Master; Ver:4016h; Serial:PZ51652009;
+ FRU-Part-Number:CA20369-B17X 005AC/9999999 ;
+ Power_Supply_System: ;
+ Memory_Size:256 GB;
(Omitted)
BB#01 Status:Normal; Role:Standby; Ver:4016h; Serial:PZ51651016; (*1)
+ FRU-Part-Number:CA20369-B17X 005AC/9999999 ;
+ Power_Supply_System: ;
+ Memory_Size:1024 GB;
(Omitted)

*1 The XSCF of BB#01 is in the standby state.

Next, log in to Oracle Solaris on the control domain in the physical partition containing the SPARC M12-2S to be released. Then, check the operation status of logical domains and the assignment status of resources.

Check the operation status of the logical domains defined in PPAR ID#00.
The following example assumes that the user logs in to Oracle Solaris on the control domain and checks the defined logical domains and operation status.

primary# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL NORM UPTIME
primary active -n-cv- UART 32 16G 1.3% 1.3% 7d 17h 26m (*1)
guest active -n---- 5000 32 16G 0.0% 0.0% 7d 16h (*2)
ldom active -n---- 5001 32 32G 0.0% 0.0% 7d 14h 17m (*3)

Check the list of the set virtual services for the control domain.
- Virtual console service: The port numbers that the guest domain can use for primary-vcc0 for the virtual console service range from 5000 to 5100.

- Virtual switch: Physical LAN ports net0 and net4 are assigned to virtual switches primary-vsw0 and primary-vsw4, respectively.

- Virtual disk service: vdisk00, vdisk01, vdisk10, vdisk11, and vol_iso are assigned to primary-vds0 for the virtual disk service.

primary# ldm list-services
VCC
NAME LDOM PORT-RANGE
primary-vcc0 primary 5000-5100
VSW
NAME LDOM MAC NET-DEV ID DEVICE LINKPROP
DEFAULT-VLAN-ID PVID VID MTU MODE INTER-VNET-LINK
primary-vsw0 primary 00:14:4f:fb:e1:a8 net0 0 switch@0
1 1 1500 on
primary-vsw4 primary 00:14:4f:f8:42:2f net4 1 switch@1
1 1 1500 on
VDS
NAME LDOM VOLUME OPTIONS MPGROUP DEVICE
primary-vds0 primary vdisk00 /dev/zvol/dsk/rpool/export/ovm/vdisk00
vdisk01 /dev/zvol/dsk/rpool/export/ovm/vdisk01
vol_iso ro /export/ovm/sol11u3_iso/sol-11_3-text-sparc.iso
vdisk10 /dev/zvol/dsk/rpool/export/ovm/vdisk10
vdisk11 /dev/zvol/dsk/rpool/export/ovm/vdisk11

Check the mounting locations of the physical LAN ports.
This example checks in advance that Oracle Solaris on the control domain does not use net4 to net7, to release BB#01 using PPAR DR.

Physical LAN Port	Mounting Location
net0 to net3	BB#00 on-board LAN
net4 to net7	BB#01 on-board LAN

primary# dladm show-phys -L
LINK DEVICE LOC
net0 ixgbe0 /BB0/CMUL
net1 ixgbe1 /BB0/CMUL
net2 ixgbe2 /BB0/CMUU
net3 ixgbe3 /BB0/CMUU
net4 ixgbe4 /BB1/CMUL
net5 ixgbe5 /BB1/CMUL
net6 ixgbe6 /BB1/CMUL
net7 ixgbe7 /BB1/CMUL
(Omitted)

Check the resources assigned to the logical domain guest and the logical domain ldom.
The virtual network devices vnet0 and vnet1 are assigned to the logical domain guest. From the results of steps 4 to 6, you can also see that vnet0 uses the physical LAN port net0 of BB#00 and vnet1 uses the physical LAN port net4 of BB#01. In this example, the assignment of vnet1 is released from the logical domain guest beforehand in order to use the physical partition dynamic reconfiguration (PPAR DR) function to release BB#01.
From the assigned I/O resources, you can see that the logical domain ldom is the root domain configured by the root complexes of BB#00 and BB#01. In this example, the assignment of PCIE10 (pci@8a00) is released from the logical domain ldom beforehand in order to use the physical partition dynamic reconfiguration (PPAR DR) function to release BB#01.

primary# ldm list-bindings guest
(Omitted)
NETWORK
NAME SERVICE ID DEVICE MAC MODE PVID VID MTU
MAXBW LINKPROP
vnet0 primary-vsw0@primary 0 network@0 00:14:4f:f9:58:4e 1 1500
PEER MAC MODE PVID VID MTU MAXBW LINKPROP
primary-vsw0@primary 00:14:4f:fb:e1:a8 1 1500
NAME SERVICE ID DEVICE MAC MODE PVID VID MTU
MAXBW LINKPROP
vnet1 primary-vsw4@primary 1 network@1 00:14:4f:fb:4d:fe 1 1500
PEER MAC MODE PVID VID MTU MAXBW LINKPROP
primary-vsw4@primary 00:14:4f:f8:42:2f 1 1500
DISK
NAME VOLUME TOUT ID DEVICE SERVER MPGROUP
vdisk00 vdisk00@primary-vds0 0 disk@0 primary
(Omitted)

primary# ldm list-bindings ldom
(Omitted)
IO
DEVICE PSEUDONYM OPTIONS
pci@8600 PCIE6
pci@8a00 PCIE10
pci@8600/pci@4/pci@0/pci@9 /BB0/PCI5
pci@8600/pci@4/pci@0/pci@11 /BB0/PCI6
pci@8a00/pci@4/pci@0/pci@0 /BB1/PCI3
pci@8a00/pci@4/pci@0/pci@8 /BB1/PCI4
pci@8a00/pci@4/pci@0/pci@9 /BB1/PCI5
pci@8a00/pci@4/pci@0/pci@11 /BB1/PCI6
DISK
NAME VOLUME TOUT ID DEVICE SERVER MPGROUP
vdisk10 vdisk10@primary-vds0 0 disk@0 primary
vdisk11 vdisk11@primary-vds0 1 disk@1 primary
vol_iso vol_iso@primary-vds0 2 disk@2 primary
(Omitted)

Check whether CPU socket constraints are used.
The CPU socket constraint settings are not restored even if addboard -m bind=resource is executed. Therefore, you need to set the constraints again by using the ldm command after executing addboard.

# ldm list-socket
CONSTRAINTS
DOMAIN SOCKET_ID STATE
primary 0, 2 active
...

CPU socket constraints are used if the domain information is output below CONSTRAINTS.

Check the assignment status of I/O devices.
Check I/O devices and assigned logical domain names. In this example, the I/O devices of BB#01 were not assigned to the logical domain beforehand to use the physical partition dynamic reconfiguration (PPAR DR) function to release BB#01.

# ldm list-io
NAME TYPE BUS DOMAIN STATUS

---- ---- --- ------ ------

PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1 primary IOV
PCIE2 BUS PCIE2 primary IOV
PCIE3 BUS PCIE3 primary IOV
PCIE4 BUS PCIE4 primary IOV
PCIE5 BUS PCIE5 primary IOV
PCIE6 BUS PCIE6 ldom IOV
PCIE7 BUS PCIE7 primary IOV
PCIE8 BUS PCIE8 primary IOV
PCIE9 BUS PCIE9 primary IOV
PCIE10 BUS PCIE10 ldom IOV
PCIE11 BUS PCIE11 primary IOV
/BB0/CMUL/NET0 PCIE PCIE0 primary OCC
/BB0/CMUL/SASHBA PCIE PCIE0 primary OCC
/BB0/PCI0 PCIE PCIE1 primary EMP
/BB0/PCI3 PCIE PCIE2 primary EMP
/BB0/PCI4 PCIE PCIE2 primary EMP
/BB0/PCI7 PCIE PCIE3 primary EMP
/BB0/PCI8 PCIE PCIE3 primary EMP
/BB0/CMUL/NET2 PCIE PCIE4 primary OCC
/BB0/PCI1 PCIE PCIE5 primary EMP
/BB0/PCI2 PCIE PCIE5 primary EMP
/BB0/PCI5 PCIE PCIE6 ldom OCC
/BB0/PCI6 PCIE PCIE6 ldom EMP
/BB0/PCI9 PCIE PCIE7 primary EMP
/BB0/PCI10 PCIE PCIE7 primary EMP
/BB1/CMUL/NET0 PCIE PCIE8 primary OCC
/BB1/CMUL/NET2 PCIE PCIE8 primary OCC
/BB1/CMUL/SASHBA PCIE PCIE8 primary OCC
/BB1/PCI0 PCIE PCIE9 primary EMP
/BB1/PCI1 PCIE PCIE9 primary EMP
/BB1/PCI2 PCIE PCIE9 primary EMP
/BB1/PCI3 PCIE PCIE10 ldom EMP
/BB1/PCI4 PCIE PCIE10 ldom EMP
/BB1/PCI5 PCIE PCIE10 ldom OCC
/BB1/PCI6 PCIE PCIE10 ldom EMP
/BB1/PCI7 PCIE PCIE11 primary EMP
/BB1/PCI8 PCIE PCIE11 primary EMP
/BB1/PCI9 PCIE PCIE11 primary EMP
/BB1/PCI10 PCIE PCIE11 primary EMP
/BB0/CMUL/NET0/IOVNET.PF0 PF PCIE0 primary
/BB0/CMUL/NET0/IOVNET.PF1 PF PCIE0 primary
/BB0/CMUL/NET2/IOVNET.PF0 PF PCIE4 primary
/BB0/CMUL/NET2/IOVNET.PF1 PF PCIE4 primary
/BB1/CMUL/NET0/IOVNET.PF0 PF PCIE8 primary
/BB1/CMUL/NET0/IOVNET.PF1 PF PCIE8 primary
/BB1/CMUL/NET2/IOVNET.PF0 PF PCIE8 primary
/BB1/CMUL/NET2/IOVNET.PF1 PF PCIE8 primary
#

Release the I/O devices assigned to the logical domain, from Oracle Solaris on the control domain.
If a mirror volume is configured between multiple SPARC M12-2S units, release the HDD/SSD mounted in the target SPARC M12-2S from the mirror beforehand.

a. Release the virtual network device vnet1 assigned to the logical domain guest.

# ldm remove-vnet vnet1 guest
# ldm list-bindings guest
(Omitted)
NETWORK
NAME SERVICE ID DEVICE MAC MODE PVID VID MTU
MAXBW LINKPROP
vnet0 primary-vsw0@primary 0 network@0 00:14:4f:f9:58:4e 1 1500
PEER MAC MODE PVID VID MTU MAXBW LINKPROP
primary-vsw0@primary 00:14:4f:fb:e1:a8 1 1500
(Omitted)

b. Delete primary-vsw4, which is set for the physical LAN port net4.

# ldm remove-vsw primary-vsw4
# ldm list-bindings primary
(Omitted)
VSW
NAME MAC NET-DEV ID DEVICE LINKPROP DEFAULT-VLAN-ID PVID VID
MTU MODE INTER-VNET-LINK
primary-vsw0 00:14:4f:fb:e1:a8 net0 0 switch@0 1 1
1500 on
PEER MAC PVID VID MTU MAXBW LINKPROP INTERVNETLINK
vnet0@guest 00:14:4f:f9:58:4e 1 1500
(Omitted)

c. Release the assignment of PCIE10 (pci@8a00) from the logical domain ldom.

# ldm remove-io PCIE10 ldom
# ldm list-bindings ldom
(Omitted)
IO
DEVICE PSEUDONYM OPTIONS
pci@8600 PCIE6
pci@8600/pci@4/pci@0/pci@9 /BB0/PCI5
pci@8600/pci@4/pci@0/pci@11 /BB0/PCI6
(Omitted)

d. Release the I/O devices of BB#01.

# ldm remove-io PCIE8 primary
# ldm remove-io PCIE9 primary
# ldm remove-io PCIE11 primary

Confirm that none of the I/O devices of BB#01 is assigned.

# ldm list-io
NAME TYPE BUS DOMAIN STATUS

---- ---- --- ------ ------

(Omitted)
PCIE8 BUS PCIE8
PCIE9 BUS PCIE9
PCIE10 BUS PCIE10
PCIE11 BUS PCIE11
(Omitted)
/BB1/CMUL/NET0 PCIE PCIE8 UNK
/BB1/CMUL/NET2 PCIE PCIE8 UNK
/BB1/CMUL/SASHBA PCIE PCIE8 UNK
/BB1/PCI0 PCIE PCIE9 UNK
/BB1/PCI1 PCIE PCIE9 UNK
/BB1/PCI2 PCIE PCIE9 UNK
/BB1/PCI3 PCIE PCIE10 UNK
/BB1/PCI4 PCIE PCIE10 UNK
/BB1/PCI5 PCIE PCIE10 UNK
/BB1/PCI6 PCIE PCIE10 UNK
/BB1/PCI7 PCIE PCIE11 UNK
/BB1/PCI8 PCIE PCIE11 UNK
/BB1/PCI9 PCIE PCIE11 UNK
/BB1/PCI10 PCIE PCIE11 UNK

Log in to the XSCF shell, and release the SPARC M12-2S from the physical partition.

Release the SPARC M12-2S of BB-ID#01 from PPAR ID#01.

XSCF> deleteboard -y -c unassign 01-0
PSB#01-0 will be unconfigured from PPAR immediately. Continue?[y|n] :y
Start unconfigure preparation of PSB. [1200sec]
0end
Unconfigure preparation of PSB has completed.
Start unconfiguring PSB from PPAR. [7200sec]
0|
Processing of the incoming DR request by the LDoms Manager is pendingIncoming DR request is being processed by the LDoms ManagerDR sequence started (sequence#=2, message#=3)
Suspending the guest domain (ldom2)
Suspending the guest domain (ldom1)
Suspending the guest domain (ldom0)
\
Resumed the guest domain (ldom0)
Resumed the guest domain (ldom1)
|
Resumed the guest domain (ldom2)
../
DR sequence finished (sequence#=2, message#=3)
..
Processing of the incoming DR request by the LDoms Manager is pendingIncoming DR request is being processed by the LDoms ManagerDR sequence started (sequence#=3, message#=2)
DR sequence finished (sequence#=3, message#=2)
.end
Unconfigured PSB from PPAR.
PSB power off sequence started. [1200sec]
0..... 30..... 60..... 90.....120.....150...end
Operation has completed.

Note - You may need to assign command options depending on how the DR is used and the logical domain system configuration.
For details on command options, see "3.1.16 Deleting a System Board From a Building Block Configuration" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Domain Configuration Guide.

Check the end code of the deleteboard command.
If the end code is not 0, the deleteboard command ended abnormally. Eliminate the cause of the failure, and then execute the deleteboard command again.

In the following example, "0" is returned as the end status, so you can see that the execution of the deleteboard command has completed correctly.

XSCF> showresult
0

Note - For information on the action to take if the deleteboard command ends abnormally, see "3.1.16 Deleting a System Board From a Building Block Configuration" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Domain Configuration Guide.

Confirm that the SPARC M12-2S of BB-ID#01 has been powered off and that the system board is in the pool state.
Since the SPARC M12-2S has been released with the disconnect option, incorporation into PPAR ID#00 is reserved.

XSCF> showboards -va
PSB R PPAR-ID(LSB) Assignment Pwr Conn Conf Test Fault

---- - ------------ ----------- ---- ---- ---- ------- --------

00-0 00(00) Assigned y y y Passed Normal
01-0 * 00(01) Assigned n n n Passed Normal
XSCF>

Contents

9.4.3 Dynamically Releasing the SPARC M12-2S From the Physical Partition