9.4.3 Dynamically Releasing the SPARC <span class="nodiv">M12-2S</span> From the Physical Partition : Fujitsu Global

PPAR DR can be used in a building block configuration connecting two or more SPARC M12-2S units to a physical partition. By using PPAR DR, you can release the SPARC M12-2S from the physical partition without stopping Oracle Solaris on the logical domain.

This section describes the procedure for releasing the SPARC M12-2S by using PPAR DR in a building block configuration connecting two or more SPARC M12-2S units.

Note - The precautions for PPAR DR may vary depending on the versions of the XSCF firmware, Oracle Solaris, and Oracle VM for SPARC used and the version of the SRU applied. Before using PPAR DR, be sure to see the latest version of Fujitsu SPARC M12 Product Notes.

Note - To use PPAR DR, you need to configure the system with guaranteed availability according to "2.5 Dynamic Reconfiguration Conditions and Settings" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Domain Configuration Guide.

Note - The procedure described in this section assumes that -m unbind=resource is not used in the deleteboard command. In this case, before executing the deleteboard command, you need to remove resources assigned to logical domains, in addition to removing physical I/O devices, to ensure that enough BBs will be left for the number of CPUs and the memory size. For details, see "2.5.2 Considerations in System Operation for Dynamic Reconfiguration" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Domain Configuration Guide.

Table 9-11 lists the procedure for dynamically releasing the SPARC M12-2S from the physical partition.

Table 9-11 Procedure for Dynamically Releasing the SPARC M12-2S From the Physical Partition
Step	Operation Description	Execution Location	Command Used
1	Checking the physical partition configuration and the operation status of the target SPARC M12-2S	XSCF shell	XSCF>‌ showpcl -a XSCF>‌ showboards -p <‌ppar_id>‌
2	Logging in to the XSCF shell and confirming that the XSCF of the target SPARC M12-2S is in the standby state or is the slave XSCF. (*1)	XSCF shell	XSCF>‌ showhardconf
3	Checking the operation status of logical domains	primary domain	primary# ldm list-domain
4	Checking the connected virtual services in the control domain	primary domain	primary# ldm list-services
5	Checking the mounting locations of the physical LAN ports	Service domain	# dladm show-phys -L
6	Checking the virtual services and hardware resources assigned to logical domains	primary domain	primary# ldm list-bindings <‌domain-name>‌ primary# ldm list-socket
7	Checking the assignment status of I/O devices from the control domain in the physical partition to which the target SPARC M12-2S belongs	primary domain	primary# ldm list-io
8	Releasing the assignment of I/O devices of the target SPARC M12-2S from the control domain in the physical partition to which the target SPARC M12-2S belongs	primary domain	When releasing a virtual disk primary# ldm remove-vdisk <‌disk-name>‌ <‌domain-name>‌ When releasing a virtual network primary# ldm remove-vnet <‌if-name>‌ <‌domain-name>‌ primary# ldm remove-vsw <‌vswitch-name>‌ When releasing an I/O device primary# ldm remove-io <‌bus/device/vf-name>‌ <‌domain-name>‌
9	Confirming that the I/O devices are not assigned to logical domains, from the control domain in the physical partition to which the target SPARC M12-2S belongs	primary domain	primary# ldm list-io
10	Using the XSCF shell to release the target SPARC M12-2S from the physical partition	XSCF shell	For maintenance on the target SPARC M12-2S XSCF>‌ deleteboard -c disconnect <‌psb>‌ For changing the physical partition configuration XSCF>‌ deleteboard -c unassign <‌psb>‌
11	Checking the release process end status from the XSCF shell	XSCF shell	XSCF>‌ showresult
12	Checking the status of the released SPARC M12-2S from the XSCF shell	XSCF shell	XSCF>‌ showboards -va
1 If the XSCF of the SPARC M12-2S to be released is the master XSCF, switch it to the standby state with the switchscf command. For details, see the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 XSCF Reference Manual* of the XCP version used.

The following example executes the commands for releasing BB#01 in the 2BB configuration. To restore the system after maintenance on the SPARC M12-2S, write down the status checked in steps 1 to 7.

First, log in to the XSCF shell, and check the hardware configuration information for the SPARC M12-2S to be released.

Check the PSB configuration and operation status of the physical partition.
The following example assumes that the user logs in to the master XSCF, checks the PSB configuration, LSB number, and operation status of PPAR ID#00, and then checks the operation status of the PSB. Here, it is assumed that BB#01 (PSB#01-0) will be released.

XSCF> showpcl -p 0
PPAR-ID  LSB   PSB    Status
00                    Running (*1)
         00    00-0 (*2)
         01    01-0 (*3)

XSCF> showboards -p 0
PSB  PPAR-ID(LSB) Assignment  Pwr  Conn Conf Test    Fault

---- ------------ ----------- ---- ---- ---- ------- --------

00-0 00(00) Assigned y y y Passed Normal (*3)
01-0 00(01) Assigned y y y Passed Normal (*3)

*1 PPAR ID#00 is running.
*2 LSB#00 is defined for PSB#00-0 of PPAR ID#00, and LSB#01 is defined for PSB#01-0.
*3 PSB#00-0 and PSB#01-0 are assigned to PPAR ID#00, and both are running.

Check the status of the XSCF of the SPARC M12-2S to be released.
The following example assumes that the user logs in to the master XSCF and confirms that the XSCF of the SPARC M12-2S to be released is in the standby state or is the slave XSCF.

XSCF> showhardconf
SPARC M12-2S;
    + Serial:PZ51652009; Operator_Panel_Switch:Locked;
    + System_Power:On; System_Phase:Cabinet Power On;
    Partition#0 PPAR_Status:Running;
    BB#00 Status:Normal; Role:Master; Ver:4016h; Serial:PZ51652009;
        + FRU-Part-Number:CA20369-B17X 005AC/9999999              ;
        + Power_Supply_System: ;
        + Memory_Size:256 GB;
  (Omitted)
    BB#01 Status:Normal; Role:Standby; Ver:4016h; Serial:PZ51651016; (*1)
        + FRU-Part-Number:CA20369-B17X 005AC/9999999              ;
        + Power_Supply_System: ;
        + Memory_Size:1024 GB;
  (Omitted)

*1 The XSCF of BB#01 is in the standby state.

Next, log in to Oracle Solaris on the control domain in the physical partition containing the SPARC M12-2S to be released. Then, check the operation status of logical domains and the assignment status of resources.

Check the operation status of the logical domains defined in PPAR ID#00.
The following example assumes that the user logs in to Oracle Solaris on the control domain and checks the defined logical domains and operation status.

primary# ldm list-domain
NAME             STATE      FLAGS   CONS    VCPU  MEMORY   UTIL  NORM  UPTIME
primary          active     -n-cv-  UART    32    16G      1.3%  1.3%  7d 17h 26m (*1)
guest            active     -n----  5000    32    16G      0.0%  0.0%  7d 16h (*2)
ldom             active     -n----  5001    32    32G      0.0%  0.0%  7d 14h 17m (*3)

Check the list of the set virtual services for the control domain.
- Virtual console service: The port numbers that the guest domain can use for primary-vcc0 for the virtual console service range from 5000 to 5100.

- Virtual switch: Physical LAN ports net0 and net4 are assigned to virtual switches primary-vsw0 and primary-vsw4, respectively.

- Virtual disk service: vdisk00, vdisk01, vdisk10, vdisk11, and vol_iso are assigned to primary-vds0 for the virtual disk service.

primary# ldm list-services
VCC
    NAME             LDOM             PORT-RANGE
    primary-vcc0     primary          5000-5100
VSW
  NAME             LDOM      MAC                  NET-DEV     ID        DEVICE             LINKPROP
  DEFAULT-VLAN-ID  PVID      VID                  MTU         MODE      INTER-VNET-LINK
  primary-vsw0     primary   00:14:4f:fb:e1:a8    net0        0         switch@0
  1                1                              1500        on
  primary-vsw4     primary   00:14:4f:f8:42:2f    net4        1         switch@1
  1                1                              1500        on
VDS
  NAME           LDOM     VOLUME    OPTIONS  MPGROUP  DEVICE
  primary-vds0   primary  vdisk00                     /dev/zvol/dsk/rpool/export/ovm/vdisk00
                          vdisk01                     /dev/zvol/dsk/rpool/export/ovm/vdisk01
                          vol_iso   ro                /export/ovm/sol11u3_iso/sol-11_3-text-sparc.iso
                          vdisk10                     /dev/zvol/dsk/rpool/export/ovm/vdisk10
                          vdisk11                     /dev/zvol/dsk/rpool/export/ovm/vdisk11

Check the mounting locations of the physical LAN ports.
This example checks in advance that Oracle Solaris on the control domain does not use net4 to net7, to release BB#01 using PPAR DR.

Physical LAN Port	Mounting Location
net0 to net3	BB#00 on-board LAN
net4 to net7	BB#01 on-board LAN

primary# dladm show-phys -L
LINK              DEVICE       LOC
net0              ixgbe0       /BB0/CMUL
net1              ixgbe1       /BB0/CMUL
net2              ixgbe2     /BB0/CMUU
net3              ixgbe3       /BB0/CMUU
net4              ixgbe4       /BB1/CMUL
net5              ixgbe5       /BB1/CMUL
net6              ixgbe6       /BB1/CMUL
net7              ixgbe7       /BB1/CMUL
        (Omitted)

Check the resources assigned to the logical domain guest and the logical domain ldom.
The virtual network devices vnet0 and vnet1 are assigned to the logical domain guest. From the results of steps 4 to 6, you can also see that vnet0 uses the physical LAN port net0 of BB#00 and vnet1 uses the physical LAN port net4 of BB#01. In this example, the assignment of vnet1 is released from the logical domain guest beforehand in order to use the physical partition dynamic reconfiguration (PPAR DR) function to release BB#01.
From the assigned I/O resources, you can see that the logical domain ldom is the root domain configured by the root complexes of BB#00 and BB#01. In this example, the assignment of PCIE10 (pci@8a00) is released from the logical domain ldom beforehand in order to use the physical partition dynamic reconfiguration (PPAR DR) function to release BB#01.

primary# ldm list-bindings guest
        (Omitted)
NETWORK
  NAME   SERVICE               ID  DEVICE      MAC                MODE    PVID     VID    MTU
  MAXBW  LINKPROP
  vnet0  primary-vsw0@primary  0   network@0   00:14:4f:f9:58:4e          1               1500
     PEER                   MAC                MODE  PVID   VID   MTU   MAXBW    LINKPROP
     primary-vsw0@primary   00:14:4f:fb:e1:a8       1            1500
  NAME   SERVICE               ID  DEVICE      MAC                MODE    PVID     VID    MTU
  MAXBW  LINKPROP
  vnet1  primary-vsw4@primary  1   network@1   00:14:4f:fb:4d:fe          1        1500
     PEER                   MAC                MODE  PVID   VID   MTU   MAXBW    LINKPROP
     primary-vsw4@primary   00:14:4f:f8:42:2f       1      1500
DISK
  NAME     VOLUME                  TOUT  ID  DEVICE   SERVER    MPGROUP
  vdisk00  vdisk00@primary-vds0          0   disk@0   primary
        (Omitted)

primary# ldm list-bindings ldom
        (Omitted)
IO
DEVICE                         PSEUDONYM OPTIONS
pci@8600                       PCIE6
pci@8a00                       PCIE10
pci@8600/pci@4/pci@0/pci@9     /BB0/PCI5
pci@8600/pci@4/pci@0/pci@11    /BB0/PCI6
pci@8a00/pci@4/pci@0/pci@0     /BB1/PCI3
pci@8a00/pci@4/pci@0/pci@8     /BB1/PCI4
pci@8a00/pci@4/pci@0/pci@9     /BB1/PCI5
pci@8a00/pci@4/pci@0/pci@11    /BB1/PCI6
DISK
  NAME     VOLUME                  TOUT  ID  DEVICE   SERVER    MPGROUP
  vdisk10  vdisk10@primary-vds0          0   disk@0   primary
  vdisk11  vdisk11@primary-vds0          1   disk@1   primary
  vol_iso  vol_iso@primary-vds0          2   disk@2   primary
        (Omitted)

Check whether CPU socket constraints are used.
The CPU socket constraint settings are not restored even if addboard -m bind=resource is executed. Therefore, you need to set the constraints again by using the ldm command after executing addboard.

# ldm list-socket
CONSTRAINTS
DOMAIN SOCKET_ID STATE
primary 0, 2 active

...

CPU socket constraints are used if the domain information is output below CONSTRAINTS.

Check the assignment status of I/O devices.
Check I/O devices and assigned logical domain names. In this example, the I/O devices of BB#01 were not assigned to the logical domain beforehand to use the physical partition dynamic reconfiguration (PPAR DR) function to release BB#01.

# ldm list-io
NAME TYPE BUS DOMAIN STATUS

---- ---- --- ------ ------

PCIE0                        BUS          PCIE0     primary       IOV
PCIE1                        BUS          PCIE1     primary       IOV
PCIE2                        BUS          PCIE2     primary       IOV
PCIE3                        BUS          PCIE3     primary       IOV
PCIE4                        BUS          PCIE4     primary       IOV
PCIE5                        BUS          PCIE5     primary       IOV
PCIE6                        BUS          PCIE6     ldom          IOV
PCIE7                        BUS          PCIE7     primary       IOV
PCIE8                        BUS          PCIE8     primary       IOV
PCIE9                        BUS          PCIE9     primary       IOV
PCIE10                       BUS          PCIE10    ldom          IOV
PCIE11                       BUS          PCIE11    primary       IOV
/BB0/CMUL/NET0               PCIE         PCIE0     primary       OCC
/BB0/CMUL/SASHBA             PCIE         PCIE0     primary       OCC
/BB0/PCI0                    PCIE         PCIE1     primary       EMP
/BB0/PCI3                    PCIE         PCIE2     primary       EMP
/BB0/PCI4                    PCIE         PCIE2     primary       EMP
/BB0/PCI7                    PCIE         PCIE3     primary       EMP
/BB0/PCI8                    PCIE         PCIE3     primary       EMP
/BB0/CMUL/NET2               PCIE         PCIE4     primary       OCC
/BB0/PCI1                    PCIE         PCIE5     primary       EMP
/BB0/PCI2                    PCIE         PCIE5     primary       EMP
/BB0/PCI5                    PCIE         PCIE6     ldom          OCC
/BB0/PCI6                    PCIE         PCIE6     ldom          EMP
/BB0/PCI9                    PCIE         PCIE7     primary       EMP
/BB0/PCI10                   PCIE         PCIE7     primary       EMP
/BB1/CMUL/NET0               PCIE         PCIE8     primary       OCC
/BB1/CMUL/NET2               PCIE         PCIE8     primary       OCC
/BB1/CMUL/SASHBA             PCIE         PCIE8     primary       OCC
/BB1/PCI0                    PCIE         PCIE9     primary       EMP
/BB1/PCI1                    PCIE         PCIE9     primary       EMP
/BB1/PCI2                    PCIE         PCIE9     primary       EMP
/BB1/PCI3                    PCIE         PCIE10    ldom          EMP
/BB1/PCI4                    PCIE         PCIE10    ldom          EMP
/BB1/PCI5                    PCIE         PCIE10    ldom          OCC
/BB1/PCI6                    PCIE         PCIE10    ldom          EMP
/BB1/PCI7                    PCIE         PCIE11    primary       EMP
/BB1/PCI8                    PCIE         PCIE11    primary       EMP
/BB1/PCI9                    PCIE         PCIE11    primary       EMP
/BB1/PCI10                   PCIE         PCIE11    primary       EMP
/BB0/CMUL/NET0/IOVNET.PF0    PF           PCIE0     primary
/BB0/CMUL/NET0/IOVNET.PF1    PF           PCIE0     primary
/BB0/CMUL/NET2/IOVNET.PF0    PF           PCIE4     primary
/BB0/CMUL/NET2/IOVNET.PF1    PF           PCIE4     primary
/BB1/CMUL/NET0/IOVNET.PF0    PF           PCIE8     primary
/BB1/CMUL/NET0/IOVNET.PF1    PF           PCIE8     primary
/BB1/CMUL/NET2/IOVNET.PF0    PF           PCIE8     primary
/BB1/CMUL/NET2/IOVNET.PF1    PF           PCIE8     primary
#

Release the I/O devices assigned to the logical domain, from Oracle Solaris on the control domain.
If a mirror volume is configured between multiple SPARC M12-2S units, release the HDD/SSD mounted in the target SPARC M12-2S from the mirror beforehand.

a. Release the virtual network device vnet1 assigned to the logical domain guest.

# ldm remove-vnet vnet1 guest
# ldm list-bindings guest
        (Omitted)
NETWORK
    NAME   SERVICE               ID  DEVICE     MAC                MODE  PVID  VID    MTU
    MAXBW  LINKPROP
    vnet0  primary-vsw0@primary  0   network@0  00:14:4f:f9:58:4e        1            1500
      PEER                 MAC               MODE   PVID   VID   MTU   MAXBW   LINKPROP
      primary-vsw0@primary 00:14:4f:fb:e1:a8        1            1500
        (Omitted)

b. Delete primary-vsw4, which is set for the physical LAN port net4.

# ldm remove-vsw primary-vsw4
# ldm list-bindings primary
        (Omitted)
VSW
  NAME            MAC                NET-DEV  ID  DEVICE    LINKPROP  DEFAULT-VLAN-ID PVID VID
  MTU             MODE               INTER-VNET-LINK
  primary-vsw0    00:14:4f:fb:e1:a8  net0     0   switch@0            1               1
  1500            on
    PEER           MAC                PVID        VID     MTU    MAXBW    LINKPROP  INTERVNETLINK
    vnet0@guest    00:14:4f:f9:58:4e  1                   1500
        (Omitted)

c. Release the assignment of PCIE10 (pci@8a00) from the logical domain ldom.

# ldm remove-io PCIE10 ldom
# ldm list-bindings ldom
        (Omitted)
IO
    DEVICE                        PSEUDONYM    OPTIONS
    pci@8600                      PCIE6
    pci@8600/pci@4/pci@0/pci@9    /BB0/PCI5
    pci@8600/pci@4/pci@0/pci@11   /BB0/PCI6
        (Omitted)

d. Release the I/O devices of BB#01.

# ldm remove-io PCIE8 primary
# ldm remove-io PCIE9 primary
# ldm remove-io PCIE11 primary

Confirm that none of the I/O devices of BB#01 is assigned.

# ldm list-io
NAME TYPE BUS DOMAIN STATUS

---- ---- --- ------ ------

        (Omitted)
PCIE8               BUS     PCIE8
PCIE9               BUS     PCIE9
PCIE10              BUS     PCIE10
PCIE11              BUS     PCIE11
        (Omitted)
/BB1/CMUL/NET0      PCIE    PCIE8                  UNK
/BB1/CMUL/NET2      PCIE    PCIE8                  UNK
/BB1/CMUL/SASHBA    PCIE    PCIE8                  UNK
/BB1/PCI0           PCIE    PCIE9                  UNK
/BB1/PCI1           PCIE    PCIE9                  UNK
/BB1/PCI2           PCIE    PCIE9                  UNK
/BB1/PCI3           PCIE    PCIE10                 UNK
/BB1/PCI4           PCIE    PCIE10                 UNK
/BB1/PCI5           PCIE    PCIE10                 UNK
/BB1/PCI6           PCIE    PCIE10                 UNK
/BB1/PCI7           PCIE    PCIE11                 UNK
/BB1/PCI8           PCIE    PCIE11                 UNK
/BB1/PCI9           PCIE    PCIE11                 UNK
/BB1/PCI10          PCIE    PCIE11                 UNK

Log in to the XSCF shell, and release the SPARC M12-2S from the physical partition.

Release the SPARC M12-2S of BB-ID#01 from PPAR ID#01.

XSCF> deleteboard -y -c unassign 01-0
PSB#01-0 will be unconfigured from PPAR immediately. Continue?[y|n] :y
Start unconfigure preparation of PSB. [1200sec]
0end
Unconfigure preparation of PSB has completed.
Start unconfiguring PSB from PPAR. [7200sec]
0|
Processing of the incoming DR request by the LDoms Manager is pendingIncoming DR request is being processed by the LDoms ManagerDR sequence started (sequence#=2, message#=3)
Suspending the guest domain (ldom2)
Suspending the guest domain (ldom1)
Suspending the guest domain (ldom0)
\
Resumed the guest domain (ldom0)
Resumed the guest domain (ldom1)
|
Resumed the guest domain (ldom2)

../

DR sequence finished (sequence#=2, message#=3)
..
Processing of the incoming DR request by the LDoms Manager is pendingIncoming DR request is being processed by the LDoms ManagerDR sequence started (sequence#=3, message#=2)
DR sequence finished (sequence#=3, message#=2)

.end

Unconfigured PSB from PPAR.
PSB power off sequence started. [1200sec]
0..... 30..... 60..... 90.....120.....150...end
Operation has completed.

Note - You may need to assign command options depending on how the DR is used and the logical domain system configuration.
For details on command options, see "3.1.16 Deleting a System Board From a Building Block Configuration" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Domain Configuration Guide.

Check the end code of the deleteboard command.
If the end code is not 0, the deleteboard command ended abnormally. Eliminate the cause of the failure, and then execute the deleteboard command again.

In the following example, "0" is returned as the end status, so you can see that the execution of the deleteboard command has completed correctly.

XSCF> showresult
0

Note - For information on the action to take if the deleteboard command ends abnormally, see "3.1.16 Deleting a System Board From a Building Block Configuration" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Domain Configuration Guide.

Confirm that the SPARC M12-2S of BB-ID#01 has been powered off and that the system board is in the pool state.
Since the SPARC M12-2S has been released with the disconnect option, incorporation into PPAR ID#00 is reserved.

XSCF> showboards -va
PSB R PPAR-ID(LSB) Assignment Pwr Conn Conf Test Fault

---- - ------------ ----------- ---- ---- ---- ------- --------

00-0 00(00) Assigned y y y Passed Normal
01-0 * 00(01) Assigned n n n Passed Normal
XSCF>

Contents

9.4.3 Dynamically Releasing the SPARC M12-2S From the Physical Partition