Skip to main content
  1. Home >
  2. Products >
  3. Computing Products >
  4. Servers >
  5. Fujitsu SPARC servers >
  6. Downloads >
  7. User Manuals >
  8. Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Domain Configuration Guide >
  9. A.3.3 Example of the Active Replacement Procedure (for Oracle VM Server for SPARC 3.1.x)

A.3.3 Example of the Active Replacement Procedure (for Oracle VM Server for SPARC 3.1.x)


A.3.3 Example of the Active Replacement Procedure (for Oracle VM Server for SPARC 3.1.x)
This section describes an example of the procedure for actively replacing the BB#01 SPARC M10-4S using PPAR DR for the 2BB configuration system described in Figure A-3 Example of the 2BB Configuration for Operational Continuity (With Free Resources).
Note - If the XSCF in SPARC M10-4S to be actively replaced is defective, you cannot perform active replacement using PPAR DR.
You must stop the physical partition to which the SPARC M10-4S to be actively replaced belongs and then perform maintenance with the input power to the SPARC M10-4S to be replaced turned off.
  1. Log in to the master XSCF.
    Execute the showbbstatus command to check that the XSCF to which you have logged in is the master XSCF.
    If you have logged in to a standby XSCF, log out and then log in to the master XSCF again.
XSCF> showbbstatus
BB#00 (Master)
  1. Execute the showhardconf command to check that [Status] of the XSCF in the SPARC M10-4S to be replaced is "Normal."
XSCF> showhardconf
SPARC M10-4S;
+ Serial: 2081230011; Operator_Panel_Switch:Locked;
+ System_Power:On; System_Phase:Cabinet Power On;
Partition#0 PPAR_Status:Running;
BB#00 Status:Normal; Role:Master; Ver:2003h; Serial:2081231002;
+ FRU-Part-Number: CA07361-D202 A1 ;
+ Power_Supply_System: ;
+ Memory_Size:256 GB;
CMUL Status:Normal; Ver:0101h; Serial:PP1236052K ;
+ FRU-Part-Number:CA07361-D941 C4 /7060911 ;
+ Memory_Size:128 GB; Type: A ;
CPU#0 Status:Normal; Ver:4142h; Serial:00322658;
+ Freq:3.000 GHz; Type:0x10;
+ Core:16; Strand:2;

BB#01 Status:Normal; Role:Standby; Ver:0101h;Serial:7867000297;
+ FRU-Part-Number: CA20393-B50X A2 ;
+ Power_Supply_System: ;
+ Memory_Size:256 GB;
CMUL Status:Normal; Ver:0101h; Serial:PP123406CB ;
+ FRU-Part-Number:CA07361-D941 C4 /7060911 ;
+ Memory_Size:128 GB; Type: A ;
Note - If the XSCF in SPARC M10-4S to be actively replaced is defective, you cannot perform active replacement using PPAR DR.
You must stop the physical partition to which the SPARC M10-4S to be actively replaced belongs and then perform maintenance with the input power to the SPARC M10-4S to be replaced turned off.
  1. Execute the showbbstatus command to confirm that the XSCF in the SPARC M10-4S to be replaced is not the master XSCF.
XSCF> showbbstatus
BB#00 (Master)
  1. If the SPARC M10-4S to be replaced is the master XSCF, execute the switchscf command to switch the XSCF.
XSCF> switchscf -t Standby
The XSCF unit switch between the Active and Standby states.
Continue? [y|n] :y
Note - Confirm that the XSCF has been switched and rebooted before you release the SPARC M10-4S.
  1. Execute the console command to connect to the console of the control domain and then log in to it.
XSCF> console -p 0
  1. Check the operation status and resource usage status of the logical domain.
  1. a. Execute the ldm list-domain command to check the operation status of the logical domain.

    To check the logical domain operation status, check the [STATE] and [FLAGS] combination. If [STATE] indicates "active", the second character from the left of the string in [FLAGS] has the following meaning.

    "n": Oracle Solaris is operating

    "t": OpenBoot PROM status

    "-": In another state (including [STATE] other than "active")

    The following example shows that the control domain, two root domains, and two guest domains are operating.
  1. Check whether all domains are in "active", which indicates that Oracle Solaris is in operating state, or "inactive" state. If there is a domain with OpenBoot PROM status or bound status, the dynamic reconfiguration of the physical partition may fail.
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 16 14G 0.0% 10h 7m
guest0 active -n---- 5100 32 32G 0.0% 2h 20s
guest1 active -n---- 5101 32 32G 0.0% 2h 5s
root-dom0 active -n--v- 5000 24 24G 0.0% 2h 43s
root-dom1 active -n--v- 5001 24 24G 0.0% 2h 20s
  1. b. Execute the ldm list-devices command with the -a option specified to check the resource usage status.

    In the following example, the -a option is specified to display all resources bound to the logical domain and all resources that are not bound.
# ldm list-devices -a
CORE
    ID      %FREE    CPUSET
    0       0        (0, 1)
    4       0        (8, 9)
    8       0        (16, 17)
(Omitted)
    944     0      (1888, 1889)
    948     0      (1896, 1897)
    952     0      (1904, 1905)
    956     0      (1912, 1913)
VCPU
    PID     %FREE    PM
    0       0        no
    1       0        no
    8       0        no
    9       0        no
(Omitted)
    1904    0       no
    1905    0       no
    1912    0       no
    1913    0       no
(Omitted)
  1. Release the redundant configuration of the system volume and I/O devices in the control domain.
    This step describes how to release the I/O devices of the SPARC M10-4S to be replaced and which are used in the control domain in order enable the release of BB-lD#01 SPARC M10-4S, If you are using other redundant configuration software, see the documentation for the software for that redundant configuration for details on how to cancel the configuration.
  1. a. Cancel the redundant configuration of the system volume in the control domain.

    The following example describes how to cancel the ZFS mirroring function for the system volume in the control domain.

    Execute the zpool status command in the control domain to check the mirroring configuration status.
# zpool status rpool
pool: rpool
state: ONLINE
scan: resilvered 28.7M in 0h0m with 0 errors on Tue Jan 21 10:10:01 2014
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t50000393E802CCE2d0s0 ONLINE 0 0 0
c3t50000393A803B13Ed0s0 ONLINE 0 0 0
errors: No known data errors
  1. Execute the zpool detach command to release the disk from the mirroring configuration.
# zpool detach rpool c3t50000393A803B13Ed0
  1. Execute the zpool status command to confirm that the mirroring configuration has been canceled.
# zpool status rpool
pool: rpool
state: ONLINE
scan: resilvered 28.7M in 0h0m with 0 errors on Tue Jan 21 10:10:01 2014
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t50000393E802CCE2d0s0 ONLINE 0 0 0
errors: No known data errors
  1. If you are using other devices in BB#01, remove the redundant configuration or stop using those devices. For details on how to cancel a redundant configuration or stop using the devices, see the documentation for the software for that redundant configuration and Oracle Solaris.
  1. b. Delete the I/O configuration of the control domain.

    From among the physical I/O devices assigned to the control domain, delete the root complexes in BB#01 through delayed reconfiguration.

    First, place the control domain in delayed reconfiguration mode.
# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.
  1. Execute the ldm list-io command to check the root complexes assigned to the primary.

    The following example shows that the root complexes with BB1 devices are PCIE8 and PCIE12.
# ldm list-io | grep primary
PCIE0 BUS PCIE0 primary IOV
PCIE4 BUS PCIE4 primary IOV
PCIE8 BUS PCIE8 primary IOV
PCIE12 BUS PCIE12 primary IOV
/BB0/CMUL/NET0 PCIE PCIE0 primary OCC
/BB0/CMUL/SASHBA PCIE PCIE0 primary OCC
/BB0/CMUL/NET2 PCIE PCIE4 primary OCC
/BB1/CMUL/NET0 PCIE PCIE8 primary OCC
/BB1/CMUL/SASHBA PCIE PCIE8 primary OCC
/BB1/CMUL/NET2 PCIE PCIE12 primary OCC
  1. Execute the ldm remove-io command to delete PCIE8 and PCIE12 from the primary, and then restart Oracle Solaris.
# ldm remove-io PCIE8 primary
------------------------------------------------------------------------------

Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

# ldm remove-io PCIE12 primary
------------------------------------------------------------------------------

Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

# shutdown -i6 -g0 -y
....
  1. Once Oracle Solaris has started, execute the ldm list-io command to confirm that the root complexes in BB#01 have been deleted from the control domain.
# ldm list-io | grep primary
PCIE0 BUS PCIE0 primary IOV
PCIE4 BUS PCIE4 primary IOV
/BB0/CMUL/NET0 PCIE PCIE0 primary OCC
/BB0/CMUL/SASHBA PCIE PCIE0 primary OCC
/BB0/CMUL/NET2 PCIE PCIE4 primary OCC
  1. c. Cancel the redundant configuration of the virtual I/O devices assigned to a guest domain.

    To first shut down the root domain (root-dom1) to which the root complexes in BB#01 are assigned, log in to each guest domain, and then cancel the redundant configuration of the virtual I/O device from root-dom1.

    For details on how to use the redundant configuration software, see the documentation about the software for that redundant configuration.
  1. In the following example, a virtual network device (vnet1) is canceled from the IPMP configuration. For details on the command, see the manual for Oracle Solaris.

    Log in to the guest domain (guest0).
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 64 56G 0.0% 4h 17m
guest0 active -n---- 5100 64 64G 0.0% 1h 13m
guest1 active -n---- 5101 64 64G 0.0% 1h 4m
root-dom0 active -n--v- 5000 32 32G 0.0% 1h 47m
root-dom1 active -n--v- 5001 32 32G 0.0% 1h 19m
# telnet localhost 5100
....
guest0#
  1. Execute the dladm show-phys command to check the mapping between the virtual network interface (vnet1) and the network interface name (net1).
guest0# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 0 unknown vnet0
net1 Ethernet up 0 unknown vnet1
  1. Execute the ipmpstat -i command to check the configuration information for the network interfaces configuring IPMP.
guest0# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
net0 yes ipmp0 -smbM-- up disabled ok
net1 no ipmp0 is----- up disabled ok
  1. Execute the if_mpadm -d command to release net1 from the IPMP group, and then execute the ipmpstat -i command to confirm that it has been released. The following example confirms that STATE is offline.
guest0# if_mpadm -d net1
guest0# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
net0 yes ipmp0 -smbM-- up disabled ok
net1 no ipmp0 -s---d- up disabled offline
  1. Execute the ipadm delete-ip command to delete net1.
guest0# ipadm delete-ip net1
  1. Similarly, perform the same release processing for the guest domain (guest1).
  1. d. Remove the virtual I/O devices assigned from the root domain to be stopped.

    Execute the ldm remove-vdisk and ldm remove-vnet commands to remove the assigned virtual disk (vdisk) and virtual network device (vnet) from the root domain to be stopped according to the following step.

    The following example shows the execution of the commands for removing the virtual disk (vdisk10) and virtual network device (vnet10) that use the virtual I/O service of the BB#01 root domain (root-dom1).

    Perform the same deletion for the guest domain (guest1).
# ldm remove-vdisk vdisk10 guest0
# ldm remove-vnet vnet10 guest0
  1. Check the resource usage status of the I/O devices, and then cancel all the I/O devices in the SPARC M10-4S to be replaced.
  1. a. Check the logical domain to which the root complexes in the SPARC M10-4S to be released are assigned.

    Execute the ldm list-io command to check the logical domain to which the root complexes in BB#01 are assigned.

    The following example shows that only root-dom1 has PCIe endpoints starting with "/BB1/." You can see that the PCIe endpoint root complexes (BUS) PCIE9, PCIE10, PCIE11, PCIE13, PCIE14, and PCIE15 are assigned to root-dom1.
# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------

PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1 root-dom0 IOV
PCIE2 BUS PCIE2 root-dom0 IOV
PCIE3 BUS PCIE3 root-dom0 IOV
PCIE4 BUS PCIE4 primary IOV
PCIE5 BUS PCIE5 root-dom0 IOV
PCIE6 BUS PCIE6 root-dom0 IOV
PCIE7 BUS PCIE7 root-dom0 IOV
PCIE8 BUS PCIE8
PCIE9 BUS PCIE9 root-dom1 IOV
PCIE10 BUS PCIE10 root-dom1 IOV
PCIE11 BUS PCIE11 root-dom1 IOV
PCIE12 BUS PCIE12
PCIE13 BUS PCIE13 root-dom1 IOV
PCIE14 BUS PCIE14 root-dom1 IOV
PCIE15 BUS PCIE15 root-dom1 IOV
....
/BB1/CMUL/NET0 PCIE PCIE8 UNK
/BB1/CMUL/SASHBA PCIE PCIE8 UNK
/BB1/PCI0 PCIE PCIE9 root-dom1OCC
/BB1/PCI3 PCIE PCIE10 root-dom1OCC
/BB1/PCI4 PCIE PCIE10 root-dom1OCC
/BB1/PCI7 PCIE PCIE11 root-dom1OCC
/BB1/PCI8 PCIE PCIE11 root-dom1OCC
/BB1/CMUL/NET2 PCIE PCIE12 UNK
/BB1/PCI1 PCIE PCIE13 root-dom1OCC
/BB1/PCI2 PCIE PCIE13 root-dom1OCC
/BB1/PCI5 PCIE PCIE14 root-dom1OCC
/BB1/PCI6 PCIE PCIE14 root-dom1OCC
/BB1/PCI9 PCIE PCIE15 root-dom1OCC
/BB1/PCI10 PCIE PCIE15 root-dom1OCC
  1. b. Stop the root domain to which the root complexes in SPARC M10-4S to be released are assigned and then release the SPARC M10-4S.

    The following example executes the ldm stop-domain and ldm unbind-domain commands to release the root domain (root-dom1) and shows that the root domain is in the inactive state.
# ldm stop-domain root-dom1
LDom root-dom1 stopped
# ldm unbind-domain root-dom1
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 16 14G 0.2% 4h 59m
guest0 active -n---- 5100 32 32G 0.0% 1h 55m
guest1 active -n---- 5101 32 32G 0.0% 1h 46m
root-dom0 active -n--v- 5000 24 24G 0.0% 2h 29m
root-dom1 inactive ------ 24 24G
  1. c. Confirm that all of the I/O devices in the SPARC M10-4S to be replaced have been canceled.

    Execute the ldm list-io command to confirm that all the I/O devices have been released.
# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------

PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1 root-dom0 IOV
PCIE2 BUS PCIE2 root-dom0 IOV
PCIE3 BUS PCIE3 root-dom0 IOV
PCIE4 BUS PCIE4 primary IOV
PCIE5 BUS PCIE5 root-dom0 IOV
PCIE6 BUS PCIE6 root-dom0 IOV
PCIE7 BUS PCIE7 root-dom0 IOV
PCIE8 BUS PCIE8
PCIE9 BUS PCIE9
PCIE10 BUS PCIE10
PCIE11 BUS PCIE11
PCIE12 BUS PCIE12
PCIE13 BUS PCIE13
PCIE14 BUS PCIE14
PCIE15 BUS PCIE15
(Omitted)
  1. Release the system board (PSB<BB>) of the SPARC M10-4S from the physical partition.
  1. a. Execute the deleteboard -c disconnect command to release the PSB from the physical partition.
XSCF> deleteboard -c disconnect 01-0
PSB#01-0 will be unconfigured from PPAR immediately. Continue?[y|n] :y
Start unconfigure preparation of PSB. [1200sec]
0end
Unconfigure preparation of PSB has completed.
Start unconfiguring PSB from PPAR. [7200sec]
0..... 30..... 60....end

Unconfigured PSB from PPAR.
PSB power off sequence started. [1200sec]
0..... 30..... 60..... 90.....120.....150.....end

Operation has completed.
  1. b. Execute the showresult command to check the exit status of the deleteboard command that was just executed.

    An end value of 0 indicates the normal termination of the deleteboard command.

    If the end value is other than 0 or if an error message is displayed upon executing the deleteboard command, it indicates abnormal termination of the deleteboard command. By referring to "C.1.2 deleteboard" based on the error message, identify the error and then take corrective action.
XSCF> showresult
0
  1. c. Execute the showboards command to check the PSB status.

    Confirm that the PSB in the SPARC M10-4S to be replaced is in the “Assigned" state and that the [Pwr], [Conn], and [Conf] columns all show "n."
XSCF> showboards -p 0
PSB PPAR-ID(LSB) Assignment Pwr Conn Conf Test Fault
---- ------------ ----------- ---- ---- ---- ------- --------

00-0 00(00) Assigned y y y Passed Normal
01-0 00(01) Assigned n n n Passed Normal
  1. Execute the replacefru command to replace the SPARC M10-4S.
XSCF> replacefru
Note - For details on the replacement of SPARC M10-4Ss by using the replacefru command, see "5.8 Releasing a SPARC M10-4/M10-4S FRU from the System with the replacefru Command" and "6.2 Incorporating a SPARC M10-4/M10-4S FRU into the System with the replacefru Command" in the Fujitsu M10-4/Fujitsu M10-4S/SPARC M10-4/SPARC M10-4S Service Manual.
  1. Incorporate the PSB into the physical partition.
  1. a. Execute the showboards command to check the PSB status.

    Confirm that the PSB in the SPARC M10-4S to be replaced is in the Assigned state and that the [Pwr], [Conn], and [Conf] columns all show "n."
XSCF> showboards -p 0
PSB PPAR-ID(LSB) Assignment Pwr Conn Conf Test Fault
---- ------------ ----------- ---- ---- ---- ------- --------

00-0 00(00) Assigned y y y Passed Normal
01-0 00(01) Assigned n n n Passed Normal
  1. b. Execute the addboard -c configure command to incorporate the PSB into the physical partition.

    To recover the original logical domain configuration, execute the addboard -c configure command with the -m bind=resource option specified.
XSCF> addboard -c configure -m bind=resource -p 0 01-0
PSB#01-0 will be configured into PPAR-ID 0. Continue?[y|n] :y
Start connecting PSB to PPAR. [3600sec]
0..... 30..... 60..... 90.....120.....150.....180.....210.....240.....

270.....300.....330.....360.....390.....420.....450.....480.....510.....

540.....570.....600.....630.....660.....690.....720.....750.....780.....

810.....840.....870.....900.....930.....960.....end

Connected PSB to PPAR.
Start configuring PSB to Logical Domains (LDoms) Manager.
[1800sec] 0.....end
Configured PSB to Logical Domains (LDoms) Manager.
Operation has completed.
Note - If an error message appears during execution of the addboard command, see "C.1.1 addboard," and then identify the error and take corrective action.
  1. c. Execute the showresult command to check the exit status of the addboard command that was just executed.

    An end value of 0 indicates the normal termination of the addboard command.

    If the end value is other than 0 or if an error message is displayed upon executing the addboard command, it indicates abnormal termination of the addboard command. By referring to "C.1.1 addboard" based on the error message, identify the error and then take corrective action.
XSCF> showresult
0
  1. d. Execute the showboards command to check the PSB status.

    Confirm that both of the [Conn] and [Conf] columns show "y" after the PSB in the replaced SPARC M10-4S has been successfully incorporated.
XSCF> showboards -p 0
PSB  PPAR-ID(LSB) Assignment  Pwr  Conn Conf Test    Fault
---- ------------ ----------- ---- ---- ---- ------- --------

00-0 00(00)       Assigned    y    y    y    Passed  Normal
01-0 00(01)       Assigned    y    y    y    Passed  Normal
  1. Check the logical domain operation status.
  1. a. Execute the console command to connect to the console of the control domain and then log in to it.
XSCF> console -p 0
  1. b. Execute the ldm list-domain command to confirm that the logical domain operation status has not changed after the addition of the SPARC M10-4S PSB (BB).

    To check the logical domain operation status, check the [STATE] and [FLAGS] combination. If [STATE] indicates "active", the second character from the left of the string in [FLAGS] has the following meaning.

    "n": Oracle Solaris is operating

    "t": OpenBoot PROM status

    "-": In another state (including [STATE] other than "active")
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 16 14G 0.0% 10h 30m
guest0 active -n---- 5100 32 32G 0.0% 2h 50s
guest1 active -n---- 5101 32 32G 0.0% 2h 30s
root-dom0 active -n--v- 5000 24 24G 0.0% 3h 20s
root-dom1 inactive ------ 24 24G
  1. Restart the use of the I/O devices.
  1. a. Reassign root complexes.

    Execute the ldm bind-domain and ldm start-domain commands to start the root domain in the unbind state to which root complexes in the replaced SPARC M10-4S were assigned.

    The following example starts the root domain (root-dom1) in the unbind state, and confirms that it has started.
# ldm bind-domain root-dom1
# ldm start-domain root-dom1
LDom root-dom1 started
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 16 16G 0.2% 3h 8m
guest0 active -n---- 5100 32 32G 0.0% 3h 8m
guest1 active -n---- 5101 32 32G 0.0% 3h 8m
root-dom0 active -n--v- 5000 24 24G 0.0% 3h 8m
root-dom1 active -n--v- 5001 24 24G 7.3% 8s
  1. Execute the ldm list-io command to confirm that the physical I/O devices are assigned to the root domain that has just started.
# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------

PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1 root-dom0 IOV
PCIE2 BUS PCIE2 root-dom0 IOV
PCIE3 BUS PCIE3 root-dom0 IOV
PCIE4 BUS PCIE4 primary IOV
PCIE5 BUS PCIE5 root-dom0 IOV
PCIE6 BUS PCIE6 root-dom0 IOV
PCIE7 BUS PCIE7 root-dom0 IOV
PCIE8 BUS PCIE8
PCIE9 BUS PCIE9 root-dom1 IOV
PCIE10 BUS PCIE10 root-dom1 IOV
PCIE11 BUS PCIE11 root-dom1 IOV
PCIE12 BUS PCIE12
PCIE13 BUS PCIE13 root-dom1 IOV
PCIE14 BUS PCIE14 root-dom1 IOV
PCIE15 BUS PCIE15 root-dom1 IOV
....
/BB1/CMUL/NET0 PCIE PCIE8 UNK
/BB1/CMUL/SASHBA PCIE PCIE8 UNK
/BB1/PCI0 PCIE PCIE9 root-dom1OCC
/BB1/PCI3 PCIE PCIE10 root-dom1OCC
/BB1/PCI4 PCIE PCIE10 root-dom1OCC
/BB1/PCI7 PCIE PCIE11 root-dom1OCC
/BB1/PCI8 PCIE PCIE11 root-dom1OCC
/BB1/CMUL/NET2 PCIE PCIE12 UNK
/BB1/PCI1 PCIE PCIE13 root-dom1OCC
/BB1/PCI2 PCIE PCIE13 root-dom1OCC
/BB1/PCI5 PCIE PCIE14 root-dom1OCC
/BB1/PCI6 PCIE PCIE14 root-dom1OCC
/BB1/PCI9 PCIE PCIE15 root-dom1OCC
/BB1/PCI10 PCIE PCIE15 root-dom1OCC
  1. b. Add the virtual I/O device from the root domain to the guest domain.

    Execute the ldm add-vdisk and ldm add-vnet commands to add, to each guest domain, the virtual disk (vdisk) and virtual network device (vnet) supported for the virtual I/O service of the started root domain.

    The following example indicates the execution of the command for adding the virtual disk (vdisk10) and virtual network device (vnet10) that use the virtual I/O service of BB#01 root domain (root-dom1).

    Perform the same addition for the guest domain (guest1).
# ldm add-vdisk id=1 vdisk10 vol10@vds1 guest0
# ldm add-vnet id=1 vnet10 vsw10 guest0
Note - To add the virtual I/O device again, it is necessary to specify the ID assigned beforehand. You can check the ID from the result of execution of the ldm list -l command in the status used before the virtual I/O device is removed.
  1. c. Incorporate the virtual I/O devices assigned to a guest domain into the redundant configuration.

    Once the root domain (root-dom1) to which root complexes in BB#1 have been assigned has started, the virtual I/O device services corresponding to each guest domain are also started.

    Log in to each guest domain, and then incorporate the virtual I/O devices from root-dom1 that were previously canceled into the redundant configuration. For details on how to use the redundant configuration software, see the documentation for the software for that redundant configuration.
  1. The following describes an example of incorporating a virtual network device (vnet1) into the IPMP configuration. For details on the commands, see the manual for Oracle Solaris.

    First, log in to the guest domain (guest0).
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 16 14G 0.0% 4h 17m
guest0 active -n---- 5100 32 32G 0.0% 1h 13m
guest1 active -n---- 5101 32 32G 0.0% 1h 4m
root-dom0 active -n---- 5000 24 24G 0.0% 1h 47m
root-dom1 active -n---- 5001 24 24G 0.0% 1h 19m
# telnet localhost 5100
....
guest0#
  1. Execute the dladm show-phys command to check the mapping between the virtual network interface (vnet1) and the network interface name (net1).
guest0# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 0 unknown vnet0
net1 Ethernet up 0 unknown vnet1
  1. Execute the ipadm create-ip command, the ipadm set-ifprop command, and the ipadm add-ipmp command to register net1 as a standby device of ipmp0.
guest0# ipadm create-ip net1
guest0# ipadm set-ifprop -p standby=on -m ip net1
guest0# ipadm add-ipmp -i net1 ipmp0
  1. Execute the ipmpstat -i command to confirm that STATE of the network interface configuring IPMP indicates ok.

    Perform the same step for the other guest domain (guest1).
guest0# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
net0 yes ipmp0 -smbM-- up disabled ok
net1 no ipmp0 -s---d- up disabled ok
  1. Restore the system volume and I/O devices on the control domain to a redundant configuration.
  1. a. Add the root complex configuration for the control domain.

    By means of delayed configuration, add the root complexes in BB#01 that were previously removed from the control domain.

    First, place the control domain in delayed reconfiguration mode.
# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.
  1. Execute the ldm list-io command to check the unassigned root complexes.

    The following example shows that root complexes with BB1 devices, PCIE8 and PCIE12 are not assigned.
# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------

PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1 root-dom0 IOV
PCIE2 BUS PCIE2 root-dom0 IOV
PCIE3 BUS PCIE3 root-dom0 IOV
PCIE4 BUS PCIE4 primary IOV
PCIE5 BUS PCIE5 root-dom0 IOV
PCIE6 BUS PCIE6 root-dom0 IOV
PCIE7 BUS PCIE7 root-dom0 IOV
PCIE8 BUS PCIE8
PCIE9 BUS PCIE9 root-dom1 IOV
PCIE10 BUS PCIE10 root-dom1 IOV
PCIE11 BUS PCIE11 root-dom1 IOV
PCIE12 BUS PCIE12
PCIE13 BUS PCIE13 root-dom1 IOV
(Omitted)
  1. Execute the ldm add-io command to add PCIE8 and PCIE12 to primary, and then restart Oracle Solaris.
# ldm add-io PCIE8 primary
------------------------------------------------------------------------------

Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

# ldm add-io PCIE12 primary
------------------------------------------------------------------------------

Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

# shutdown -i6 -g0 -y
....
  1. Once Oracle Solaris has started, execute the ldm list-io command to confirm that the root complexes in BB#01 have been added to the control domain.
# ldm list-io | grep primary
PCIE0 BUS PCIE0 primary IOV
PCIE4 BUS PCIE4 primary IOV
PCIE8 BUS PCIE8 primary
PCIE12 BUS PCIE12 root-dom1
/BB0/CMUL/NET0 PCIE PCIE0 primary OCC
/BB0/CMUL/SASHBA PCIE PCIE0 primary OCC
/BB0/CMUL/NET2 PCIE PCIE4 primary OCC
  1. b. Place the system volume in the control domain in a redundant configuration.

    Execute the zpool status command in the control domain to check the mirroring configuration status.

    The following example describes how to configure the ZFS mirroring function for the system volume in the control domain.
# zpool status rpool
pool: rpool
state: ONLINE
scan: resilvered 29.1M in 0h0m with 0 errors on Thu Jan 23 17:27:59 2014
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c2t50000393E802CCE2d0s0 ONLINE 0 0 0
errors: No known data errors
  1. Execute the zpool attach command to incorporate the disks into a mirroring configuration.
# zpool attach rpool c2t50000393E802CCE2d0s0 c3t50000393A803B13Ed0s0
Make sure to wait until resilver is done before rebooting.
#
  1. Execute the zpool status command, and then confirm that the mirroring configuration has been established.

    Use the zpool status command to confirm whether synchronization processing (resilver) is completed.

    The following shows an example of the display during synchronization processing.
# zpool status rpool
pool: rpool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function in a degraded state.
action: Wait for the resilver to complete.
Run 'zpool status -v' to see device specific details.
scan: resilver in progress since Mon Jan 27 15:55:47 2014
21.1G scanned out of 70.6G at 120M/s, 0h7m to go
21.0G resilvered, 29.84% done
config:
NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c2t50000393E802CCE2d0s0 ONLINE 0 0 0
c3t50000393A803B13Ed0s0 DEGRADED 0 0 0 (resilvering)
errors: No known data errors
  1. Once synchronization processing is complete, the displayed screen will be as follows:
# zpool status rpool
pool: rpool
state: ONLINE
scan: resilvered 70.6G in 0h9m with 0 errors on Mon Jan 27 16:05:34 2014
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t50000393E802CCE2d0s0 ONLINE 0 0 0
c3t50000393A803B13Ed0s0 ONLINE 0 0 0
errors: No known data errors
  1. If you are using other devices in BB#01, establish a redundant configuration or resume the use of the devices. For details on how to establish a redundant configuration or resume the use of devices, see the documentation for the software for that redundant configuration and Oracle Solaris.