Skip to main content

5.4.1 Releasing the assignment of I/O devices


5.4.1 Releasing the assignment of I/O devices
This section describes the work to be done before maintenance using PCI hot plug (PHP) or dynamic reconfiguration (PPAR DR) of the physical partition.
  1. When using PCI hot plug (PHP)
    If the PCIe end point of the PCIe card requiring maintenance is assigned to an I/O domain or if the SR-IOV virtual function is assigned, return the PCIe end point to the root or control domain.
    (1) Stopping the use of an I/O device
    (2) Deleting the assignment of an I/O device
  2. When using dynamic reconfiguration (PPAR DR) for the physical partition
    If the PCIe end point of the PCIe card in the chassis requiring maintenance is assigned to an I/O domain or if the SR-IOV virtual function is assigned, return the PCIe end point to the root or control domain. Then, remove the root complex from the root or control domain.
    (1) Stopping the use of an I/O device
    (2) Deleting the assignment of an I/O device
    (3) Deleting the root complex
(1) Stopping the use of an I/O device
If there is any logical domain that is physically or virtually using the PCIe card mounted in the chassis requiring maintenance, stop the use of the I/O device to keep it unused.
With a redundant configuration, use the duplication function to release the target I/O device so that it will not be used. For details on the work, see the manual for the application being used.
With a non-redundant configuration, place the I/O device in the unused state, such as by deactivating the network or unmounting the disk to stop the use. For details on the work, see the Oracle Solaris documents and the manual for the application being used.
(2) Deleting the assignment of an I/O device
When using virtual I/O
When using a virtual I/O service, delete the virtual I/O device. For details, see the Oracle VM Server for SPARC Administration Guide from Oracle.
The following is a command execution example for deleting the virtual disk (vdisk11) and virtual network device (vnet10) that use the virtual I/O service of the control domain or root domain.
# ldm remove-vdisk vdisk11 guest0
# ldm remove-vnet vnet10 guest0
When using PCIe end point assignment
If the PCIe end point device of the chassis requiring maintenance is assigned to an I/O domain, delete the PCIe end point device from the I/O domain and return it to the root domain (includes the control domain). The work procedure varies depending on whether the dynamic reconfiguration function for the PCIe end point device is used. For details, see Table 5-2.
Table 5-2  Procedure that depends on whether the dynamic reconfiguration function for the PCIe end point device is used
-: Unnecessary
Item Task Command Dynamic reconfiguration function used (*1) Dynamic reconfiguration function not used (*2)
1 Enable the hotplug service. svcadm enable hotplug Yes Yes (*3)
2 Stop the I/O domain. ldm stop-domain - Yes
3 Delete the PCIe end point device from the I/O domain. ldm remove-io Yes Yes
4 Set the root domain to delayed reconfiguration. ldm start-reconf - Yes
5 Assign the PCIe end point device to the root domain. ldm add-io Yes Yes
6 Restart the root domain. shutdown -i6 -g0 -y - Yes
7 Disable the hotplug service. svcadm disable hotplug Yes Yes (*3)
*1 Dynamic reconfiguration of the PCIe end point device is supported by XCP 2230 or later and Oracle VM Server for SPARC 3.1.1.1 or later.
*2 If dynamic reconfiguration of the PCIe end point devices is not used, stop the I/O domain, delete the physical I/O device, and then return the physical I/O device to the root domain.
*3 If dynamic reconfiguration of the PCIe end point device is not used, the hotplug service operates regardless of its enable/disable setting.
Note - Note the following when deleting the physical I/O device (PCIe end point device) of the boot disk.
If the boot disk of the I/O domain is in a redundant configuration, you need to stop the I/O domain on Oracle VM Server for SPARC 3.1.1 or earlier. However, you do not need to stop it on Oracle VM Server for SPARC 3.1.1.1 or later.
  1. If the hotplug service is disabled for the I/O and root domains, execute the svcadm enable command to enable the hotplug service.
# svcadm enable hotplug
  1. Stop the I/O domain.
    If you are using the dynamic reconfiguration function for the PCIe end point device, this step is unnecessary. Go to step 3.
# ldm stop-domain <I/O domain name>‌
  1. Delete the physical I/O device from the I/O domain.
# ldm remove-io <‌device name>‌ <I/O domain name>‌
  1. Transition the root domain to delayed reconfiguration mode.
    If you are using the dynamic reconfiguration function for the PCIe end point device, this step is unnecessary. Go to step 5.
# ldm start-reconf <‌root domain name>‌
  1. Reassign the physical I/O device deleted in step 3 to the root domain.
# ldm add-io <‌device name>‌ <‌root domain name>‌
  1. Log in to the root domain, and restart Oracle Solaris.
    If you are using the dynamic reconfiguration function for the PCIe end point device, this step is unnecessary. Go to step 7.
# telnet localhost <‌port number>‌
...

# shutdown -i6 -g0 -y
  1. If you need to disable the hotplug service for the I/O and root domains, disable the hotplug service.
# svcadm disable hotplug
When using the SR-IOV virtual function
Suppose that a virtual function (VF) using the SR-IOV function on the PCIe card mounted in the PCI expansion unit requiring maintenance was created and assigned to a domain. In this case, delete and discard the virtual function from the control domain by using the ldm command. The work procedure differs between the dynamic SR-IOV function and the static SR-IOV function. For details, see Table 5-3.
For details on the procedure, see the Oracle VM Server for SPARC Administration Guide from Oracle.
Table 5-3  Release procedure that depends on the function: dynamic SR-IOV function or static SR-IOV function
-: Unnecessary
Item Task Command Dynamic SR-IOV function used (*1) Static SR-IOV function used
1 Enable the hotplug service. svcadm enable hotplug Yes Yes (*2)
2 Stop the I/O domain. ldm stop-domain - Yes
3 Delete the virtual function (VF) from the I/O domain. ldm remove-io Yes Yes
4 Set the root domain of the physical function (PF) to delayed reconfiguration. ldm start-reconf - Yes
5 Discard the virtual function (VF). ldm destroy-vf Yes Yes
6 Restart the root domain. shutdown -i6 -g0 -y - Yes
7 Disable the hotplug service. svcadm disable hotplug Yes Yes (*2)
*1 Dynamic reconfiguration for the SR-IOV virtual function is supported by XCP 2210 or later and Oracle VM Server for SPARC 3.1 or later.
*2 If static reconfiguration for the SR-IOV virtual function is used, the hotplug service operates regardless of its enable/disable setting.
  1. If the hotplug service is disabled for the I/O and root domains, execute the svcadm enable command to enable the hotplug service.
# svcadm enable hotplug
  1. Stop the I/O domain.
    If the dynamic SR-IOV function is used, this step is unnecessary. Go to step 3.
# ldm stop-domain <I/O domain name>‌
  1. The virtual function (VF) has been assigned to an I/O domain. Delete it from the I/O domain.
    Execute the ldm remove-io command to delete the virtual function (VF) assigned to the I/O domain from the I/O domain. If multiple VFs are assigned from the physical function (PF) of a single PCIe card, delete all the VFs corresponding to that PF from the I/O domain.
# ldm remove-io <VF name>‌ <I/O domain name>‌
  1. Transition the root domain that has the assigned physical function (PF) to delayed reconfiguration mode.
    If the dynamic SR-IOV function is used, this step is unnecessary. Go to step 5.
# ldm start-reconf <‌root domain name>‌
  1. Discard the virtual function (VF).
    Execute the ldm destroy-vf command to discard the virtual function (VF). However, the VF needs to be discarded in the reverse order to that in which it was created.
# ldm destroy-vf <VF name>‌
  1. Log in to the root domain, and restart Oracle Solaris.
    If the dynamic SR-IOV function is used, this step is unnecessary. Go to step 7.
# telnet localhost <‌port number>‌
...

# shutdown -i6 -g0 -y
  1. If you need to disable the hotplug service for the I/O and root domains, disable the hotplug service.
# svcadm disable hotplug
(3) Deleting the root complex
Deleting the root complex from the root domain
If the I/O device of the chassis requiring maintenance is assigned to the root domain, delete the physical I/O device (root complex) from the root domain. The work procedure varies depending on whether the dynamic reconfiguration function (Dynamic PCIe bus assignment) for the root complex is used. For details, see Table 5-4.
Table 5-4  Procedure that depends on whether the dynamic reconfiguration function (Dynamic PCIe bus assignment) for the root complex is used
-: Unnecessary
Item Task Command Dynamic reconfiguration function used (*1) Dynamic reconfiguration function not used (*2)
1 Stop the root domain. ldm stop-domain - Yes
2 Delete the physical I/O device (root complex) from the root domain. ldm remove-io Yes Yes
*1 Dynamic reconfiguration of the root complex is supported by XCP 2240 or later and Oracle VM Server for SPARC 3.2 or later. In addition, the root domain must be Oracle Solaris 11.2 SRU 11.2.8 or later.
*2 If dynamic reconfiguration of the root complex is not used, stop the root domain, and then delete the physical I/O device.
  1. Stop the root domain.
    If you are dynamically reconfiguring the root complex, this step is unnecessary. Go to step 2.
# ldm stop-domain <‌root domain name>‌
  1. Delete the physical I/O device (root complex) from the root domain.
# ldm remove-io <‌physical I/O device name>‌ <‌root domain name>‌
  1. Confirm that the I/O device of the chassis to be released is unused and unassigned.
# ldm list-io
Deleting the root complex from the control domain
If the I/O device of the chassis requiring maintenance is assigned to the control domain, delete the physical I/O device (root complex) from the control domain. The work procedure varies depending on whether the dynamic reconfiguration function (Dynamic PCIe bus assignment) for the root complex is used. For details, see Table 5-5.
Table 5-5  Procedure that depends on whether the dynamic reconfiguration function (Dynamic PCIe bus assignment) for the root complex is used
-: Unnecessary
Item Task Command Dynamic reconfiguration function used (*1) Dynamic reconfiguration function not used (*2)
1 Set the control domain to delayed reconfiguration. ldm start-reconf - Yes
2 Delete the physical I/O device from the control domain. ldm remove-io Yes Yes
3 Restart the control domain. shutdown -i6 -g0 -y - Yes
*1 Dynamic reconfiguration of the root complex is supported by XCP 2240 or later and Oracle VM Server for SPARC 3.2 or later. In addition, the control domain must be Oracle Solaris 11.2 SRU 11.2.8 or later.
*2 If dynamic reconfiguration of the root complex is not used, set the control domain to delayed reconfiguration, and then delete the physical I/O device.
  1. Transition the control domain to delayed reconfiguration mode.
    If you are dynamically reconfiguring the root complex, this step is unnecessary. Go to step 2.
ldm start-reconf <‌control domain name>‌
  1. Delete the physical I/O device from the control domain.
ldm remove-io <‌physical I/O device name>‌ <‌control domain name>‌
  1. Restart the control domain.
    If you are dynamically reconfiguring the root complex, this step is unnecessary.
shutdown -i6 -g0 -y
Note - When set to delayed reconfiguration mode, the control domain does not have control over the guest domains. If you have set the control domain to delayed reconfiguration mode, restart the control domain as soon as possible.