Skip to main content
  1. Home >
  2. Products >
  3. Computing Products >
  4. Servers >
  5. Fujitsu SPARC servers >
  6. Downloads >
  7. User Manuals >
  8. Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Domain Configuration Guide >
  9. 2.5.2 Considerations in System Operation for Dynamic Reconfiguration

2.5.2 Considerations in System Operation for Dynamic Reconfiguration


2.5.2 Considerations in System Operation for Dynamic Reconfiguration
This section describes points of consideration in system operation for dynamic reconfiguration.
About a logical domain and a kernel zone that have Oracle Solaris 11.3 or later installed
  1. If Oracle Solaris 11.3 is installed on the control domain, reboot the SPARC M10-4S system after adding the following line to the /etc/system file in the control domain in advance. However, this procedure is unnecessary if the XCP firmware is XCP 2260 or later.
set enable_user_tick_stick_emulation = 0
  1. If a logical domain and an Oracle Solaris kernel zone (excluding the control domain) on which Oracle Solaris 11.3 or later is installed exist in the physical partition configured for the SPARC M12-2S/M10-4S, reboot the logical domain and Oracle kernel zone after adding the following line to the /etc/system files in the applicable logical domain and the Oracle Solaris kernel zone.
set uhrt_enable = 0x0
Suspend time of Oracle Solaris during deleteboard command execution
When the deleteboard command is executed, Oracle Solaris on the logical domains may temporarily stop (suspend). During this time, physical device communication and I/Os may stop, and the operation of the applications running on Oracle Solaris may also stop, which may have an impact on the business processes, such as shutting down the network with a remote unit. It is therefore necessary to determine the operations and other matters by determining, in advance, the time at which each logical domain may stop (suspend time).
The maximum suspend time is that for the suspend/resume processing caused by memory movement and the time required for suspend/resume processing for each I/O device.
An approximate value can be calculated based on the memory mounted in the chassis on which the deleteboard command will be executed, as well as the types and number of physical I/O devices mounted in the physical partition.
This is calculated using the following formula.

Suspend time = Memory movement time + Sum of suspend/resume times for on-board devices + Sum of suspend/resume times for PCI cards

The suspend time due to memory movement is 168 seconds per TB. For the calculation, check the amount of memory mounted in the chassis in which the deleteboard command is to be executed, using the showpparinfo command.
In the output example shown below, the suspend time is 21 seconds because the install size value for PSB 01-0 is 128 GB, which is displayed under "Memory:", if the system board (BB#01) is disconnected.

  128 (GB) x 168 (seconds)/1024 (GB) = 21 (seconds)
XSCF> showpparinfo -p 0
PPAR#00 Information:
--------------------

....
Memory:
-------

install
PID PSB size GB
00 00-0 128
00 01-0 128
IO Devices:
-----------

PID PSB device
....
Calculate the suspend time for physical I/Os from the types and number of all physical I/O devices mounted in the partition. The suspend/resume processing time for an on-board device is as indicated below.
- For SPARC M12,
5.2 seconds per chassis
Example) 10.4 seconds for 2BB configuration

- For SPARC M10,
21.4 seconds per chassis
Example) 42.8 seconds for 2BB configuration


For details on the suspend/resume processing times for PCI cards, see "Cards That Support PCI Hot Plug and Dynamic Reconfiguration" in the PCI Card Installation Guide for your server.
Combination with live migration
Do not perform live migration and physical partition dynamic reconfiguration at the same time.
Placement of CPU cores and memory
If the CPU or memory configuration does not satisfy the following conditions, executing the deleteboard command may cause the following message to be displayed. Take action while referring to each condition.
Some domain will be reduced with DR operation. But reducing resource is not allowed.
  1. When configured with only the control domain:
    If you use the deleteboard command to delete the system board through physical partition dynamic reconfiguration and both of the following cases are applicable, the above message will be produced and this operation will end abnormally in error: - When the logical domain configuration information is configured with only the control domain, such as factory-default, and - When the CPU cores and memory region on multiple system boards are assigned to the control domain.

    To prevent this, perform the following:
    - Specify the -m unbind=resource option in the deleteboard command to dynamically delete the CPU cores and memory region.

    - Delete the CPU and memory in advance with the ldm remove-core or ldm remove-memory command to obtain the free resource space for the BB (PSB) to be removed.

    To execute PPAR DR without reducing the number of CPU cores or amount of memory used with the control domain, it is necessary to set the configuration so that the free space is obtained in advance for the number of CPU cores and the memory region size assigned to the domain.
    The -m unbind=resource option of the deleteboard command is supported in Oracle VM Server for SPARC 3.2 or later. The PPAR DR policy is supported in Oracle VM Server for SPARC 3.4 or later. For details on the policy, see "8.15 Setting the Physical Partition Dynamic Reconfiguration Policy" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide.
  1. When configured with the control domain as well as the logical domain:

    [CPU cores]
    If the number of CPU cores assigned to a logical domain would exceed the number of CPU cores remaining after deletion by the deleteboard command, the deleteboard command will fail. This is because the number of unassigned CPU cores, which would be the destination for the assigned CPU cores, will be insufficient.

    To prevent this, perform the following:
    - Configure the logical domain in advance while leaving as many unassigned CPU cores as the number of CPU cores to be deleted with the deleteboard command.

    - Specify the -m unbind=resource option in the deleteboard command to dynamically delete the CPU cores and memory region.

    - Delete the CPU cores and memory in advance with the ldm remove-core or ldm remove-memory command to obtain the resource free space for the system board to be removed.

    The -m unbind=resource option of the deleteboard command is supported in Oracle VM Server for SPARC 3.2 or later. The PPAR DR policy is supported in Oracle VM Server for SPARC 3.4 or later. For details on the policy, see "8.15 Setting the Physical Partition Dynamic Reconfiguration Policy" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide.

    Check the assignment status of the number of CPU cores, as follows:
    1. Check the total number of CPU cores that are assigned to each logical domain.
    This is the total number of CPU cores whose "%FREE" field contains a value other than "100". You can check this number by executing the ldm list-devices -a core command on the control domain. The following example executes the ldm list-devices -a -p core command to display the total number of CPU cores assigned to a logical domain.
# ldm list-devices -a core
CORE
ID %FREE CPUSET
0 0 (0, 1)
4 0 (8, 9)
8 0 (16, 17)
12 0 (24, 25)
(Omitted)
# ldm list-devices -a -p core | egrep -v "CORE|VERSION|free=100" | wc -l
112
  1. 2. Check the total number of CPU cores on the system board that have not been removed.
    This is the total number of "cores" for the PSB numbers that have not been deleted. You can check this number by executing the showpparinfo command on the XSCF. The following example executes the showpparinfo command for the SPARC M10-4S.
XSCF>showpparinfo -p 0
PPAR#00 Information:
--------------------

CPU(s) :      8
CPU Cores :     128
CPU Threads :     256
Memory size (GB) :     256
CoD Assigned (Cores) :     256
CPU(s):
-------

PID  PSB   CPU#  Cores  Threads
00   00-0  0        16       32
00   00-0  1        16       32
00   00-0  2        16       32
00   00-0  3        16       32
00   01-0  0        16       32
00   01-0  1        16       32
00   01-0  2        16       32
00   01-0  3        16       32
(Omitted)
  1. Using the formula below, calculate the CPU core shortfall that will occur after releasing the SPARC M12-2S/M10-4S chassis.


    CPU core shortfall = Number of cores used in logical domain (1) - Number of physical cores after release (2)


    If there are not enough cores, you must reduce the number of cores by deleting those CPU cores assigned to a logical domain with the ldm remove-core command.


  1. [Memory]
    Suppose you executed the deleteboard command through physical partition dynamic reconfiguration when the memory region on the system board to be deleted is assigned to a logical domain. In this case, the contents of the memory region assigned to the logical domain are reassigned to the memory region on the system board that has not been deleted, in order to move the contents.
    Therefore, if there is no available space that is larger than the amount of memory that has been moved and which exists as an unused memory region on the destination (on the undeleted system board), the deleteboard command will end abnormally with an error.

    To prevent this, perform the following:
    - Specify the -m unbind=resource option in the deleteboard command to dynamically delete the CPU cores and memory region.

    - Delete the memory in advance with the ldm remove-memory command to obtain sufficient resource free space for the system board to be removed.

    The -m unbind=resource option of the deleteboard command is supported in Oracle VM Server for SPARC 3.2 or later. The PPAR DR policy is supported in Oracle VM Server for SPARC 3.4 or later. For details on the policy, see "8.15 Setting the Physical Partition Dynamic Reconfiguration Policy" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide.

Check the memory region use status, as follows:
  1. 1. Check the use status of the continuous region of the memory (memory block).
    Execute the prtdiag command to check the correspondence between the physical addresses of the memory and the SPARC M10-4S having a building block configuration.
# prtdiag
(Omitted)
======================= Physical Memory Configuration ========================
Segment Table:
--------------------------------------------------------------

Base Segment Interleave Bank Contains
Address Size Factor Size Modules
--------------------------------------------------------------

0x7e0000000000 32 GB 4 8 GB /BB0/CMUL/CMP0/MEM00A
(Omitted)
0x7c0000000000 32 GB 4 8 GB /BB0/CMUL/CMP1/MEM10A
(Omitted)
0x7a0000000000 32 GB 4 8 GB /BB0/CMUU/CMP0/MEM00A
(Omitted)
0x780000000000 32 GB 4 8 GB /BB0/CMUU/CMP1/MEM10A
(Omitted)
0x760000000000 32 GB 4 8 GB /BB1/CMUL/CMP0/MEM00A
(Omitted)
0x740000000000 32 GB 4 8 GB /BB1/CMUL/CMP1/MEM10A
(Omitted)
0x720000000000 32 GB 4 8 GB /BB1/CMUU/CMP0/MEM00A
(Omitted)
0x700000000000 32 GB 4 8 GB /BB1/CMUU/CMP1/MEM10A
(Omitted)
  1. The result of this example is rearranged in ascending order of physical addresses in memory. The following table lists the correspondence between the physical addresses and the SPARC M10-4S.
Table 2-10  Example of Correspondence Between Physical Addresses and the SPARC M10-4S
Base Address (Physical Address) SPARC M10-4S
0x700000000000... Building block BB-ID#01
0x720000000000... Building block BB-ID#01
0x740000000000... Building block BB-ID#01
0x760000000000... Building block BB-ID#01
0x780000000000... Building block BB-ID#00
0x7a0000000000... Building block BB-ID#00
0x7c0000000000... Building block BB-ID#00
0x7e0000000000... Building block BB-ID#00
  1. Next, execute the ldm list-devices -a memory command on the control domain to display the memory regions assigned to each logical domain, as well as any unused memory regions.
# ldm list-devices -a memory
MEMORY
PA SIZE BOUND
0x700000000000 24G root-dom1
0x700600000000 8G
0x720000000000 32G guest0
0x740000000000 32G guest1
0x760000800000 1272M _sys_
0x760050000000 24G root-dom0
0x760650000000 6912M
0x780000000000 32G
0x7a0000000000 32G
0x7c0000000000 32G
0x7e0000800000 1272M _sys_
0x7e0050000000 512M _sys_
0x7e0070000000 256M _sys_
0x7e0080000000 14G primary
0x7e0400000000 16G
  1. From the above results and physical locations in "Table 2-10 Example of Correspondence Between Physical Addresses and the SPARC M10-4S," you can determine the memory block usage status as shown below.
Table 2-11  Example of Memory Block Use Statuses
SPARC M10-4S Physical Address Size Logical Domain
Building block BB-ID#01
(to be replaced)
0x700000000000
24 GB
root-dom1
0x700600000000
8 GB
Unassigned
0x720000000000
32 GB
guest0
0x740000000000
32 GB
guest1
0x760050000000
24 GB
root-dom0
0x760650000000 6912 MB Unassigned
Building block BB-ID#00 0x780000000000
32 GB
Unassigned
0x7a0000000000
32 GB
Unassigned
0x7c0000000000
32 GB
Unassigned
0x7e0080000000
14 GB
primary
0x7e0400000000
16 GB
Unassigned
  1. 2. Check the size and quantity of the moved source memory blocks.
    While referring to the check results for the memory block use status, check the memory blocks (hereinafter referred to as "source memory blocks") assigned to the SPARC M10-4S to be replaced.
    In "Table 2-11 Example of Memory Block Use Statuses," you can determine that the number of memory blocks assigned to the logical domain is 32 GB x 2 (assignment to guest0 and guest1) and 24 GB x 1 (root-dom0) on the building block BB-ID#01 side.
Note - If root domain root-dom1 to which I/O of building block BB-ID#01 is assigned is unbound and placed in the inactive state before that building block is disconnected, you can exclude root-dom1 from movement.
  1. 3. Check the empty memory blocks.
    Next, based on the check results obtained in step 1, check the memory blocks (hereinafter referred to as "empty memory blocks") not assigned to the logical domain on the SPARC M10-4S that is not disconnected.
    For the example given in "Table 2-11 Example of Memory Block Use Statuses," it is possible to determine that the number of empty memory blocks is 32 GB x 3 and 16 GB x 1.
  1. 4. Check whether the memory block can be moved.
    Using the check results obtained in steps 2 and 3, check whether the source memory block can be moved to the empty memory block.
    This is possible if the size of the empty memory block is equal to or greater than that of the source memory block.


  1. When the destination contains empty resources

    For example, in "Table 2-11 Example of Memory Block Use Statuses," there are 32 GB x 3 empty memory blocks that are the destinations for guest0 (32 GB), guest1 (32 GB), and root-dom0 (24 GB). So, you can determine that the memory is placed such that building block BB-ID#01 can be disconnected. This is summarized in "Table 2-12 Destination Candidate Memory Blocks."
Table 2-12  Destination Candidate Memory Blocks
SPARC M10-4S Size Logical Domain Destination Candidate
Building block BB-ID#01
(to be replaced)
24 GB
root-dom1 -
8 GB
Unassigned -
32 GB
guest0
32 GB of building block BB-ID#00
32 GB
guest1
32 GB of building block BB-ID#00
24 GB
root-dom0
32 GB of building block BB-ID#00
6912 MB Unassigned -
Building block BB-ID#00
32 GB
Unassigned Moved here
32 GB
Unassigned Moved here
32 GB
Unassigned Moved here
14 GB
primary -
16 GB
Unassigned Excluded from destination candidates due to size insufficiency
  1. When there are no empty resources at the destination
    For example, in the configuration in "Table 2-13 Example of Placement of Memory Blocks With No Destinations," the number of source memory blocks is 32 GB x 2 and 24 GB x 2. Meanwhile, the free memory blocks at the destination are 32 GB x 3 and 16 GB x 1.
    As a result, one memory block of 32 GB (guest0) and two memory blocks of 24 GB (two of guest1, guest2, and root-dom0) can be moved.
    However, the number of remaining empty destination memory blocks is 16 GB x 1 and 8 GB x 2, obtained after the 24-GB memory block has been moved to a 32-GB memory block. So, any of guest1, guest2, and root-dom0, to which a 24-GB memory block is assigned, cannot be moved.  In this case, you need to reduce the size of a possibly unmovable memory block in the logical domain to be equal to or less than the memory block size at the destination.
    In the above example, you must execute the ldm remove-memory command to change any one of guest1, guest2, and root-dom0 from 24 GB to 16 GB or less.
Table 2-13  Example of Placement of Memory Blocks With No Destinations
SPARC M10-4S Size Logical Domain Destination Candidate
Building block BB-ID#01
(to be replaced)
24 GB
guest2 May not be movable
8 GB
root-dom1 -
32 GB
guest0
32 GB of building block BB-ID#00
32 GB
guest1 May not be movable
24 GB
root-dom0 May not be movable
6912 MB Unassigned -
Building block BB-ID#00
32 GB
Unassigned Moved here
32 GB
Unassigned Any one of guest1, guest2, and root-dom0 (24 GB) moves here, with 8 GB remaining.
32 GB
Unassigned Any one of guest1, guest2, and root-dom0 (24 GB) moves here, with 8 GB remaining.
14 GB
primary -
16 GB
Unassigned Excluded from destination candidates due to size insufficiency
Dynamic reconfiguration operations when recovery mode is enabled
  1. Suppose that you add a system board using dynamic reconfiguration of physical partitions in the condition where the domain configuration has been recovered in the degraded configuration. The added resource is not allocated automatically to any logical domain. Allocate the added resource manually. Alternatively, execute the ldm set-spconfig command to select the original domain configuration and then reboot the physical partition using the poweron and poweroff commands of XSCF firmware.
  2. Suppose that you delete a system board (PSB) using the deleteboard command while the physical partition (PPAR) is powered on after the domain configuration is recovered in a degraded configuration. This deleteboard command may fail if the version of Oracle VM Server for SPARC is earlier than 3.2. After a domain configuration is recovered in a degraded configuration, do not delete a system board using dynamic reconfiguration of physical partitions.
Combination with Oracle Solaris kernel zones
When an Oracle Solaris kernel zone is running on any logical domain in the physical partition (PPAR), physical partition dynamic reconfiguration cannot be performed. Stop the Oracle Solaris kernel zone and then perform physical partition dynamic reconfiguration.
For the logical domain to which the virtual service is assigned
If the logical domain to which the virtual service is assigned is any of the following, it is necessary to remove the virtual disk server device (vdsdev) and the virtual network switch (vsw) assigned to the physical I/O, and the virtual disks (vdisk) and the virtual network (vnet) assigned to them by using the ldm remove-vdsdev command, the ldm remove-vsw command, the ldm remove-vdisk command, and the ldm remove-vnet command respectively beforehand.
- When a PCIe endpoint is deleted dynamically with the ldm remove-io command

- When the physical I/O of the logical domains is deleted dynamically by the dynamic reconfiguration of the physical partition
Changing the USB device path after a PPAR DR operation
If you delete a system board (PSB<BB>) and then add one using the physical partition dynamic reconfiguration function, the "usb@4,1" part of the USB device path for the deleted/added PSB (BB) changes to "usb@4", as shown in the following example.
- Example for an external DVD drive (front)

Before deletion: /pci@8000/pci@4/pci@0/pci@1/pci@0/usb@4,1/hub@2/cdrom@1/disk@0,0:a

After addition: /pci@8000/pci@4/pci@0/pci@1/pci@0/usb@4/hub@2/cdrom@1/disk@0,0:a

- Example for a DVD drive for remote storage

Before deletion: /pci@8000/pci@4/pci@0/pci@1/pci@0/usb@4,1/storage@3/disk@0,0:a

After addition: /pci@8000/pci@4/pci@0/pci@1/pci@0/usb@4/storage@3/disk@0,0:a

However, the device path returns to the state before deletion after the restart of the logical domain assigned the USB device whose device path was changed.
This phenomenon does not occur for USB devices connected to PSBs (BBs) that are not deleted/added using the physical partition dynamic reconfiguration function.
To configure a system using the dynamic reconfiguration of physical partitions, the document including more detailed considerations and best practices has been posted to the Fujitsu SPARC servers Documentations site.
https://www.fujitsu.com/global/products/computing/servers/unix/sparc/downloads/documents/index.html
Refer to "1.2 Overview of PPAR DR" and "Appendix.A PPAR DR deleteboard Best Practice" in the Building High Availability System on Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 Servers (Maintenance procedure).