Skip to main content

A.5.2 Configuration Procedure


A.5.2 Configuration Procedure
The following gives an example of the procedure for installing a SPARC M10-4S mounted on a SPARC64 X+ processor to the physical partition mounted on the SPARC64 X processor to which the XCP firmware of XCP 2210 or earlier was applied.
  1. Save logical domain configuration information to an XML file.
    If the PPAR DR function is enabled after the XCP firmware is updated, the logical domain configuration returns to the factory setting status (factory-default). So, the logical domain configuration information saved to XSCF cannot be used. For this reason, save the current logical domain configuration information to the XML file and then update the XCP firmware. After updating, restore the logical domain configuration information from the XML file to enable the easy reconfiguration of the logical domain.
    The following explains the procedure for saving the logical domain configuration information saved in XSCF to the XML file.
    For details on the configuration information saved to the XML file, however, see the Oracle VM Server for SPARC Administration Guide.
  1. a. Log in to the control domain.

    b. Switch to the configuration information to be saved.

    Execute the ldm list-spconfig command on the control domain to list the logical domain configuration information. The line with [current] indicates the currently applied configuration information. In the following example, ldm-set3 is applied.
# ldm list-spconfig
factory-default
ldm-set1
ldm-set2
ldm-set3 [current]
  1. If only [next poweron] is displayed, there will be a difference between the logical domain configuration information stored in the XSCF and that stored in the control domain. So, execute the ldm add-spconfig command to save the current logical domain configuration information with a different name.
# ldm list-spconfig
factory-default
ldm-set1
ldm-set2
ldm-set3 [next poweron]
# ldm add-spconfig ldm-set4
# ldm list-spconfig
factory-default
ldm-set1
ldm-set2
ldm-set3
ldm-set4 [current]
  1. When the logical domain configuration information indicated in [current] matches the configuration information you want to save, proceed to step 1-c.

    If they are different, execute the ldm set-spconfig command to switch to the configuration information you want to save.

    In the following example, the configuration information is switched with the ldm set-spconfig command and then ldm-set1 is saved.
# ldm set-spconfig ldm-set1
# ldm list-spconfig
factory-default
ldm-set1 [next poweron]
ldm-set2
ldm-set3 [current]
  1. Execute the poweroff and poweron commands on XSCF to power off the physical partition (PPAR) and then power it back on. In this case, execute the poweroff command after changing the logical domain state to the state in which Oracle Solaris is running or the inactive state.

    The following example powers off PPAR-ID 0 and then powers it back on.
XSCF> poweroff -p 0
XSCF> poweron -p 0
  1. Execute the ldm list-spconfig command to check that the specified logical domain configuration information is set. The following example indicates that ldm-set1 is set as the current configuration information.
# ldm list-spconfig
factory-default
ldm-set1 [current]
ldm-set2
ldm-set3
  1. c. Execute the ldm list-constraints command to save the current logical domain configuration information to the XML file.

    Execute the ldm list-constraints command to save the current logical configuration information to the XML file.

    The following example saves the current logical domain configuration information to ldm-set1.xml. To prevent the saved XML file from being lost, back it up to another media or the like.
# ldm list-constraints -x > /ldm-set1.xml
  1. d. Check whether the configuration information is saved to the XML file.
    Execute the more command of Oracle Solaris or the like to check that the information has been saved to the XML file.
# more /ldm-set1.xml
<?xml version="1.0"?>
<LDM_interfaceversion="1.3" xmlns:xsi=http://www.w3.org/2001/XMLSchema-instancce
  1. e. To save any other logical domain configuration information, repeat steps 1.b to 1.e.

    If there is any other configuration information that you want to save, repeat steps 1.b to 1.e.
  1. Save the XSCF setting information.
    XSCF setting information can be saved to a USB device or to an external server through a network. The following describes each method.
  1. - Saving the XSCF setting information to a USB device
  1. a. Log in to the master XSCF.

    Execute the showbbstatus command to check that the XSCF to which you have logged in is the master XSCF.
Note - On a system with a building block configuration, if you logged in to the XSCF in the standby state, log in again to the master XSCF.
XSCF> showbbstatus
BB#00 (Master)
  1. b. Save the setting information to the USB device of the master XSCF.

    i. Connect the USB device to the USB port on the XSCF unit panel (rear panel) of the master XSCF.

    ii. Specify the output file name on the local USB device on the XSCF and then execute the dumpconfig command.

    The setting information is saved with the specified file name in base64 encoding text format.
XSCF> dumpconfig file:///media/usb_msd/backup-file.txt
operation completed
  1. When the "operation completed" message appears, data transfer has ended normally.
  1. iii. Once data transfer has ended, remove the USB device from the USB port.
  1. c. Using an editor or the like on a PC, check the saved setting file information.

    Check the following information.

    - User-Comments: Comments made when the -c option was specified in the dumpconfig command

    - Created: Date and time when the information was saved

    - Platform: Model name

    - Serial-No: System serial number
Note - Any file saved to a USB device cannot be opened on the XSCF. It is necessary to download the file for checking.
XSCF Configuration File
User-Comments:
Encrypted: No
Created: Mon Jan 27 13:47:38 2014
Platform: M10-4S
Serial-No: 2111234001
Chassis-serial80:
Chassis-serial81:
Chassis-serial82:
Chassis-serial83:
Chassis-serial00:2111234001
Chassis-serial01:2111234003
Chassis-serial02:
Chassis-serial03:
Chassis-serial04:
Chassis-serial05:
Chassis-serial06:
Chassis-serial07:
Chassis-serial08:
Chassis-serial09:
Chassis-serial10:
Chassis-serial11:
Chassis-serial12:
Chassis-serial13:
Chassis-serial14:
Chassis-serial15:
Version: 0001
begin-base64_common
U1VOVyxTUEFSQy1FbnRlcnByaXNlAAAAAAAAAFLmZ6gAAPrfADhbdAAAAAIyMTExMjM0MDAzAAAA
...
  1. - Saving the setting information to an external server through a network
  1. a. Log in to the master XSCF.

    Execute the showbbstatus command to check that the XSCF to which you have logged in is the master XSCF.
Note - On a system with a building block configuration, if you logged in to the XSCF in the standby state, log in again to the master XSCF.
XSCF> showbbstatus
BB#00 (Master)
  1. b. Specify the target directory through a network to save the XSCF setting information.

    Specify the target directory and output file name and then execute the dumpconfig command. The setting information is saved with the specified file name in base64 encoding text format.

    user_name specifies the user name of the save destination server.
XSCF> dumpconfig -u user-name ftp://server/backup/backup-sca-ff2-16.txt
operation completed
  1. When the "operation completed" message appears, data transfer has ended normally.
  1. c. Using an editor or the like on a PC, check the saved setting file information.

    Check the following information.
    - User-Comments: Comments made when the -c option was specified in the dumpconfig command

    - Created: Date and time when the information was saved

    - Platform: Model name

    - Serial-No: System serial number
XSCF Configuration File
User-Comments:
Encrypted: No
Created: Mon Jan 27 13:47:38 2014
Platform: M10-4S
Serial-No: 2111234001
Chassis-serial80:
Chassis-serial81:
Chassis-serial82:
Chassis-serial83:
Chassis-serial00:2111234001
Chassis-serial01:2111234003
Chassis-serial02:
Chassis-serial03:
Chassis-serial04:
Chassis-serial05:
Chassis-serial06:
Chassis-serial07:
Chassis-serial08:
Chassis-serial09:
Chassis-serial10:
Chassis-serial11:
Chassis-serial12:
Chassis-serial13:
Chassis-serial14:
Chassis-serial15:
Version: 0001
begin-base64_common
U1VOVyxTUEFSQy1FbnRlcnByaXNlAAAAAAAAAFLmZ6gAAPrfADhbdAAAAAIyMTExMjM0MDAzAAAA
...
  1. Update Oracle Solaris and Oracle VM Server for SPARC.
    Obtain Oracle Solaris and Oracle VM Server for SPARC that support physical partition dynamic reconfiguration and apply them to the system.
  1. a. Update Oracle VM Server for SPARC of the control domain.

    See the latest Product Notes for your server to check the latest versions of Oracle Solaris and Oracle VM Server for SPARC. Then, update Oracle VM Server for SPARC using the following procedure.

    i. Obtain Oracle VM Server for SPARC for use with the control domain.

    - If the control domain is Oracle Solaris 10

    Obtain Oracle VM Server for SPARC that supports physical partition dynamic reconfiguration. For details on how to obtain this, see the latest Product Notes for your server.

    - If the control domain is Oracle Solaris 11

    Obtain SRU11.1.14.0 or later.

    ii. Whenever Oracle VM Server for SPARC or Oracle Solaris for the control domain has been updated, restart the control domain.

    Restarting the control domain may cause an I/O domain to panic or a guest domain I/O to stop. If such a logical domain exists, stop it in advance with the shutdown command or stop it by executing the ldm stop-domain command from the control domain.

    Using the following method, you can check whether the logical domain is to be stopped.

    - For an I/O domain to which the PCle endpoint is assigned from the control domain

    Execute the ldm list-io command to determine whether the root complex (BUS) of the PCle endpoint assigned to the logical domain is assigned to primary. The following example indicates that the bus "PCIE2" of PCle endpoints "/BB0/PCI3" and "/BB0/PCI4" assigned to iodom0 is assigned to primary (control domain). Note that, since the configuration explained here differs from that in "A.5.1 Configuration Example," changes have been made to the logical domains and some other information.
# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------

PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1 primary IOV
PCIE2 BUS PCIE2 primary IOV
PCIE3 BUS PCIE3 primary IOV
....
/BB0/CMUL/NET0 PCIE PCIE0 primary OCC
/BB0/CMUL/SASHBA PCIE PCIE0 primary OCC
/BB0/PCI0 PCIE PCIE1 primary OCC
/BB0/PCI3 PCIE PCIE2 iodom0 OCC
/BB0/PCI4 PCIE PCIE2 iodom0 OCC
/BB0/PCI7 PCIE PCIE3 primary OCC
/BB0/PCI8 PCIE PCIE3 primary EMP
....
- For the guest domain to which the virtual service of the control domain is assigned

Execute ldm list-bindings primary to check the correspondence between the virtual network switch (VSW) and the connection destination (PEER), as well as between the virtual disk service (VDS) and the logical domain (CLIENT) that uses a virtual disk. In the following example, you can check that "guestdom0" is set for VSW PEER and VDS CLIENT.
# ldm list-bindings primary
....
VSW
NAME MAC NET-DEV ID DEVICE LINKPROP DEFAULT-VLAN-ID PVID
VID MTU MODE INTER-VNET-LINK
vsw0 00:14:4f:f9:88:ca net0 0 switch@0 1
1 1500 on
PEER MAC PVID VID MTU MAXBW LINKPROP
INTERVNETLINK
vnet0@guestdom0 00:14:4f:fa:64:dd 1 1500
VDS
NAME VOLUME OPTIONS MPGROUP DEVICE
....
CLIENT VOLUME
vdisk0@guestdom0 vol0
  1. Execute the ldm stop-domain command to stop the logical domain checked above. In the following example, the ldm stop-domain command is executed to stop iodom0 and guestdom0 and then the ldm list-domain command is executed to check that the bound state is set.
# ldm stop-domain guestdom0
LDom guestdom0 stopped
# ldm stop-domain iodom0
LDom iodom0 stopped
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 64 58G 0.0% 6h 3m
guestdom0 bound ------ 5100 64 64G
iodom0 bound ------ 5000 32 32G
  1. iii. Update Oracle VM Server for SPARC of the control domain.

    - For Oracle Solaris 10

    Uninstall the old Oracle VM Server for SPARC and then install the new Oracle VM Server for SPARC. For details, see the README file provided with the obtained Oracle VM Server for SPARC.

    - For Oracle Solaris 11

    Apply the SRU. For details, see the installation manual of the obtained SRU.

    iv. Execute the shutdown command to restart the control domain.
# shutdown -i6 -g0 -y
....
  1. v. Execute the ldm start-domain command to start the logical domain stopped in ii, above.

    In the following example, the ldm start-domain command is executed to start "iodom0" and "guestdom0" and then the ldm list-domain command is executed to check that the active state is set.
# ldm start-domain guestdom0
# ldm start-domain iodom0
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 64 58G 0.0% 6h 3m
guestdom0 active -n---- 5100 64 64G
iodom0 active -n---- 5000 32 32G
  1. b. Update Oracle Solaris for logical domains other than the control domain.

      See the latest Product Notes for your server, and update Oracle Solaris for logical domains other than the control domain. For details on the update procedure, see the information relating to each update.
  1. Update the XCP firmware.
    Install the SPARC M10-4S mounted on the SPARC64 X+ processor in the physical partition configured with the SPARC64 X processor. To do this, update the physical partition configured with the SPARC64 X processor to the XCP firmware of XCP 2220 or later.
    a. Obtain the latest XCP firmware.

    i. Download the program file for the XCP firmware.

    Download, from the web site, the program file (XCPxxxx.tar.gz or XCPxxxx.exe) for the XCP firmware to any folder on a PC connected to this system.

    Use one of the following methods to obtain the firmware for the server you are using.

    - Japanese site

    A customer who has entered into a contract with the SupportDesk can obtain firmware from SupportDesk-Web.

    - Global site

    For details on the method used to obtain the latest file of the firmware, contact our sales personnel. The following files are provided.

    - Firmware program file
    (XSCF Control Package (XCP) file)

    - XSCF extended MIB (XSCF-SP-MIB) definition file

    ii. Check the XCP version of the downloaded program file.

    Check the version of the program file of the downloaded XCP firmware. For the XCP version, refer to the 4-digit number in the file name of the firmware program (tar.gz format) to check that it is the XCP firmware version update to be applied. For example, if the name of the program file is "XCP2220.tar.gz," the XCP version is 2220.

    iii. Decompress the downloaded program file.

    Decompress the program file of the downloaded XCP firmware. The XCP image file to import into the system is expanded. For example, if "XCP2220.tar.gz" is decompressed, "BBXCP2220.tar.gz" is expanded.

    b. Check the current XCP firmware version.

    i. Log in to the master XSCF.

    Execute the showbbstatus command to check that the XSCF to which you have logged in is the master XSCF.
Note - On a system with a building block configuration, if you logged in to the XSCF in the standby state, log in again to the master XSCF.
XSCF> showbbstatus
BB#00 (Master)
  1. ii. Execute the version command to check the XCP version of the current system.

    Before updating the firmware, check the XCP version of the current system. In the following example, the version command is executed with the -c xcp option added, to check if the XCP version is XCP 2041 that does not support physical partition dynamic reconfiguration.
XSCF> version -c xcp
BB#00-XSCF#0 (Master)
XCP0 (Current): 2041
XCP1 (Reserve): 2041
  1. c. Update the XCP firmware.

    i. Execute the poweroff command to power off all the physical partitions.
XSCF> poweroff -a
  1. Execute the showpparstatus command to check that all the physical partitions are powered off.
XSCF> showpparstatus -a
PPAR-ID PPAR Status
0 Powered Off
  1. ii. Execute the showhardconf command to check that [Status] of the master and standby XSCFs is "Normal."
XSCF> showhardconf
SPARC M10-4S;
+ Serial: 2081230011; Operator_Panel_Switch:Service;
+ System_Power:Off; System_Phase:Cabinet Power Off;
Partition#0 PPAR_Status:Powered Off;
BB#00 Status:Normal; Role:Master; Ver:2003h; Serial:2081231002;
+ FRU-Part-Number: CA07361-D202 A1 ;
+ Power_Supply_System: ;
+ Memory_Size:256 GB;
  :
  1. iii. Execute the getflashimage command to import the XCP image file.

    In the following example, the USB device is connected to the USB port (MAINTENANCE ONLY printed) on the XSCF unit panel (rear panel) of the master XSCF and the XCP image file is imported.
XSCF> getflashimage file:///mnt/share/scf-firm/xscf/user/scfadmin/BBXCP2220.tar.gz
0MB received
1MB received
2MB received
...
86MB received
87MB received
88MB received
Download successful: 90539 Kbytes in 58 secs (1562.668 Kbytes/sec) Checking file...
MD5: 2b89c06548205ce35a8ecb6c2321d999
When the normal end messages "Download successful: ..." and "MD5: ..." appear, they indicate that the importing of the XCP image file has ended.
Note - The message "Warning: About to delete existing old versions." may appear when the XCP image file is imported. This message asks you to confirm whether to delete an old XCP image file that has already been imported. If "Continue?" appears, enter "y" to continue import processing.
Note - If the message "Error:File is invalid or corrupt" appears after the XCP image file is imported, this indicates that the imported XCP image file is inappropriate. The XCP image file may have been destroyed. So, obtain the correct XCP image file and then import it.
  1. iv. Execute the getflashimage -l command to check the version of the imported XCP image file.
XSCF> getflashimage -l
Existing versions:
Version Size Date
BBXCP2220.tar.gz 92712351 Thu May 23 15:01:42 JST 2014
  1. v. Execute the flashupdate -c check command to check whether the imported XCP image file can be used for update.

    Execute the showresult command immediately after the execution of the flashupdate command. If the end value is 0, update is possible.
XSCF> flashupdate -c check -m xcp -s 2220
XCP update is started. [3600sec]
0XSCF>
XSCF> showresult
0
XSCF>
  1. vi. Execute the flashupdate command to update the firmware.
XSCF> flashupdate -c update -m xcp -s 2220
The XSCF will be reset. Continue? [y|n]: y
XCP update is started. [3600sec]
0.....30.....60.....90.....120.....150.....180.....210.....240.....-

270.....300.....330.....360.....390.....420.....450.....480.....510.....|

540.....570.....600.....630.....660.....690.....720.....750.....780.....-

810.....840.....870.....900.....930

:
Here, the XSCF is rebooted and the XSCF session is disconnected. At this point, the XCP firmware update has not yet been completed.
Note - In the firmware work time, the update takes 45 minutes. On a system with a building block configuration, the update takes about 60 minutes, and automatic switching of the XSCF after the completion of the update takes about 10 minutes.
Note - To perform the update safely, do not operate the power supply of the physical partition until the message "XCP update has been completed" appears, to indicate that the XCP firmware has been updated.
  1. vii. Connect to the master XSCF again.
Note - Immediately after an XSCF reboot on a system with a building block configuration, the master and standby XSCFs will be in the exact reverse of their original states. For example, if the firmware update is executed with the master XSCF of BB-ID 0 and a connection is then made to the XSCF again, BB-ID 1 enters the master state and BB-ID 0 enters the standby state.
Note - If an inherited IP address is set and is used for connection, a connection is automatically made to the master XSCF.
  1. viii. Execute the showbbstatus command to check that you have logged in to the master XSCF.
Note - On a system with a building block configuration, if you logged in to the XSCF in the standby state, log in again to the master XSCF.
XSCF> showbbstatus
BB#00 (Master)
  1. ix. Execute the showlogs monitor command to check that XCP firmware update has been completed.

    If the message "XCP update has been completed" appears, XCP firmware update has been completed.
XSCF> showlogs monitor
Jan 23 16:42:57 5 M10-4S-0 Event: SCF:XCP update is started (XCP version=2220:
last version=2041)
Jan 23 16:44:52 M10-4S-0 Event: SCF:XSCF update is started (BBID=0, bank=0)
Jan 23 16:45:11 M10-4S-0 Event: SCF:XSCF writing is performed (BBID=0, XSCF
version=02220000)
:
Jan 23 16:54:05 01:50 M10-4S-0 Event: SCF:XSCF bank apply has been completed
(BBID=0, bank=0, XCP version=2220: last version=2041)
:
Jan 23 17:19:57 32:38 M10-4S-1 Event: SCF:CMU update has been completed (BBID=0)
Jan 23 17:19:59 32:41 M10-4S-1 Even SCF:XCP update has been completed (XCP
version=2220: last version=2041)
Note - If the message "XCP update has been completed" does not appear, update has not yet been completed. Execute the showlogs monitor command again to check that update has been completed.
  1. ⅹ. Execute the showhardconf command to check that [Status] of the XSCF is "Normal."
XSCF> showhardconf
SPARC M10-4S;
+ Serial: 2081230011; Operator_Panel_Switch:Service;
+ System_Power:Off; System_Phase:Cabinet Power Off;
Partition#0 PPAR_Status:Powered Off;
BB#00 Status:Normal; Role:Master; Ver:2003h; Serial:2081231002;
+ FRU-Part-Number: CA07361-D202 A1 ;
+ Power_Supply_System: ;
+ Memory_Size:256 GB;
  :
  1. xi. Execute the version command to check that the firmware is the new version.
XSCF> version -c xcp
BB#00-XSCF#0 (Master)
XCP0 (Current): 2220
XCP1 (Reserve): 2220
Note - If the physical partition is powered on and the firmware is updated, the current bank of the CMU firmware will be new. If the physical partition is powered off and the firmware is updated, both the reserve bank and current bank of the CMU firmware will be new. For details on the versions of the CMU firmware associated with the XCP versions, see "Existing XCP Firmware Versions and Support Information" in the latest product notes.
  1. Enable the physical partition dynamic reconfiguration function (PPAR DR function).
Note - To enable each SPARC M10-4S of the SPARC64 X processor and of the SPARC64 X+ processor to coexist in a single physical partition, execute the setpparmode command to set the CPU operational mode to compatible mode (SPARC64 X compatibility mode). For details, see "2.6.3 Conditions for a Mixed Configuration With the SPARC64 X+ Processor and the SPARC64 X Processor Within a PPAR."
Note - To use physical partition dynamic reconfiguration, it is necessary to enable the PPAR DR mode using the setpparmode command. However, if the PPAR DR mode is enabled, the logical domain configuration information is changed to the factory-default. So, the existing logical domain configuration information cannot be used. In this case, you must create new logical domain configuration information.
To use the existing logical domain configuration information, disable the PPAR DR mode. Suppose that the PPAR DR mode is enabled and that the power to the physical partition is turned on while the existing logical domain configuration information is used. Problems such as a hypervisor abort and a failure to start Oracle VM Server for SPARC normally will occur.
  1. a. Execute the showpparmode command to reference the PPAR DR mode setting.

    The following example specifies physical partition number 0 (PPAR ID 0).
XSCF> showpparmode -p 0
Host-ID :9007002b
Diagnostic Level :min
Message Level :normal
Alive Check :on
Watchdog Reaction :reset
Break Signal :on
Autoboot(Guest Domain) :on
Elastic Mode :off
IOreconfigure :false
PPAR DR(Current) :-
PPAR DR(Next) :off
  1. b. If the CPU operational mode is auto, execute the setpparmode command to set that mode to compatible.
XSCF> setpparmode -p 0 -m cpumode=compatible
  1. c. If the PPAR DR mode is disabled, execute the setpparmode command to enable that mode.
XSCF> setpparmode -p 0 -m ppar_dr=on
  1. d. Execute the showpparmode command to check that the PPAR DR mode has been enabled.
XSCF> showpparmode -p 0
Host-ID :9007002b
Diagnostic Level :min
Message Level :normal
Alive Check :on
Watchdog Reaction :reset
Break Signal :on
Autoboot(Guest Domain) :on
Elastic Mode :off
IOreconfigure :false
PPAR DR(Current) :-
PPAR DR(Next) :on
  1. Execute the showsscp command to check whether the IP address of the SP to SP communication protocol (SSCP) is the default value or a user-specified value.
XSCF> showsscp
  1. When the IP address is the default value and you are using the default value of the IP address of the SPARC M10-4S to be expanded, go to the next step.
    To set a user value, set the IP address with the setsscp command. Then, use the applynetwork command to apply and check the IP address of the SSCP of the SPARC M10-4S to be expanded. Afterwards, execute the rebootxscf command to complete the setting and then go to the next step. For details on the procedure, see "7.5.6 Applying network settings" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide.
  1. Expand SPARC M10-4S.
    a. Execute the addfru command to expand SPARC M10-4S according to the message.
Note - If the IP address of the SSCP has not yet been set, executing the addfru command results in an error.
Note - If the addfru command is executed, the firmware version of the SPARC M10-4S to be expanded is automatically set to that of the SPARC M10-4S on which the master XSCF is running. In this example, the firmware version of the SPARC M10-4S (BB-ID 1) to be expanded is automatically set to that of the SPARC M10-4S (BB-ID 0) on which the master XSCF is running.
  1. The following example expands BB#1.
XSCF> addfru
------------------------------------------------------------------------------

Maintenance/Addition Menu
Please select the chassis including added FRU.
No. FRU                 Status
--- ------------------- --------------

1  /BB#0               Normal
2  /BB#1               Unmount
3  /BB#2               Unmount
4  /BB#3               Unmount
------------------------------------------------------------------------------

Select [1-16|c:cancel] :2
Maintenance/Addition Menu
Please select the BB or a type of FRU to be added.
1. BB itself

2. PSU (Power Supply Unit)

------------------------------------------------------------------------------

Select [1,2|c:cancel] :1
Maintenance/Addition Menu
Please select a FRU to be added.
No. FRU                 Status
--- ------------------- --------------

1  /BB#1               Unmount
Select [1|b:back] :1
  1. b. Mount SPARC M10-4S to be expanded, to the rack.

    While the previously mentioned addfru command is executed, if the message "After the added device is connected with the system, please turn on the breaker of the BB#1." appears, mount the SPARC M10-4S to be expanded in the rack.

    For details on the mounting procedure, see "3.4.1 Mounting the SPARC M10-4S in a rack" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide.
  1. c. Set the identification ID (BB-ID) of the SPARC M10-4S to be expanded.

    See "4.1 Setting the ID (BB-ID) Identifying a Chassis" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide.
  1. d. Connect the crossbar cable.

    Connect the crossbar cable between the existing SPARC M10-4S and the expanded SPARC M10-4S. For the connection cable routing figure and cable list, see "Appendix B Cable Connection Information on Building Block Configurations" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide. For details on direct connections between chassis, see "4.2 Connecting Cables (for Direct Connections between Chassis)" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide.
  1. e. Connect the XSCF BB control cable.

    Connect the existing SPARC M10-4S and expanded SPARC M10-4S through the XSCF BB control cable. For the routing figure and cable list of the XSCF BB control cables, see "Appendix B Cable Connection Information on Building Block Configurations" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide. For details on direct connections between chassis, see "4.2 Connecting Cables (for Direct Connections between Chassis)" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide.
  1. f. Connect the XSCF DUAL control cable.

    Connect the existing SPARC M10-4S and expanded SPARC M10-4S through the XSCF DUAL control cable. For details, see "4.2 Connecting Cables (for Direct Connections between Chassis)" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide.
  1. g. Connect the serial cable or LAN cable.

    Connect the serial cable to the XSCF serial port of the SPARC M10-4S to be expanded. Moreover, connect the cable to each LAN port of the XSCF-LAN, GbE port, and PCIe card. For details, see "5.1 Connecting Cables to the SPARC M10-4S" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide.
  1. h. Connect the input power to the SPARC M10-4S to be expanded.

    Connect the power cord of the SPARC M10-4S to be expanded to the input power.
  1. i. Enter "f" from the input screen of the addfru command.

    Enter "f" on the input screen for the addfru command executed in step a. to perform SPARC M10-4S expansion.
2) Please select[f:finish] :f

Waiting for BB#1 to enter install state.
[This operation may take up to 20 minute(s)] (progress scale
reported in seconds)
0..... 30.... done



Waiting for BB#1 to enter ready state.
[This operation may take up to 45 minute(s)] (progress scale
reported in seconds)
0..... 30..... 60... done


Do you want to start to diagnose BB#1? [s:start|c:cancel] :
  1. j. Skip the diagnosis of the SPARC M10-4S to be expanded to exit the addfru command.

    Input "c" to the input screen of the addfru command on the master XSCF, and then skip the diagnosis processing for the SPARC M10-4S to be expanded. If "The addition of BB#1 has completed." appears, input "f" and then input "c" to exit the addfru command.
Do you want to start to diagnose BB#1? [s:start|c:cancel] :c

Diagnostic tests are about to be skipped.
Running diagnostic tests are strongly recommended before using BB#1.
Are you sure you want to skip testing? [y:yes|n:no] :y

------------------------------------------------------------------------------

Maintenance/Addition Menu Status of the added FRU.

FRU Status
------------------- --------------

/BB#1 Normal
------------------------------------------------------------------------------

[Warning:007]
Running diagnostic tests on BB#1 is strongly recommended after addfru has completed.
The addition of BB#1 has completed.[f:finish] :f

------------------------------------------------------------------------------

Maintenance/Addition Menu
Please select the chassis including added FRU.

No. FRU Status
--- ------------------- --------------

1 /BB#0 Normal
2 /BB#1 Normal


------------------------------------------------------------------------------

Select [1,2|c:cancel] :c
  1. Diagnose the expanded SPARC M10-4S.
    a. Execute the testsb command to perform a diagnosis test for the expanded SPARC M10-4S.

    Execute the testsb command to perform the diagnosis test. Specify the physical system board (PSB) number of the expanded SPARC M10-4S and check the initial diagnosis and connection I/O.
XSCF> testsb -v -p -s -y 01-0
Initial diagnosis is about to start, Continue?[y|n] :y
PSB#01-0 power on sequence started.
  1. If an error is displayed, see "A.2.4 Checking diagnosis results" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide.
  1. b. Specify the -t option with the diagxbu command to perform the crossbar cable diagnosis test of the existing SPARC M10-4S and the expanded SPARC M10-4S.

    The following example specifies the identification ID "00" (BB-ID 00) of the existing SPARC M10-4S and the identification ID "01" (BB-ID 01) of the expanded SPARC M10-4S to diagnose the crossbar cable connection status.
XSCF> diagxbu -y -b 00 -t 01
XBU diagnosis is about to start, Continue?[y|n] :y
Power on sequence started. [7200sec]
0..... 30..... 60..end

XBU diagnosis started. [7200sec]
0..... 30..... 60..... 90.....120.....150.....180.....210.....240...../

270.....300.....330.....360.....390.....420.....450.....480.....510.....\

540.....570.....600.....630.....660.....690.....720.....750.....780.....\

810.....840.....870.....900.....930...end

completed.
Power off sequence started. [1200sec]
0..... 30..... 60..... 90.....120.....150.....180.end

completed.
  1. c. Execute the showlogs error command to check that no error is displayed.
XSCF> showlogs error
  1. If an error is displayed, see "A.2.4 Checking diagnosis results" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide.
  1. d. Execute the showhardconf command to check the configuration and status of the expanded SPARC M10-4S.

    Execute the showhardconf command to check the hardware configuration (CPU, memory, or the like) of the expanded SPARC M10-4S, and then check that Status of each unit is Normal.
XSCF> showhardconf
SPARC M10-4S;
+ Serial: 2081230011; Operator_Panel_Switch:Service;
+ System_Power:Off; System_Phase:Cabinet Power Off;
Partition#0 PPAR_Status:Powered Off;
BB#00 Status:Normal; Role:Master; Ver:2003h; Serial:2081231002;
+ FRU-Part-Number: CA07361-D202 A1 ;
+ Power_Supply_System: ;
+ Memory_Size:256 GB;
CMUL Status:Normal; Ver:0101h; Serial:PP1236052K ;
+ FRU-Part-Number:CA07361-D941 C4 /7060911 ;
+ Memory_Size:128 GB; Type: A ;
CPU#0 Status:Normal; Ver:4142h; Serial:00322658;
+ Freq:3.000 GHz; Type:0x10;
+ Core:16; Strand:2;
   :

BB#01 Status:Normal; Role:Standby; Ver:0101h;Serial:7867000297;
+ FRU-Part-Number: CA20393-B50X A2 ;
+ Power_Supply_System: ;
+ Memory_Size:256 GB;
CMUL Status:Normal; Ver:0101h; Serial:PP123406CB ;
+ FRU-Part-Number:CA07361-D941 C4 /7060911 ;
+ Memory_Size:128 GB; Type: A ;
   :
  1. Set the XSCF network of the expanded SPARC M10-4S.
    Set the XSCF network for the expanded SPARC M10-4S. For details, see "7.5.2 Setting an Ethernet (XSCF-LAN) IP address" and "7.5.3 Setting a takeover IP address" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide. After making this setting, execute the applynetwork command to apply and check the setting. Afterwards, execute the rebootxscf command to complete the setting and then go to the next step. For details on the procedure, see "7.5.6 Applying network settings" in the Fujitsu M10-4S/SPARC M10-4S Installation Guide.
  1. Set mirror mode for the memory of the expanded SPARC M10-4S.
    If you are not using memory mirror mode, this step is not necessary.

    For details on memory mirror mode, see "14.1 Configuring Memory Mirroring" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide.

    a. If you are making a mirror configuration of the memory in the expanded SPARC M10-4S, use the setupfru command to set memory mirror mode.

    The following example places all the memories for the CPUs under the SPARC M10-4S physical system board (PSB 01-0) in memory mirror mode.
XSCF> setupfru -m y sb 01-0
  1. b. Execute the showfru command to check the mirror mode setting of the memory.
XSCF> showfru sb 01-0
Device Location Memory Mirror Mode
sb 01-0
cpu 01-0-0 yes
cpu 01-0-1 yes
cpu 01-0-2 yes
cpu 01-0-3 yes
  1. Register the system board of the expanded SPARC M10-4S in the physical partition configuration information.

    a. Execute the showpcl command to check the physical partition configuration information.
XSCF> showpcl -p 0
PPAR-ID LSB PSB Status
00 Powered Off
00 00-0
  1. b. Execute the setpcl command to register the system board in the physical partition configuration information.

    Execute the setpcl command to register the expanded SPARC M10-4S system board in the physical partition configuration information for the built-in destination.

    In the following example, physical system board (PSB) 01-0 is mapped to logical system board (LSB) 01 of physical partition 0.

XSCF> setpcl -p 0 -a 01=01-0
  1. c. Execute the showpcl command to check the physical partition configuration information.
XSCF> showpcl -p 0
PPAR-ID LSB PSB Status
00 Powered Off
00 00-0
01 01-0
  1. Register the CPU Activation key to assign CPU core resources.
    a. Execute the showcodusage command to check the CPU core resource information.

    Execute the showcodusage command to check whether the physical partition contains an assignable CPU core resource.

    As shown here, the system has 128 mounted CPU core resources and 128 registered CPU Activations, 64 of the CPU core resources are in use, and the number of CPU Activations currently not in use is 64.
XSCF> showcodusage -p resource
Resource In Use Installed CoD Permitted Status
-------- ------ --------- ------------- ------

PROC 64 128 128 OK: 64 cores available

Note:
Please confirm the value of the "In Use" by the ldm command of Oracle VM Server for SPARC.

The XSCF may take up to 20 minutes to reflect the "In Use" of logical domains.
Note - If the number of registered CPU Activations is not enough for the number of CPUs to be used, purchase CPU Activations and add the CPU Activation keys. For details on how to add a CPU Activation key, see "5.3 Adding CPU Core Resources" in the Fujitsu SPARC M12 and Fujitsu M10/SPARC M10 System Operation and Administration Guide.
  1. b. Execute the showcod command to check the CPU core resource information for the physical partition.

    Execute the showcod command to check information on the CPU core resources assigned to the physical partition.
XSCF> showcod -p 0
PROC Permits assigned for PPAR 0: 64
  1. If the assigned resources are insufficient, execute the setcod command to assign CPU core resources to the physical partition.

    The following example adds 64 CPU core resources to physical partition 0.
XSCF> setcod -p 0 -s cpu -c add 64
PROC Permits assigned for PPAR 0 : 64 -> 128

PROC Permits assigned for PPAR will be changed.
Continue? [y|n] :y

Completed.
Note - XSCF firmware of versions XCP 2250 and earlier do not support the -c add, -c delete, and -c set options. Specify the options of the setcod command as shown below to interactively add and delete CPU core resources.
XSCF> setcod -s cpu
  1. When the CPU core resources have been assigned, execute the showcod command again to check the information for the CPU core resources assigned to the physical partition.
XSCF> showcod -p 0
PROC Permits assigned for PPAR 0: 128
  1. Incorporate the system board (PSB<BB>) of the expanded SPARC M10-4 into the physical partition.
    a. Return to the XSCF shell, and then execute the showboards command to check the PSB status.

    Execute the showboards command to check that the PSB status of the expanded SPARCM10-4S is "SP" (system board pool).
XSCF> showboards -p 0
PSB PPAR-ID(LSB) Assignment Pwr Conn Conf Test Fault
---- ------------ ----------- ---- ---- ---- ------- --------

00-0 00(00) Assigned n n n Passed Normal
01-0 SP Available n n n Passed Normal
  1. b. Execute the addboard command to assign the PSB to the physical partition.

    Execute the addboard -c assign command to assign the PSB to the physical partition.
XSCF> addboard -c assign -p 0 01-0
PSB#01-0 will be assigned into PPAR-ID 0. Continue?[y|n] :y
XSCF>
Note - If an error message appears during execution of the addboard command, see "C.1.1 addboard," and then identify the error and take corrective action.
  1. c. Execute the showresult command to check the exit status of the addboard command that was just executed.

    An end value of 0 indicates the normal termination of the addboard command.

    If the end value is other than 0 or if an error message is displayed upon executing the addboard command, it indicates abnormal termination of the addboard command. By referring to "C.1.1 addboard" based on the error message, identify the error and then take corrective action.
XSCF> showresult
0
  1. d. Execute the showboards command to check the PSB status.

    Confirm that the PSB in the added SPARC M10-4S is in the Assigned state and that the Pwr, Conn, and Conf columns all show "n."
XSCF> showboards -p 0
PSB PPAR-ID(LSB) Assignment Pwr Conn Conf Test Fault
---- ------------ ----------- ---- ---- ---- ------- --------

00-0 00(00) Assigned n n n Passed Normal
01-0 00(01) Assigned n n n Passed Normal
  1. Power on the physical partition.
    Execute the showdomainconfig command to check that the configuration information for the logical domains to be started is set to the factory-default. The following example specifies physical partition number 0 (PPAR ID 0).
XSCF> showdomainconfig -p 0
PPAR-ID :0
Booting config
(Current) : factory-default
(Next) : factory-default
----------------------------------------------------------------

Index :1
config_name :factory-default
domains :1
date_created:-
----------------------------------------------------------------

Index :2
config_name :ldm-set1
domains :8
date_created:'2012-08-0811:34:56'
----------------------------------------------------------------

Index :3
config_name :ldm-set2
domains :20
date_created:'2012-08-0912:43:56
:
  1. Execute the poweron command to power on the physical partitions.
XSCF> poweron -p 0
  1. Using the XML file, reconfigure the logical domain.
    After completing the startup of the physical partition, reconfigure the logical domain from the XML file saved in step 1.

    a. Check that the system has started with the factory-default settings for the control domain.

    Execute the ldm list-spconfig command to check that [current] is displayed on the factory-default line.

    The following shows an example of command execution.
# ldm list-spconfig
factory-default [current]
ldm-set1
ldm-set2
ldm-set3
  1. b. Using the ldm init-system command, apply the logical domain configuration information from the XML file.

    Execute the ldm init-system command to apply the saved XML file setting. Then, execute the shutdown command to restart the control domain.
# ldm init-system -i /ldm-set1.xml
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.
# shutdown -y -g0 -i6
  1. c. To enable physical partition dynamic reconfiguration when the version of Oracle VM Server for SPARC is earlier than 3.2, adjust the memory size of the existing logical domain.

    Execute the ldm list-domain command or the like to check that the logical domains have been reconfigured from the XML file.
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 32 28G 0.0% 6m
guest0 inactive ------ 64 64G
root-dom0 inactive ------ 32 32G
  1. To enable the use of physical partition dynamic reconfiguration, set the memory size for each logical domain to "a multiple of the number of CPU cores of the logical domain x 256 MB."

    To re-set the memory size of a logical domain other than the control domain, use the ldm set-memory command.

    To re-set the memory size of the control domain, do so as follows: First enter delayed reconfiguration mode with the ldm start-reconf command, set the same number of cores with the ldm set-core command, and then re-set the memory size with the ldm set-memory command. Finally, restart Oracle Solaris.

    In this example, since the number of CPU cores of the control domain is 32, set a multiple of 32 x 256 MB = 8192 MB.

    First, obtain a multiple of 8192 MB to make the value closer to the original setting value (58 GB = 59392 MB). The result is 59392/8192 = 7.25. So, round it down to 7. As a result, the memory size to be reassigned to the control domain is 8192 MB × 7 = 56 GB.

    The following example shows the execution of the command for re-setting the control domain memory to 56 GB.
# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.
# ldm set-core 32 primary
------------------------------------------------------------------------------

Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

# ldm set-memory 56G primary
------------------------------------------------------------------------------

Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

# shutdown -i6 -g0 -y
  1. Execute the ldm list-domain command to check that the memory size (MEMORY) has been set properly.
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 64 56G 0.0% 6m
guest0 inactive ------ 64 64G
root-dom0 inactive ------ 32 32G
  1. Execute the ldm bind-domain command to bind each logical domain, and then start each logical domain by executing the ldm-start-domain command.

    The following shows command execution examples.
# ldm bind-domain root-dom0
# ldm bind-domain guest0
# ldm start-domain root-dom0
LDom root-dom0 started.
# ldm start-domain guest0
LDom guest0 started.
  1. Execute the ldm list-domain command, and then confirm that each logical domain has started. Check that [STATE] is "active" and that the second character from the left of the [FLAGS] string is "n."
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 32 28G 0.0% 18m
guest0 active -n---- 5100 64 64G 0.1% 2m
root-dom0 active -n--v- 5000 32 32G 0.0% 3m
  1. Add a new logical domain.
    Configure a new logical domain and assign the resources of the added SPARC M10-4S. The following example performs the procedure for creating a mirroring configuration according to the configuration example. In other words, add guest domain guest1 and root domain root-dom1 to create a redundant configuration of the virtual I/O for each guest domain. Furthermore, create a mirroring configuration of the system volume of the control domain with an internal hard disk.

    a. Execute the ldm list-devices command to check the state of the added hardware resources.

    When the ldm list-devices command is executed, any unassigned hardware resources are listed. Check that the hardware resources have been added. The following example checks the number of unassigned CPU cores by counting the number of lines of "free = 100" from the output of the ldm list-devices -a -p core command.
# ldm list-devices -a -p core | grep "free=100" | wc -l
64
  1. For memory, check the mapping between the mounted SPARC M10-4S location and physical address in memory from [Contains Modules and Base Address] of memory information output with the prtdiag command. Next, check the physical address and size of unassigned memory with the ldm-list-devices memory command.

    The following example enables you to determine that unassigned memory has been added. This is clear from the fact that the head of the physical address of BB-ID 1 memory is 0x700000000000 to 0x760000000000 and that the addresses in this range are displayed with the ldm list-devices memory command.
# prtdiag
....
======================= Physical Memory Configuration ========================
Segment Table:
--------------------------------------------------------------

Base Segment Interleave Bank Contains
Address Size Factor Size Modules
--------------------------------------------------------------

....
0x760000000000 32 GB 4 8 GB /BB1/CMUL/CMP0/MEM00A
....
0x740000000000 32 GB 4 8 GB /BB1/CMUL/CMP1/MEM10A
....
0x720000000000 32 GB 4 8 GB /BB1/CMUU/CMP0/MEM00A
....
0x700000000000 32 GB 4 8 GB /BB1/CMUU/CMP1/MEM10A
....
# ldm list-devices memory
MEMORY
PA SIZE BOUND
0x700000000000 32G
0x720000000000 32G
0x740000000000 32G
0x760050000000 31488M
....
  1. b. Create a new logical domain to which to assign the CPU cores and memory resources.

    However, when a domain is configured, if the number of CPU cores is first set with the ldm set-core command and then the memory size to be assigned is set with the ldm set-memory command, the memory blocks assigned to the logical domain become difficult to distribute (fragmentation) and memory placement becomes easy to consider at SPARC M10-4S release.

    The following example adds the root domain (root-dom1) and guest domain (guest1) to which CPU cores and memory are assigned.
# ldm add-domain root-dom1
# ldm add-domain guest1
# ldm set-core 16 root-dom1
# ldm set-core 32 guest1
# ldm set-memory 32G root-dom1
# ldm set-memory 64G guest1
Note - To perform physical partition dynamic reconfiguration when the version of Oracle VM Server for SPARC is earlier than 3.2, set "a multiple of number of CPU cores x 256 MB" for the memory size to be assigned to each logical domain.
  1. c. Assign the root complex to the root domain.

    Assign the root complexes of the added SPARC M10-4S to the added root domain (root-dom1). Execute the ldm list-io command to check the root complexes to be added to the root domain (root-dom1). A device whose NAME begins with "/BB1" is on BB-ID 1 and the string shown in BUS of that device indicates the root complex.
# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------

PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1 root-dom0 IOV
PCIE2 BUS PCIE2 root-dom0 IOV
PCIE3 BUS PCIE3 root-dom0 IOV
PCIE4 BUS PCIE4 primary IOV
PCIE5 BUS PCIE5 root-dom0 IOV
PCIE6 BUS PCIE6 root-dom0 IOV
PCIE7 BUS PCIE7 root-dom0 IOV
PCIE8 BUS PCIE8
PCIE9 BUS PCIE9
PCIE10 BUS PCIE10
PCIE11 BUS PCIE11
PCIE12 BUS PCIE12
PCIE13 BUS PCIE13
PCIE14 BUS PCIE14
PCIE15 BUS PCIE15
....
/BB1/CMUL/NET0 PCIE PCIE8
/BB1/CMUL/SASHBA PCIE PCIE8
/BB1/PCI0 PCIE PCIE9
/BB1/PCI3 PCIE PCIE10
/BB1/PCI4 PCIE PCIE10
/BB1/PCI7 PCIE PCIE11
/BB1/PCI8 PCIE PCIE11
/BB1/CMUL/NET2 PCIE PCIE12
/BB1/PCI1 PCIE PCIE13
/BB1/PCI2 PCIE PCIE13
/BB1/PCI5 PCIE PCIE14
/BB1/PCI6 PCIE PCIE14
/BB1/PCI9 PCIE PCIE15
/BB1/PCI10 PCIE PCIE15
  1. Execute the ldm add-io command to add the root complexes to be added to the root domain (root-dom1). Then, execute the ldm list-io command to perform a check.

    The following example adds BB1 root complexes (PCIE9, PCIE10, PCIE11, PCIE13, PCIE14, and PCIE15) other than root complexes (PCIE8 and PCIE12) of the above-mentioned on-board devices (/BB1/CMUL/NET0, /BB1/CMUL/SASHBA, and /BB1/CMUL/NET2), to root-dom1.
# ldm add-io PCIE9 root-dom1
# ldm add-io PCIE10 root-dom1
....
# ldm list-io
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------

PCIE0 BUS PCIE0 primary IOV
PCIE1 BUS PCIE1 root-dom0 IOV
PCIE2 BUS PCIE2 root-dom0 IOV
PCIE3 BUS PCIE3 root-dom0 IOV
PCIE4 BUS PCIE4 primary IOV
PCIE5 BUS PCIE5 root-dom0 IOV
PCIE6 BUS PCIE6 root-dom0 IOV
PCIE7 BUS PCIE7 root-dom0 IOV
PCIE8 BUS PCIE8 root-dom1 IOV
PCIE9 BUS PCIE9
PCIE10 BUS PCIE10 root-dom1 IOV
PCIE11 BUS PCIE11 root-dom1 IOV
PCIE12 BUS PCIE12
PCIE13 BUS PCIE13 root-dom1 IOV
PCIE14 BUS PCIE14 root-dom1 IOV
PCIE15 BUS PCIE15 root-dom1 IOV
  1. d. Assign the root complex to the root domain.

    Execute the ldm set-vconsole command to assign the console device to the root domain (root-dom1).
# ldm set-vconsole port=5001 vcc0 root-dom1
  1. Execute the ldm bind-domain and ldm start-domain commands to bind again information, including that on configured root complexes, into the root domain (root-dom1) and to start the root domain.
# ldm bind-domain root-dom1
# ldm start-domain root-dom1
LDom root-dom1 started
  1. Execute the telnet command to connect to the console of the root domain.
# telnet localhost 5001
....
  1. e. Install Oracle Solaris in the root domain.

    Install Oracle Solaris in the started root domain. For details on how to install Oracle Solaris, see the documentation related to Oracle Solaris.
  1. f. Check the physical device to be assigned to the virtual service of the root domain.

    After installing Oracle Solaris, log in to the root domain and then execute the format command to check the name of the device to be assigned to the virtual disk service. Executing the ldm list-io -l command on the control domain before logging in to the root domain displays the device path associated with the PCIe endpoint. So, you can check the path through which the disk is connected, by comparing the name of that device path with the name of the device path of the disk displayed with the format command.
# ldm list-io -l
NAME TYPE BUS DOMAIN STATUS
---- ---- --- ------ ------

....
/BB1/PCI0 PCIE PCIE9 root-dom1OCC
[pci@8900/pci@4/pci@0/pci@0]
/BB1/PCI3 PCIE PCIE10 root-dom1OCC
[pci@8a00/pci@4/pci@0/pci@0]
/BB1/PCI4 PCIE PCIE10 root-dom1OCC
[pci@8a00/pci@4/pci@0/pci@8]....
# telnet localhost 5000
....
root-dom1# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
0. c3t500000E01BDA70B2d0 <FUJITSU-MBB2147RC-3703 cyl 14087 alt 2 hd 24 sec 848>
/ pci@8900/pci@4/pci@0/pci@0/scsi@0/iport@f/disk@w500000e01bda70b2,0
1. c3t50000393A803B13Ed0 <TOSHIBA-MBF2300RC-3706 cyl 46873 alt 2 hd 20 sec 625>
/pci@8a00/pci@4/pci@0/pci@0/scsi@0/iport@f/disk@w50000393a803b13e,0
....
Specify disk (enter its number): ^C
  1. Subsequently, executing the dladm show-phys -L command enables you to check the Ethernet interface name and physical location as viewed from the top of the root domain.
root-dom0# dladm show-phys -L
LINK DEVICE LOC
net0 igb0 /BB1/PCI0
net1 igb1 /BB1/PCI0
net2 igb2 /BB1/PCI0
net3 igb3 /BB1/PCI0
  1. g. Assign the virtual I/O service to the root domain.

    Return to the control domain to add the virtual I/O service to the root domain (root-dom1). For details on the I/O device setting, see the Oracle VM Server for SPARC Administration Guide provided by Oracle Corporation.

    The following example adds the virtual disk service (vds1) and virtual switch network interfaces (vsw10 and vsw11) to the added root domain (root-dom1).
# ldm add-vdiskserver vds1 root-dom1
# ldm add-vdiskserverdevice /dev/dsk/cXtXXXXXXXXXXXXXXXXdXsX vol0@vds1
# ldm add-vdiskserverdevice /dev/dsk/cXtXXXXXXXXXXXXXXXXdXsX vol1@vds1
# ldm add-vswitch net-dev=net1 vsw10 root-dom1
# ldm add-vswitch net-dev=net2 vsw11 root-dom1
  1. h. Assign the virtual I/O device to the guest domain.

    Add the virtual I/O device that uses the added virtual I/O service, to each guest domain.

    The following example adds the virtual I/O devices (vdisk10, vnet10) that use the virtual I/O service of root-dom1 to the existing guest domain (guest0). It also adds the virtual I/O devices (vdisk1, vdisk11, vnet1, and vnet11) that use the virtual I/O services of root-dom0 and root-dom1 and the virtual console (vcons), to a guest domain (guest1) to be newly added.
# ldm add-vdisk vdisk10 vol0@vds1 guest0
# ldm add-vnet vnet10 vsw10 guest0
# ldm add-vdisk vdisk1 vol1@vds0 guest1
# ldm add-vdisk vdisk11 vol11@vds1 guest1
# ldm add-vnet vnet1 vsw1 guest1
# ldm add-vnet vnet11 vsw11 guest1
# ldm set-vconsole port=5101 guest1
  1. i. Start the added guest domain.

    Execute the ldm bind-domain and ldm start-domain commands to start the added guest domain (guest1).
# ldm bind-domain guest1
# ldm start-domain guest1
LDom guest1 started
  1. j. Install Oracle Solaris in the added guest domain.

    Install Oracle Solaris in the added guest domain. For details on how to install Oracle Solaris, see the documentation related to Oracle Solaris.
  1. k. Establish a redundant configuration for the virtual I/Os of each guest domain.

    The following describes an example of a procedure for making two virtual network interfaces (vnets), assigned to the guest domain (guest0), into a redundant configuration, using IPMP. For details on the procedures for other redundant configurations, see the documentation for the software of each redundant configuration.


    Log in to guest domain guest0. In the example below, the ldm list-domain command is used to check the port number of the console of guest0, and then the telnet command is used to connect to port number 5100.
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 16 14G 0.0% 8h 7m
guest0 active -n---- 5100 32 32G 0.0% 20s
guest1 active -n---- 5101 32 32G 0.0% 19s
root-dom0 active -n--v- 5000 24 24G 0.0% 43s
root-dom1 active -n--v- 5001 24 24G 0.0% 20s

# telnet localhost 5100
....
guest0 console login: root
Password:
...
guest0#
  1. Execute the dladm command to check that the virtual network devices are visible. In the example below, it is possible to refer to virtual network devices as network interfaces (net0 and net1).
guest0# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
net0 Ethernet up 0 unknown vnet0
net1 Ethernet up 0 unknown vnet1
  1. Execute the ipadm show-if command to check that net0 and net1 are not displayed.
guest0# ipadm show-if
IFNAME CLASS STATE ACTIVE OVER
lo0 loopback ok yes --
  1. Execute the ipadm create-ip command to create IP interfaces net0 and net1, and then use the ipadm show-if command to check that they have been created normally.
guest0# ipadm create-ip net0
guest0# ipadm create-ip net1
guest0# ipadm show-if
IFNAME CLASS STATE ACTIVE OVER
lo0 loopback ok yes --
net0 ip down no --
net1 ip down no --
  1. Execute the ipadm create-ipmp command to create IPMP interface ipmp0, and then execute the ipadm add-ipmp command to add IP interfaces net0 and net1 to the IPMP group.
guest0# ipadm create-ipmp ipmp0
guest0# ipadm add-ipmp -i net0 -i net1 ipmp0
  1. Execute the ipadm create-addr command to assign an IP address to IPMP interface ipmp0, and then execute the ipadm show-addr command to check the setting. The following example assigns a fixed IP address.
guest0# ipadm create-addr -T static -a local=xx.xx.xx.xx/24 ipmp0/v4
guest0# ipadm show-addr
ADDROBJ TYPE STATE ADDR
lo0/v4 static ok 127.0.0.1/8
ipmp0/v4 static ok xxx.xxx.xxx.xxx/24
lo0/v6 static ok ::1/128
  1. Execute the ipadm set-ifprop command to set a standby interface, and then execute the ipmpstat -i command to check the IPMP configuration.
guest0# ipadm set-ifprop -p standby=on -m ip net1
guest0# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
net1 no ipmp0 is----- up disabled ok
net0 yes ipmp0 --mbM-- up disable
  1. Perform the same procedure for the other guest domains (guest1 in the example).
  1. l. Establish a redundant configuration for the system volume of the control domain.

    The on-board device of the added SPARC M10-4S is assigned to the control domain to create a redundant configuration. Switch the control domain to delayed reconfiguration mode.
# ldm start-reconf primary
Initiating a delayed reconfiguration operation on the primary domain.
All configuration changes for other domains are disabled until the primary
domain reboots, at which time the new configuration for the primary domain
will also take effect.
  1. Add the root complexes (PCIE8 and PCIE12) of the added SPARC M10-4S to the control domain and then restart Oracle Solaris.
# ldm add-io PCIE8 primary
------------------------------------------------------------------------------

Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

# ldm add-io PCIE12 primary
------------------------------------------------------------------------------

Notice: The primary domain is in the process of a delayed reconfiguration.
Any changes made to the primary domain will only take effect after it reboots.
------------------------------------------------------------------------------

# shutdown -i6 -g0 –y
....
  1. After restarting Oracle Solaris, establish a redundant configuration for the system volume. For details on how to set the redundant configuration, see the documentation related to the software for each redundant configuration. The following example checks the disk drive with the format command and then establishes a mirror configuration with ZFS.
# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c2t50000393E802CCE2d0 <TOSHIBA-MBF2300RC-3706 cyl 46873 alt 2 hd 20 sec 625>
/pci@8000/pci@4/pci@0/pci@0/scsi@0/iport@f/disk@w50000393e802cce2,0
/dev/chassis/FUJITSU-BBEXP.500000e0e06d027f/0123_HDD00/disk
1. c3t50000393A803B13Ed0 <TOSHIBA-MBF2300RC-3706 cyl 46873 alt 2 hd 20 sec 625>
/pci@8800/pci@4/pci@0/pci@0/scsi@0/iport@f/disk@w50000393a803b13e,0
/dev/chassis/FUJITSU-BBEXP.500000e0e06d243f/022U_HDD01/disk
Specify disk (enter its number): ^C
# zpool status rpool
pool: rpool
state: ONLINE
scan: resilvered 70.6G in 0h9m with 0 errors on Mon Jan 27 16:05:34 2014
config:


NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
c2t50000393E802CCE2d0s0 ONLINE 0 0 0


errors: No known data errors
# zpool attach rpool c2t50000393E802CCE2d0s0 c3t50000393A803B13Ed0s0
Make sure to wait until resilver is done before rebooting.
  1. Execute the zpool status command, and then confirm that the mirroring configuration has been established. Use the zpool status command to confirm whether synchronization processing (resilver) has been completed. The following example shows that synchronization processing is in progress.
# zpool status rpool
pool: rpool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function in a degraded state.
action: Wait for the resilver to complete.
Run 'zpool status -v' to see device specific details.
scan: resilver in progress since Mon Jan 27 15:55:47 2014
21.1G scanned out of 70.6G at 120M/s, 0h7m to go
21.0G resilvered, 29.84% done
config:

NAME STATE READ WRITE CKSUM
rpool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
c2t50000393E802CCE2d0s0 ONLINE 0 0 0
c3t50000393A803B13Ed0s0 DEGRADED 0 0 0 (resilvering)

errors: No known data errors
  1. Once synchronization processing is complete, the displayed screen will be as follows:
# zpool status rpool
pool: rpool
state: ONLINE
scan: resilvered 70.6G in 0h9m with 0 errors on Mon Jan 27 16:05:34 2014
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2t50000393E802CCE2d0s0 ONLINE 0 0 0
c3t50000393A803B13Ed0s0 ONLINE 0 0 0

errors: No known data errors
  1. m. Check and adjust the memory placement of each logical domain.

    Execute the ldm list-devices -a memory command to check the memory block placement.
# ldm list-devices -a memory
MEMORY
PA SIZE BOUND
0x700000000000 32G root-dom1
0x720000000000 32G guest1
0x740000000000 32G guest1
0x760000800000 1272M _sys_
0x760050000000 16G primary
0x760450000000 10G primary
0x7606d0000000 4864M
0x780000000000 32G guest0
0x7a0000000000 32G guest0
0x7c0000000000 256M primary
0x7c0010000000 3584M root-dom0
0x7c00f0000000 24832M primary
0x7c0700000000 4G root-dom0
0x7e0000800000 1272M _sys_
0x7e0050000000 512M _sys_
0x7e0070000000 256M _sys_
0x7e0080000000 25088M root-dom0
0x7e06a0000000 3584M primary
0x7e0780000000 2G primary
  1. Suppose that memory blocks assigned to a logical domain or unassigned memory blocks are divided into small fragments. You can collect the fragmented memory blocks into a large continuous region by unbinding the logical domain and then binding it again. This facilitates adding or deleting the SPARC M10-4S later. For details on the procedure, see step 11-j in "A.2.2 Example of the Physical Partition Configuration Procedure."
  1. n. Save logical domain configuration information.

    Execute the ldm list-domain command to check that the configured logical domain has started.
# ldm list-domain
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -n-cv- UART 64 56G 64% 2h 54m
guest0 active -n---- 5100 64 64G 42% 2h 54m
guest1 active -n---- 5101 64 64G 11% 2h 54m
root-dom0 active -n--v- 5000 32 32G 20% 2h 54m
root-dom1 active -n--v- 5001 32 32G 18% 2h 54m
  1. Execute the ldm add-spconfig command to save the logical domain configuration information to the XSCF.
# ldm add-spconfig ldm-set4
# ldm list-spconfig
factory-default
ldm-set1
ldm-set2
ldm-set3
ldm-set4 [current]
  1. o. If there are multiple items of logical domain configuration information to be restored, return the current configuration to the factory-default configuration and then repeat steps 15 and later.

    This item gives an example of the procedure for returning the current configuration to the factory-default configuration. Execute the showdomainconfig command from the XSCF to check the configuration information for the logical domain to be started.
XSCF> showdomainconfig -p 0
PPAR-ID :0
Booting config
(Current) :ldm-set1
(Next) :ldm-set1
----------------------------------------------------------------

Index :1
config_name :factory-default
domains :1
date_created:-
----------------------------------------------------------------

Index :2
config_name :ldm-set1
domains :8
date_created:'2012-08-0811:34:56'
----------------------------------------------------------------

....
  1. Execute the setdomainconfig command to specify factory-default Index1 in physical partition number 0 (PPAR ID 0).
XSCF> setdomainconfig -p 0 -i 1
  1. Execute the poweroff and then the poweron commands to restart the physical partition. Once startup is complete, repeat steps 15 and later.
XSCF> poweroff -p 0
...
XSCF> poweron -p 0
...