Updated 22 March 2010
Linux Logo

Enable Zone to Use Variable Number of CPU's


Description:
  • This procedure describes how to create a pools that have a min and max number of CPUs.

Prerequisites:
  • Resource Pools must be enable.
  • Must be root.
Notes:
  • A variable processor set is defined when the maximum and minimum number of processors are different value in the set.
  • The daemon poold will shuffle CPUs from one zone to another and the default set as demand varies.
References:

Step Action Description
1 Enable Pool
  • Use pgrep and pooladm to check to see if resource pool faclility is already running.
  • If not already running use pooladm -e to start resource pool facility.
# pooladm -e
global# pgrep -l poold
global# pooladm
pooladm: couldn't open pools state file: Facility is not active 
global# 
global# pooladm -e
global# 
global# pgrep -l poold
 1429 poold
global# pooladm
 
system titan
        string  system.comment
        int     system.version 1
[...]
2 Configure Pools Script

To make sure that a zone has a fixed number of processors assigned, the minimum and maximum values of the processor set must be of different values.

  • Create a text file consisting of the parameters for creation of the new pools and processor sets. This text file contains three definition types.
    • defined a processor set,
    • defines a pool and
    • associates a pool to a processor set.
mypool temporary text file
# myscript - poolcfg script for varying CPUs.
#
# Load using:
# pooladm -x; pooladm -s; poolcfg -f myscript; pooladm -c
# 
# 14-Jan-2005 Brendan Gregg Created this.
 
create pset work1-pset ( uint pset.min = 2; uint pset.max = 8 )
create pset work2-pset ( uint pset.min = 1; uint pset.max = 4 )
create pset work3-pset ( uint pset.min = 1; uint pset.max = 2 )
create pool work1-pool
create pool work2-pool
create pool work3-pool
create pool work4-pool
associate pool work1-pool ( pset work1-pset )
associate pool work2-pool ( pset work2-pset )
associate pool work3-pool ( pset work3-pset )

This text file consist of three entry types with key words:

  • create pset - creates a processor set definition using the keywords "create pset <set name>-pset" followed in parenthesis the unit of measure with the minimum and maximum number of processor separated by a semicolon surrounded by parenthesis. To create a variable set of processor allocation, the minimum and maximum values are different. The name of the set should use the "-pset" convention. A processor set uses an integer unit of measure ("uint").
create pset <set name>-pset ( uint pset.min=<same value> uint pset.max=<same value> )
  • create pool - creates a resource pool using the keywords "create pool <pool name>-pool". The name should use the "-pool" convention.
create pool <pool name>-pool 
  • associate pool - associates a resource pool to a processor set using the keywords "associate pool <pool name>" followed by the processor set name in parenthesis using the keyword "pset".
assocaite pool <pool name>-pool ( pset <processor set name>-pset ) 
3 Configure Pools Configure and activate the new pool and processor set configuration using the pooladm and poolcfg in this order:
pooladm -x - flush current in-memory pool configuration
pooladm -s - create default /etc/pooladm.conf configuration file
poolcfg -f <pool file name> - process and save "pool file name" configuration in /etc/pooladm.conf
global# ls /etc/pooladm.conf
/etc/pooladm.conf: No such file or directory
global# pooladm -x
global# pooladm -s
global# ls /etc/pooladm.conf
/etc/pooladm.conf
global# cat /etc/pooladm.conf
<?xml version="1.0"?>
<!DOCTYPE system PUBLIC "-//Sun Microsystems Inc//DTD Resource Management 
                          All//EN" "file:///usr/share/lib/xml/dtd/rm_pool.dtd.1">
<!--
Configuration for pools facility. Do NOT edit this file by hand - 
                                        use poolcfg(1) or libpool(3POOL) instead.
-->
<system ref_id="dummy" name="lqwv20z" comment="" version="1" bind-default="true">
  <property name="system.poold.pid" type="int">28231</property>
  <pool name="pool_default" active="true" default="true" importance="1" 
                                        comment="" res="pset_-1" ref_id="pool_0">
    <property name="pool.sys_id" type="int">0</property>
  </pool>
  <res_comp type="pset" sys_id="-1" name="pset_default" default="true" min="1" 
                      max="65536" units="population" comment="" ref_id="pset_-1">
    <property name="pset.load" type="uint">0</property>
    <property name="pset.size" type="uint">2</property>
    <comp type="cpu" sys_id="1" comment="" ref_id="cpu_1">
      <property name="cpu.status" type="string">on-line</property>
    </comp>
    <comp type="cpu" sys_id="0" comment="" ref_id="cpu_0">
      <property name="cpu.status" type="string">on-line</property>
    </comp>
  </res_comp>
</system>
global# poolcfg -f mypool
global# cat /etc/pooladm.conf
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE system PUBLIC "-//Sun Microsystems Inc//DTD Resource Management
                          All//EN" "file:///usr/share/lib/xml/dtd/rm_pool.dtd.1">
<!--
Configuration for pools facility. Do NOT edit this file by hand -
                                        use poolcfg(1) or libpool(3POOL) instead.
-->
<system ref_id="dummy" name="lqwv20z" comment="" version="1" bind-default="true">
  <property name="system.poold.pid" type="int">28231</property>
  <pool name="pool_default" active="true" default="true" importance="1" 
                                        comment="" res="pset_-1" ref_id="pool_0">
    <property name="pool.sys_id" type="int">0</property>
  </pool>
  <res_comp type="pset" sys_id="-1" name="pset_default" default="true" min="1"
                      max="65536" units="population" comment="" ref_id="pset_-1">
    <property name="pset.load" type="uint">0</property>
    <property name="pset.size" type="uint">2</property>
    <comp type="cpu" sys_id="1" comment="" ref_id="cpu_1">
      <property name="cpu.status" type="string">on-line</property>
    </comp>
    <comp type="cpu" sys_id="0" comment="" ref_id="cpu_0">
      <property name="cpu.status" type="string">on-line</property>
    </comp>
  </res_comp>
  <res_comp ref_id="id_0" sys_id="-2" type="pset" name="work1-pset" min="2"
                                          max="8" units="population" comment="">
    <property name="pset.load" type="uint">0</property>
    <property name="pset.size" type="uint">0</property>
  </res_comp>
  <property name="system._next_id" type="uint">7</property>
  <res_comp ref_id="id_1" sys_id="-2" type="pset" name="work2-pset" min="1"
                                          max="4" units="population" comment="">
    <property name="pset.load" type="uint">0</property>
    <property name="pset.size" type="uint">0</property>
  </res_comp>
  <res_comp ref_id="id_2" sys_id="-2" type="pset" name="work3-pset" min="1"
                                          max="2" units="population" comment="">
    <property name="pset.load" type="uint">0</property>
    <property name="pset.size" type="uint">0</property>
  </res_comp>
  <pool ref_id="id_3" res="id_0" name="work1-pool" active="true"
                                                     importance="1" comment=""/>
  <pool ref_id="id_4" res="id_1" name="work2-pool" active="true"
                                                     importance="1" comment=""/>
  <pool ref_id="id_5" res="id_2" name="work3-pool" active="true"
                                                     importance="1" comment=""/>
  <pool ref_id="id_6" res="pset_-1" name="work4-pool" active="true"
                                                     importance="1" comment=""/>
</system>
global# pooladm -c
Notes:
  • Creating a temporary script configuration file to define a pool and processor set is the recommended procedure. But it may also be accomplished temporarily using poolcfg -dc
global# poolcfg -dc info
global# poolcfg -dc 'create pset work1-pset'
global# poolcfg -dc 'create pool work1-pool'
global# poolcfg -dc 'modify pset work1-pset ( uint pset.min = 2; uint pset.max = 8 )'
global# poolcfg -dc 'modify pool work1-pool ( string pool.scheduler = "FSS" )'
global# poolcfg -dc 'modify pool work1-pool ( int pool.importance = 10 )'
global# poolcfg -dc 'destroy pool work1-pool'

The "-d" option changes the memory configuration directly.

  • Use "poolcfg -dc info" to verify that the new pool set does exist.
  • Be careful when rerunning those four commands on a system that already has zones and pools configured. Existing zones may need to be rebooted or poolbind to attach their processor set.
4 Load Pools
  • Use "pooladm -c" to load processor set.
  • Check the processor sets by running "psrset" before and after to check if any processor sets have been automatically created.
global# pooladm -x
global# pooladm -s
global# poolcfg -f <pool file name>
global# pooladm -c
global# psrset  
user processor set 1: processors 0 1 2
user processor set 2: processors 513 514
user processor set 3: processors 3 512
global#
  • Use "pooladm" to check the current state in more detail.
global# pooladm
 
system titan
        string  system.comment 
        int     system.version 1
        boolean system.bind-default true
        int     system.poold.pid 247
 
        pool work2-pool
                int     pool.sys_id 61
                boolean pool.active true
                boolean pool.default false
                int     pool.importance 1
                string  pool.comment
                pset    work2-pset
 
        pool work1-pool
                int     pool.sys_id 60
                boolean pool.active true
                boolean pool.default false
                int     pool.importance 1
                string  pool.comment 
                pset    work1-pset
 
        pool work4-pool
                int     pool.sys_id 63
                boolean pool.active true
                boolean pool.default false
                int     pool.importance 1
                string  pool.comment 
                pset    pset_default
 
        pool work3-pool
                int     pool.sys_id 62
                boolean pool.active true
                boolean pool.default false
                int     pool.importance 1
                string  pool.comment 
                pset    work3-pset
 
        pool pool_default
                int     pool.sys_id 0
                boolean pool.active true
                boolean pool.default true
                int     pool.importance 1
                string  pool.comment 
                pset    pset_default
 
        pset work1-pset
                int     pset.sys_id 1
                boolean pset.default false
                uint    pset.min 2
                uint    pset.max 8
                string  pset.units population
                uint    pset.load 29
                uint    pset.size 3
                string  pset.comment 
 
                cpu
                        int     cpu.sys_id 1
                        string  cpu.comment
                        string  cpu.status on-line
 
                cpu
                        int cpu.sys_id 0
                        string  cpu.comment
                        string  cpu.status on-line
 
                cpu
                        int     cpu.sys_id 2
                        string  cpu.comment 
                        string  cpu.status on-line
 
        pset work3-pset
                int     pset.sys_id 3
                boolean pset.default false
                uint    pset.min 1
                uint    pset.max 2
                string  pset.units population
                uint    pset.load 22
                uint    pset.size 2
                string  pset.comment 
     
                cpu
                        int cpu.sys_id 3
                        string  cpu.comment
                        string  cpu.status on-line
         
                cpu
                        int cpu.sys_id 512
                        string  cpu.comment 
                        string  cpu.status on-line
 
        pset work2-pset
                int     pset.sys_id 2
                boolean pset.default false
                uint    pset.min 1
                uint    pset.max 4
                string  pset.units population
                uint    pset.load 35
                uint    pset.size 2
                string  pset.comment 
         
                cpu
                        int cpu.sys_id 514
                        string  cpu.comment
                        string  cpu.status on-line
         
                cpu
                        int cpu.sys_id 513
                        string  cpu.comment
                        string  cpu.status on-line
     
        pset pset_default
            int pset.sys_id -1
            boolean pset.default true
            uint    pset.min 1
            uint    pset.max 65536
            string  pset.units population
            uint    pset.load 14040
            uint    pset.size 1
            string  pset.comment
     
            cpu
                int cpu.sys_id 515
                string  cpu.comment
                string  cpu.status on-line
global#               
Notes:
  • If for some reason the pool set or processor set cannot be confured, and error message "pooladm: configuration at '/etc/pooladm.conf' cannot be instantiated on current system" will be generated. Update configuration and perform the sets above again. For example assigning all the processor in the global server to a single zone will generate this type of error.
  • poold has choosen how to divide the CPU resources when the "pooladm -c" command was executed, and assigned CPUs to processor sets guided by the min and max values. The poolcfg transfer command can be used to manually move CPU assignments around, limited by the min and max values.
5 Configure Zones Use /usr/sbin/zonecfg to permanently assoiciate the exiting pool or the processor set to an existing zone using the set pool option.
# zonecfg -z <zone name>
zonecfg:workzone1> set pool=<pool name>
zonecfg:workzone1> verify
zonecfg:workzone1> exit
 
global# zonecfg -z workzone1
zonecfg:workzone1> set pool=work1-pool
zonecfg:workzone1> verify
zonecfg:workzone1> exit
global#
global# zonecfg -z workzone2
zonecfg:workzone2> set pool=work2-pool
zonecfg:workzone2> verify
zonecfg:workzone2> exit
global# 
global# zonecfg -z workzone3
zonecfg:workzone3> set pool=work3-pool
zonecfg:workzone3> verify
zonecfg:workzone3> exit
global# 
global# zonecfg -z workzone4
zonecfg:workzone4> set pool=work4-pool     
zonecfg:workzone4> verify
zonecfg:workzone4> exit
global# 

Verify the zone configuration has been updated by using zonecfg -z <zonename> info. This will display the updated configuration with the pool assignment displayed.

global# zonecfg -z workzone3 info
zonepath: /zones/workzone3
autoboot: true
pool: work3-pool
inherit-pkg-dir:
        dir: /lib
inherit-pkg-dir:
        dir: /platform
inherit-pkg-dir:
        dir: /sbin
inherit-pkg-dir:
        dir: /usr
inherit-pkg-dir:
        dir: /opt/sfw
net:
        address: 192.168.3.44/24
        physical: bge0
global# 
Note: You may also temporarily bind a pool and a processor set using poolbind command with the zoneid as the idtype.
global# poolbind -p work1-pool -i zoneid workzone1
6 Reboot Zones Use zoneadm with the reboot option to reboot the zone.
# zoneadm -z <zone name> reboot
 
global# zoneadm -z workzone1 reboot
global#
global# zonecfg -z workzone2 reboot
global#
global# zonecfg -z workzone3 reboot
global#
global# zonecfg -z workzone4 reboot
global# 

After the zone is running, log into the zone and verify that the number of processor are indeed enabled by using the "psrinfo -vp"

workzone1# psrinfo -vp
The physical processor has 1 virtual processor (0)
  x86 (AuthenticAMD family 15 model 5 step 1 clock 2193 MHz)
        AMD Opteron(tm) Processor 248
The physical processor has 1 virtual processor (1)
  x86 (AuthenticAMD family 15 model 5 step 1 clock 2193 MHz)
        AMD Opteron(tm) Processor 248
The physical processor has 1 virtual processor (2)
  x86 (AuthenticAMD family 15 model 5 step 1 clock 2193 MHz)
        AMD Opteron(tm) Processor 248
workzone1# 
Click for more info