Razor, Puppet, Hadoop and the CSC Cloud (pt2)

Using Razor to Boot the CSC Cloud (Part 2):

Step 3: Configuring Razor

The Image Catalog

Running razor post installation should show a single OS Image:

dhushon@razor:/opt/razor/conf$ sudo razor image get all
UUID =>  390R2duPsjqKzwmjCCP0LJ
Type =>  MicroKernel Image
ISO Filename =>  razor-microkernel-latest.iso
Path =>  /opt/razor/image/mk/390R2duPsjqKzwmjCCP0LJ
Status =>  Valid
Version =>  0.12.0
Built Time =>  Wed Feb 20 17:10:45 -0500 2013

We now download the Centos 6.4_x86_64 image from http://isoredirect.centos.org/centos/6/isos/x86_64/ and copy it to our razor host

add it to the razor catalog

dhushon@razor:~$ sudo razor image add -t os -p CentOS-6.4-x86_64-minimal.iso -n CentOS6.4x64-minimal -v 6.4x64

now we are showing the new image in the catalog:

dhushon@razor:/opt/razor/conf$ sudo razor image get all
UUID =>  2101TWCg8hOgDJvqoKl6W1
Type =>  OS Install
ISO Filename =>  CentOS-6.4-x86_64-minimal.iso
Path =>  /opt/razor/image/os/2101TWCg8hOgDJvqoKl6W1
Status =>  Valid
OS Name =>  CentOS6.4x64-minimal
OS Version =>  6.4x64
UUID =>  390R2duPsjqKzwmjCCP0LJ
Type =>  MicroKernel Image
ISO Filename =>  razor-microkernel-latest.iso
Path =>  /opt/razor/image/mk/390R2duPsjqKzwmjCCP0LJ
Status =>  Valid
Version =>  0.12.0
Built Time =>  Wed Feb 20 17:10:45 -0500 2013

Building a [Deployment] Model

Now, we must build a model (an association of a behavior with an image).  First, let’s see what templates we have:

dhushon@razor:/opt/razor/conf$ sudo razor model get template
Model Templates:
Template Name           Description
ubuntu_precise          Ubuntu Precise Model
redhat_6                RedHat 6 Model
xenserver_boston        Citrix XenServer 6.0 (boston) Deployment
ubuntu_oneiric          Ubuntu Oneiric Model
oraclelinux_6           Oracle Linux 6 Model
opensuse_12             OpenSuSE Suse 12 Model
debian_wheezy           Debian Wheezy Model
ubuntu_precise_ip_pool  Ubuntu Precise Model (IP Pool)
vmware_esxi_5           VMware ESXi 5 Deployment
sles_11                 SLES 11 Model
xenserver_tampa         Citrix XenServer 6.1 (tampa) Deployment
centos_6                CentOS 6 Model

For this run, we want to go ahead and associate centos_6 model with our iso. You will see, in the example below, that the model requires some configuration information in order to complete; things like hostname prefix, domain name and password.  Nothing big, but supports the kickstart/anaconda styled installation customization, without having to custom edit a kickstart file.

dhushon@razor:/opt/razor/conf$ sudo razor model add --template centos_6 --label install_centos6.4x64 --image-uuid 2101TWCg8hOgDJvqoKl6W1
--- Building Model (centos_6):
Please enter node hostname prefix (will append node number) (example: node)
default: node
(QUIT to cancel)
Please enter root password (> 8 characters) (example: P@ssword!)
default: test1234
(QUIT to cancel)
Please enter local domain name (will be used in /etc/hosts file) (example: example.com)
default: localdomain
(QUIT to cancel)
> mgt.cto.csc.com
Model created
Label =>  install_centos6.4x64
Template =>  linux_deploy
Description =>  CentOS 6 Model
UUID =>  2SLXLNkCw444imPJLwrdNR
Image UUID =>  2101TWCg8hOgDJvqoKl6W1

Step 4: Booting Nodes

Build a vApp for the new project

To make this set of steps just a little easier, I will build just two nodes, a small memory Job Tracker, and a large memory Name Node.  There are quite a few additional capabilities required to build a successful and available cluster, and we’ll target those in the next set of instructions.

Firstly, this lab is running vCD so the images may seem “familiar” you’ll see in my initial screenshots.

vCD Login Screen via a Secure Network Tunnel

There are 2 initial vApps, the first is Razor, which contains the Razor, Puppet, dnsmasq and other shared services.  The second is the new vApp defined to represent the emergent Hadoop cluster.

The 2 vApps are defined
the hadoop namenode and job tracker are added as VM’s in the vDC

Customize Concrete Instances

We must now create two different vm templates, a big memory one for the namenode and a small memory vm for the job tracker and other shared services. Caveat, to save some resources these nodes are under provisioned, but the different memory configurations allows us to treat these nodes differently from a bootstrap and a puppet perspective downstream.

Note the 8GB memory footprint for the namenode template

Power On

If everything works as planned, you should see a node show up, something like this:

dhushon@razor:/opt/razor/conf$ sudo razor node get
Discovered Nodes
UUID                 Last Checkin  Status                           Tags
5pWaIemUq6n4T3lnXl18Ph  0 sec         A [memsize_8GiB,nics_3,vmware_vm,IntelCorporation,cpus_2]

Here we see that the first node is now registered in Razor, and is waiting (Status “A”) and is being held in a wait state.  Once we apply a policy and update the razor state machine, we’ll see that this node will get to work.

Step 5: Razor Administration

Now that the image and model are built, the nodes enabled (my nodes are virtual and part of a CSC vDC), and the node profiled via the iPXE/microKernel procedure, we’re now ready to assign the useful tags to this node using a tag-matcher, and policy to associate the model with a tagged node.

Setting up a Tag

Starting out tarting out, we see that we have no tags:

dhushon@razor:/opt/razor/conf$ sudo razor tag
Tag Rules
< none >

First we add a new tag, in our case, we want to create our Hadoop NameNodes, so we need a “Big Memory” variety:

dhushon@razor:/opt/razor/conf$ sudo razor tag add --name BigMem --tag bigmem
Name =>  BigMem
Tags =>  bigmem
UUID =>  VwPf7QuDMCow4nfWJpt6v
Matcher =>  <none>

and to provide selection, we need a tag matcher, basically a predicate that enables the differentiation of different classes of nodes.  Being in a vDC this allows us to match capability & cost to reduce waste.

sudo razor tag 6TmVAekfc2tIbYHaOn8Ikb matcher add --key mk_hw_mem_size --compare 'equal' --value 8GiB
Tag Rule:
Key =>  mk_hw_mem_size
Compare =>  equal
Value =>  8GiB
Inverse =>  false

If you were wondering, the “–key” gives you a ton of selection capability, and the keys can be simply found here (really amazing in memory microkernel work):

dhushon@razor:/opt/razor/conf$ sudo razor node 5pWaIemUq6n4T3lnXl18Ph get --field attributes
Node Attributes:
Name                      Value
architecture              i386
boardmanufacturer         Intel Corporation
boardserialnumber         None
domain                    mgt.cto.csc.com
fqdn                      mk0050561C01F6.mgt.cto.csc.com
hardwareisa               unknown
hardwaremodel             i686
hostname                  mk0050561C01F6
interfaces                dummy0,eth0,eth1,eth2,lo
is_virtual                true
macaddress                3E:01:BB:7E:8E:07
macaddress_dummy0         3E:01:BB:7E:8E:07
macaddress_eth0           00:50:56:1C:01:F6
macaddress_eth1           00:50:56:1C:01:F7
macaddress_eth2           00:50:56:1C:01:FA
manufacturer              VMware, Inc.
memorysize                2.96 GB
memorytotal               2.96 GB
mk_hw_bus_description     Motherboard
mk_hw_bus_physical_id     0
mk_hw_bus_serial          None
mk_hw_bus_vendor          Intel Corporation
mk_hw_bus_version         None
mk_hw_cpu0_bus_info       cpu@0
mk_hw_cpu0_capacity       4230MHz
mk_hw_cpu0_description    CPU
mk_hw_cpu0_physical_id    4
mk_hw_cpu0_serial         0002-06C2-0000-0000-0000-0000
mk_hw_cpu0_size           3333MHz
mk_hw_cpu0_slot           CPU socket #0
mk_hw_cpu0_vendor         Intel Corp.
mk_hw_cpu0_version        6.12.2
mk_hw_cpu0_width          64 bits
mk_hw_cpu1_bus_info       cpu@1
mk_hw_cpu1_capacity       4230MHz
mk_hw_cpu1_description    CPU
mk_hw_cpu1_physical_id    5
mk_hw_cpu1_product        Pentium Pro
mk_hw_cpu1_serial         0002-06C2-0000-0000-0000-0000
mk_hw_cpu1_size           3333MHz
mk_hw_cpu1_slot           CPU socket #1
mk_hw_cpu1_vendor         GenuineIntel
mk_hw_cpu1_version        6.12.2
mk_hw_cpu_count           2
mk_hw_disk0_bus_info      scsi@2:0.0.0
mk_hw_disk0_description   SCSI Disk
mk_hw_disk0_logical_name  /dev/sda
mk_hw_disk0_physical_id   0.0.0
mk_hw_disk0_size          24GiB (25GB)
mk_hw_disk_count          1
mk_hw_fw_date             07/12/2011
mk_hw_fw_description      BIOS
mk_hw_fw_physical_id      0
mk_hw_fw_size             99KiB
mk_hw_fw_vendor           Phoenix Technologies LTD
mk_hw_fw_version          6.00
mk_hw_lscpu_Architecture  i686
mk_hw_lscpu_BogoMIPS      6652.08
mk_hw_lscpu_Byte_Order    Little Endian
mk_hw_lscpu_CPU_MHz       3324.999
mk_hw_lscpu_CPU_family    6
mk_hw_lscpu_CPU_op-modes  32-bit, 64-bit
mk_hw_lscpu_CPU_sockets   1
mk_hw_lscpu_L1d_cache     32K
mk_hw_lscpu_L1i_cache     32K
mk_hw_lscpu_L2_cache      256K
mk_hw_lscpu_L3_cache      12288K
mk_hw_lscpu_Model         44
mk_hw_lscpu_Stepping      2
mk_hw_lscpu_Vendor_ID     GenuineIntel
mk_hw_mem_description     System Memory
mk_hw_mem_physical_id     82
mk_hw_mem_size            8GiB
mk_hw_mem_slot            System board or motherboard
mk_hw_nic0_bus_info       pci@0000:02:00.0
mk_hw_nic0_capacity       1Gbit/s
mk_hw_nic0_clock          66MHz
mk_hw_nic0_description    Ethernet interface
mk_hw_nic0_logical_name   eth0
mk_hw_nic0_physical_id    0
mk_hw_nic0_serial         00:50:56:1c:01:f6
mk_hw_nic0_size           1Gbit/s
mk_hw_nic0_version        01
mk_hw_nic0_width          64 bits
mk_hw_nic1_bus_info       pci@0000:02:01.0
mk_hw_nic1_capacity       1Gbit/s
mk_hw_nic1_clock          66MHz
mk_hw_nic1_description    Ethernet interface
mk_hw_nic1_logical_name   eth1
mk_hw_nic1_physical_id    1
mk_hw_nic1_serial         00:50:56:1c:01:f7
mk_hw_nic1_size           1Gbit/s
mk_hw_nic1_version        01
mk_hw_nic1_width          64 bits
mk_hw_nic2_bus_info       pci@0000:02:02.0
mk_hw_nic2_capacity       1Gbit/s
mk_hw_nic2_clock          66MHz
mk_hw_nic2_description    Ethernet interface
mk_hw_nic2_logical_name   eth2
mk_hw_nic2_physical_id    2
mk_hw_nic2_serial         00:50:56:1c:01:fa
mk_hw_nic2_size           1Gbit/s
mk_hw_nic2_version        01
mk_hw_nic2_width          64 bits
mk_hw_nic_count           3
mtu_dummy0                1500
mtu_eth0                  1500
mtu_eth1                  1500
mtu_eth2                  1500
mtu_lo                    16436
physicalprocessorcount    1
processorcount            1
productname               VMware Virtual Platform
type                      Other
virtual                   vmware

To check that in fact razor has properly matched the node, notice that our node has picked up a new tag:

dhushon@razor:/opt/razor/conf$ sudo razor node get
Discovered Nodes
UUID           Last Checkin  Status                               Tags
5pWaIemUq6n4T3lnXl18Ph  32.7 min      B       [bigmem,memsize_8GiB,nics_3,vmware_vm,IntelCorporation,cpus_2]

The next step is to create a policy that will associate a model with a tag (in our case the linux_deploy template with our Centos image).  The policy is basically another matcher, this time for a node -> a model via a tag.   In our case we want to use the linux_deploy model and apply it to nodes that match the tag “bigmem”

dhushon@razor:/opt/razor/conf$ sudo razor policy add --template linux_deploy --label NameNode --tags bigmem --model 2SLXLNkCw444imPJLwrdNR
Policy created
UUID =>  5HkjZBIfOpNct7IIWPmZmb
Line Number =>  0
Label =>  NameNode
Enabled =>  false
Template =>  linux_deploy
Description =>  Policy for deploying a Linux-based operating system.
Tags =>  [bigmem]
Model Label =>  install_centos6.4x64
Broker Target =>  none
Currently Bound =>  0
Maximum Bound =>  0
Bound Counter =>  0

One last step, now that we’re comfortable is to enable the Policy (they are disabled by default), and we should see some magic happen (just as soon as the next node checkin happens the new behavior will be pulled down, and the node rebooted to the new pxe image target),

dhushon@razor:/opt/razor/conf$ sudo razor policy update 5HkjZBIfOpNct7IIWPmZmb --enabled true
UUID =>  5HkjZBIfOpNct7IIWPmZmb
Line Number =>  0
Label =>  NameNode
Enabled =>  true
Template =>  linux_deploy
Description =>  Policy for deploying a Linux-based operating system.
Tags =>  [bigmem]
Model Label =>  install_centos6.4x64
Broker Target =>  none
Currently Bound =>  0
Maximum Bound =>  0
Bound Counter =>  0
dhushon@razor:/opt/razor/conf$ sudo razor policy
#  Enabled    Label        Tags         Model Label       #/Max  Counter           UUID
0  true     NameNode    [bigmem]    install_centos6.4x64  0/-    0        5HkjZBIfOpNct7IIWPmZmb
1  false    JobTracker  [smallmem]  install_centos6.4x64  0/-    0        1NMAIlZAtU63GvhiBZadw9

And voila… CentOS 6 running on my vm…. now let’s iterate so that I can get Hadoop fully up!

Centos 6.4 x86_64 node provisioned, note the hostname prefix matches the model that we defined.

Next up, we’ll map a few puppet manifests to do the full Hadoop installation.


Thanks to Peter, Eli, Tom, Nan, Nick, and the following sites (incomplete):


Leave a Reply