Ramblings of this guy you know!

Tech Stuff and random observations on life as I see it….

Misadventures in Computing – failed JS21 update to RHEL6

My team and i were set the task of trying to upgrade our IBM JS21 blades to Red Hat 6. To be honest it was always touch and go right from the start if the blades were even capable of going to version 6 but as it turns out, the processor can’t take it. Red Hat had cranked up the minimum spec and if we had JS22’s it would have installed, but it was JS21’s we have. Nonetheless we opted to have a go.

Have you ever used a headless blade before? Ever tried to use Serial over Ethernet before? This was the problem we came up against last year when we first got the blades and trying to find out information on using JS21‘s does not yield many results. Thank goodness for Ryan’s tech notes as his helped us get started. I just wanted to share them here again as another place a struggling SysAdmin might find it. The following is an extract of his article on How to get Non-Graphical Console Access to a BladeCenter Server.

Login to your BladeCenter Advanced Management Module:
http://yourBladeCenter or https://yourBladeCenter

  • Navigate to Serial Over LAN (BladeTasks -> Serial Over LAN)
  • Enable SOL for your specific server
  • Locate the section “Serial Over LAN Status”. It should show a table of your bays and blade servers.
  • Check the server(s) for which you’d like to enable SOL
  • Click Enable Serial Over LAN link
  • Enable Serial Over LAN service by using the drop down box under Serial Over LAN Configuration
  • Click Save

Ensure that either Telnet or SSH is enabled.

To enable SSH (the more secure option)

  • Navigate to MM Control -> Security
  • Enable SSH Server using the drop down box
  • Select Save

To enable Telnet (the less secure option)

  • Navigate to MM Control -> Network Protocols
  • Scroll down to Telnet Protocol
  • Enable Telnet mode using the drop down box
  • Save

You should be good to go. Note: SSH and Telnet can be assigned to non-standard ports in the BladeCenter. This can be checked under MM Control -> Port Assignments.How to Connect to the Console

SSH or Telnet into your BladeCenter

  • Set the environment variable to your blade bay number, then type console. If you don’t know your bay number, log into the BladeCenter GUI, http://yourBladeCenter. An example for connecting to the blade on bay 1:
  • system> env -T system:blade[1]
  • system:blade[1]> console
  • Console login:

To escape back into the BladeCenter CLI, use the escape sequence listed under SOL in the Blade Center GUI. The default sequence is ^[(. That is Ctrl+[ and then Shift+(

So now we had access to the console and the IBM toolkit in the DVD. The original IBM toolkit for Power machines could not cope with the version number we were trying to install. Version 4 only works up to RHEL 5 and all it’s subversions so went looking for an upgrade version 4.2 was available so we downloaded that. Success,we can now toolkit install version 6.

Unfortunately the toolkit won’t run properly on the JS21 blade and doesn’t properly build the RAMdisk needed to bootstrap the files. Each time we tried it, looking for why it was failing took up a significant portion of time as the bladecenter we have is served by a central USB DVD drive which it seems only supports USB1.

So we tried ignoring the toolkit installer and looked to install the IBM goodies later. The installer loaded so far, loaded some files from DVD and then failed saying that it couldn’t find the DVD to install the media from. Then the worst bit happened, the open firmware got corrupted so the DVD couldn’t see the disk it was trying to install on and no matter what coaxing we tried, we just couldn’t see the disk.

We tried reviving the old version of RHEL via the old toolkit installed but still no success. No disk found.

After the failed boot you get thrown into an OpenFirmware prompt. Ever been there? It’s a little bit like engaging your hyperdrive and instead of appearing half way across the galaxy you are in a big fat nothing.

GOOGLE!!! HELP!!!

If you ever end up there and looking for a disks mount point you can navigate around using these commands

Once at an Open Firmware prompt, to display current device aliases use the devalias command. For example:


0 > devalias 
ibm,sp              /vdevice/IBM,sp@4000
disk                /pci@800000020000002/pci@2,4/pci1069,b166@1/scsi@1/sd@5,0
network             /pci@800000020000002/pci@2/ethernet@1
net                 /pci@800000020000002/pci@2/ethernet@1
network1            /pci@800000020000002/pci@2/ethernet@1,1
scsi                /pci@800000020000002/pci@2,4/pci1069,b166@1/scsi@0
nvram               /vdevice/nvram@4002
rtc                 /vdevice/rtc@4001
screen              /vdevice/vty@30000000
 ok

To display the current boot device order, use the printenv command. For example:


0 > printenv boot-device 
-------------- Partition: common -------- Signature: 0x70 ---------------
boot-device              /pci@800000020000002/pci@2,3/ide@1/disk@0 /pci@800000020000002/pci@2,4/pci1069,b166@1/scsi@1/sd@5,0 
 ok

To add the device with alias network as the first boot device, use the setenv command. For example:


0 > setenv boot-device network /pci@800000020000002/pci@2,3/ide@1/disk@0 /pci@800000020000002/pci@2,4/pci1069,b166@1/scsi@1/sd@5,0

Your system is now configured to boot off of the device with alias network as the first boot device. Should booting off this device fail, your system will fallback to the next device listed in the boot-device Open Firmware settings.

Resetting the boot point for the blade didnt make any difference so it was back to the web for another solution. Another Google search reveals the suggestion that resetting the NVRAM may help. Again Ryan’s tech blog helps here…

First we tried to reset the NVRAM through the IBM management bladecenter management tool:

  • BladeCenter AMM > Blade Tasks > Power/Restart. Look on the bottom of the page.
  • Click Restart Blade and clear NVRAM

That didn’t help… OK resort to the severe resort of manually resetting the NVRAM:

  • Power off the server. It should be in the Power/Restart screen of your Blade Center.
  • Remove the battery from the server for five minutes. Slide out the blade, open the server, attach an anti-static wristband (if you worry about these things), and remove the nickel-sized battery. A small flathead screwdriver would be useful.

Finally we have a working system back but a retry of RHEL6 results in repeating the NVRAM reset. Finally, in defeat we resorted to RHEL5.5 on the blade. It went on sweet as anything and installed with no issues. That was server 1.

Server 2 should have been simple, a re-install of RHEL5 is all that was required. Even that didn’t go smooth as time and again the message that it couldn’t find the DVD half way through the install. In the end, in desperation, we swapped the bay over with the one that worked. Can you believe that it actually worked? The same happened with servers 3 and 4… That Bladecenter is in for a kicking… Sorry, reboot at the next available point.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: