We had a boneheaded mini-disaster last week.

Last week we installed a new SAN and 3 new servers to run some new application. One of our guys was responsible for power. He decided to pull some plug because he thought everything was redundant anyway. BIG MISTAKE. This guy pulled the plug on something that was supporting a second power strip. Our primary ESX server and a database server shutdown. Phones went off the hook as expected. The ESX server came up OK, but the database server had issues. Some file locked up and the database wouldn’t start. This is another in a series of boneheaded manmade screw-ups.

I posted that I was insterested in a book called Visual Ops a couple months ago. I am interested in this book because I feel it is a practical guide for changing my department. I offered the book to my boss and he’s not interested. I’m probably going to offer it to another of my bosses. I hope I have more success with him. If not, I will have to lead by example. The only downside is I won’t have a mandate at first. So be it. Right now we are too lax about controls and change management. We constantly repeat dumb mistakes and there isn’t enough accountability for risk taking.

BTW, READ THS BOOK if you are in IT.

Note: I know the power situation is bad if we are using power strips. We are replacing all the racks and UPS within 5 months.

You need a couple special BIOS settings for Proliant servers running ESX 3.

I was upgrading one of our ESX servers (HP dl580 G2) last week and ran into a little problem. I would get Kernel Panic errors everytime the machine booted after install. The exact error was: Mod: 4670: vmkernel not loaded: cannot PSOD.

I did some searching and found this: Required MPS Table BIOS Settings for ESX Server on ProLiant.

I made the recommended changes and my server worked perfectly. I also checked one of my new servers (HP dl585 G2) and found that it is pre-set to Full Table APIC with no OS selection option.

Funny thing is that the dl580 had an ESX problem for about a year that was probably related. That machine had ESX 2.5.2 installed. ESX only detected 1 processor and no hyperthreading. I played around with it, but never fixed it because it was mainly used as a test server. Thankfully it’s all working now.

T-Mobile Blackberry 8800 is going to be available 4/23.

I was talking to someone in the know and I was told that the new Blackberry 8800 for T-Mobile will be released on Monday 4/23. I was also told that the Blackberry 8800 and the Blackberry Crimson are separate devices. I perpetuated the myth that they were the same device based on information I found here and here. I have since been told that the 8800 and the Crimson are not the same. The Crimson will be released this fall with a bunch of new features. The two I know about are GPS and Wifi.  I was also told that the Crimson will have more of a rounded bottom.

The only feature I know about on the 8800 is that Telenav will be added.

The price for the 8800 will be:

• $550 with no contract
• $430 with 1 year contract
• $300 with 2 year contract 

Keep an eye out for the 8800 starting next week. (Monday looks like a hard date.)

The next thing I’m going to play with (unless I get to something else first)

The next thing I plan to play around with is the FreeNAS application:

FreeNAS is a free NAS (Network-Attached Storage) server, supporting: CIFS (samba), FTP, NFS, AFP, RSYNC, iSCSI protocols, S.M.A.R.T., local user authentication, Software RAID (0,1,5) with a Full WEB configuration interface. FreeNAS takes less than 32MB once installed on Compact Flash, hard drive or USB key.
The minimal FreeBSD distribution, Web interface, PHP scripts and documentation are based on M0n0wall.

Our new ESX 3 environment can connect to NAS and I want to see how useful the feature is. This might be a chance to free up some expensive SAN disks.

FreeNAS is based on FreeBSD 6.2. It’s a tiny app that appears to be very powerful. I’ve been watching it grow over the last few months. The developers are doing a good job of improving the application. I’ll let you know how it goes.

When you think you’ve seen it all in IT, you haven’t.

I’ve seen anything and everything in my years working in IT.  I’ve seen things I’ve been told I wouldn’t or couldn’t see such as Token Ring, Novell, and Twinax.  I’ve seen Cisco back-plane failures, UPS failures, and computer voodoo.  And what I saw today is high on the list.

I just ordered a couple of HP dl585 servers.  One was a pre-built system and the other was a config-to-order.  The pre-built system showed up a couple weeks ago.  The box was so banged up that it looked like the server fell out of the truck.  The box looked like moldy swish cheese.  Unfortunately we accepted delivery and were somewhat stuck with it.  We powered it up and it worked perfectly.

Today I received the CTO server.  The box was in good condition.  There was no sign of trouble.  We rack the server and try to boot.  The server refuses to boot, and gives us blinking yellow lights.  We track down the error to Processor 1.  We remove the heat sink and processor, and what do we find?  Bent pins in the processor socket.  The irony of it all was suffocating.

I ordered the replacement parts and should be getting them tomorrow.  This is why I love IT.  IT is like those Choose Your Own Adventure books from when i was a kid.

Greenpeace: Apple is the worst envrionmental offender out there

 I was tipped off to this by InfoWorld.  Greenpeace just concluded their quarterly Toxic Tech rankings of the most environmentally friendly big electronics companies.  Lenovo finished in first place.  Who finished in last?  Apple!  Apple is more toxic than a sink full of dishes in the The Sims.  Check out the rankings and encourage Apple to clean up their act.

 BTW, Apple fanboys shouldn’t forget that Apple is the home of Trash as opposed to Microsoft’s Recycling Bin. :)

Green Grid: Guidelines for Energy Efficient Data Centers

In my experience I have noticed that IT people tend to be environmentally friendly, so this should be right up your alley.  An organization has been created to promote the “Green” datacenter.  It’s lead by the big boys of hardware  and software (HP, IBM, Dell, AMD, Microsoft, amongst others).  Their mission statement is:

The Green Grid is a global consortium dedicated to developing and promoting energy efficiency for data centers and information service delivery by:

• Defining meaningful, user-centric models and metrics
• Developing standards, measurement methods, processes and new technologies to improve performance against the defined metrics
• Promoting the adoption of energy efficient standards, processes, measurements and technologies

They’ve posted some helpful documents:

The Green Grid Opportunity and Guidlines for Energy-Efficient Datacenters

Also check out their website here: http://www.thegreengrid.org 

Tool to convert just about any file type to any other file type. Also convert YouTube video to file.

I found an incredibly useful tool that allows you to convert just about any file type to any other similar file type.  MP3 to AAC, WMV to AVI, DOC to PDF, etc.  This tool also allows you to enter a URL and convert the content to another format.  That means that you can convert any YouTube video to any video format and store it on your machine.  SWEET

http://www.zamzar.com

It’s always amusing dealing with resellers, vendors, manufacturers, and the other assorted shysters.

My company is installing a second SAN and VMware environment. Our integrator is a California based company that used to make their own storage hardware. They’re tight with EMC. I like them. They have worked hard for us, though there have been some rough spots.

We used these guys for our first VM-SAN installation. They did something dopey by forgetting to put SnapView in the Statement of Work (SOW). I have had to chase them down to finish the installation. It’s a year later and it’s still not done. One big reason is that we haven’t needed it and another is we were both procrastinating. So this new project has come around and everyone is being extremely thorough.

My company needs to begin installation of this new environment next week. My boss, their PM’s, and me have been trying to nail down the SOW all week. We were getting to late in the day Friday and they were sending messages saying that we needed to sign the SOW. My boss and I felt good about the SOW overall and we’re ready to sign. My boss signs the SOW and fax’s it to them. Three minutes later they send a message saying “I guess we scared him into signing. Another one for the good team.” Only problem for them was that they forgot to remove me from the “team” email.

I feel a little insulted by this. They didn’t scare us into anything. We were working hard to get it correct. We are getting everything we want; not out of fear and coercion. I know it’s naive, but I feel that they are our partner and that our success is their success. Taking care of us should be #1 for them.

Why am I sharing this? Probably because I’m annoyed. I’ll have to talk to my “teammates” on Monday.

Nomadishere: 8 Things Intelligent People, Geeks and Nerds Need To Work Happily

I really enjoyed this article.  Probably because I agree with the 8 things.

http://nomadishere.com/2007/03/12/a-note-to-employers-8-things-intelligent-people-geeks-and-nerds-need-to-work-happy/