***UPDATE 3*** A press release just popped up stating: “MTI also announced today that, due primarily to continued operational and financial difficulties experienced by its U.S. operations, it has filed for bankruptcy protection pursuant to Chapter 11 of the U.S. Bankruptcy Code.” Bye bye MTI. 😦
My company has used MTI Technology OTC:MTIC.pk for some EMC storage and VMware projects. There have been signs that they were in trouble since May. I got a letter from their CEO in May stating that some sales people had left, but everything was fine and they were still the #2 EMC reseller. That was weird, so I started looking around and found out that MTI was being de-listed from NASDAQ. I was then contacted by some of the departed sales people who wanted me to do business with them at their new company. Then more and more people that I dealt with from MTI left. Finally, I wasn’t getting calls returned from MTI employees I know. I eventually got in touch with someone I know that used to work at MTI. That person told me that “no one really works for MTI anymore.”
My understanding is that MTI has let go of almost everyone and they are on their way out. I’m not totally surprised. MTI was an IHV that attempted to transform itself into a VAR. They sold only EMC Storage. They also sold EMC and VMware software. They did a decent job with the EMC storage, eventually becoming the #2 EMC reseller behind only Dell. They always behaved as if Sales and Project Services were different organizations. They never behaved as if they were on the same team, which caused them unnecessary problems.
It’s unfortunate to see MTI go.
*UPDATE* It appears that my info and instincts are correct. MTI defaulted on a loan from Comerica and a prommisory note from Pencom Systems Inc..
*UPDATE 2* One of my sources confirmed what I learned last week. They said that all sales and project people were let go. The also said some support staff remains to deal with commitments. They said that MTI will try to do the right thing for their customers. That remains to be seen.
This post is based on information I have received from people associated with MTI that are in the know and my opinion of their services. If you have information that proves MTI is staying in business, I’ll gladly retract this post.
I came across this article from Network Computing’s Howard Marks. He writes about a two new study’s from Carnegie Mellon and Google about hard drive reliability. The short story is that hard drives die in bunches and the massive sizes of SATA and other drives put organizations at risk of data loss from multiple disk failures. Howard Marks recommends using RAID 6 for all drives 500GB and larger. It’s a worthwhile read for storage managers. Enjoy
The Truth About Storage Reliability
RAID 6 Primer
UPDATE: I called EMC today to ask if they support RAID 6 on any of their arrays. I couldn’t find it in documentation, and I felt it was worth asking. They said that do no not support it at all, and there is no known plan to support it.
UPDATE 2: Things have changed since I orignially posted this. The Clariion CX-xxx series still doesn’t support RAID 6. The CX3-xx series does support RAID 6. Thanks to Dr. Product for pointing this out.
I have been planning my company’s ESX upgrade for a while. After many delays and other conflicts, I was able to schedule it for this past weekend. I want to braindump everything I learned if possible. It’s a bit of a mish-mosh, but so is my brain.
- Plan, Document, Plan, and Document: There are so many moving parts that you’re going to want to document EVERYTHING. The upgrade is not difficult, but it is tricky.
- Be prepared for your Virtual Center upgrade to go bad. This is the only in place upgrade that you cannot avoid and it’s the least reliable. Have a backup plan whether it’s restoring the database or wiping it and starting clean. Make a decision in advance.
- If you lose VC you lose stats, permissions, VM groups, and a few other things. Document all of VC at minimum (if possible).
- VMware says you need 1200 MB of free disk space on your LUN’s. This is not enough. I had at least 2 gigs and still ran into problems.
- The VM Hardware upgrade moves VM configuration files from the ESX server to the SAN. One of these files is the VM swap file. The swap file is twice the size of the VM’s memory. Reducing the assigned memory increases free space on the LUN. This helps with insufficient disk errors at boot up.
- You can’t suspend a VM if you don’t have enough disk space.
- Rebooting the ESX servers seems to clear up “Object” errors.
- VMotion: You have to license it, set up the virtual switch as vmkernel, AND enable VMotion on the port.
- WinSCP is a great program.
- You MUST upgrade Hardware on all VM’s before putting them in a cluster. This makes sense, but isn’t obvious.
- Test as much of your upgrade as possible in advance. This helped me tremendously.
- Make sure that your VMFS2 LUN’s are formatted at 8MB block size or less. ESX cannot upgrade LUN’s that are formatted with anything larger than 8MB block size. The two LUN’s I used as backup were both formatted with 16 MB block sizes. I knew the limitation, but I didn’t think it affected me because I always used the default block size. The only thing that’s strange about them is that they are both 1.7TB.
- “unable to upgrade filesystem” + “function not implemented” errors come from the wrong block size on the VMFS2 partition.
- Renaming datastores is not destructive in ESX 3, but I wouldn’t recommend doing this until all VM’s are functional.
- The upgrade is a good chance to upgrade server firmware.
- Make sure all VMDK files are connected before upgrading Virtual Hardware. Otherwise you will get errors about disk version mismatches. I used the recommended resolution. I’m not confident that I did the right thing.
- Invalid affinity entry errors will happen if you assign a processor or memory affinity to a server and then move it to a server that cannot fulfill the entry. This could happen if you move a VM from a quad proc. server to a dual and set processor 4 as the affinity. Best way to fix this is remove the affinity. Second best way is to recreate the VM using the same disk files. (Remove from inventory, recreate.)
- Network Copy Failed for File.  /root/vmware/<servername>/nvram error is most likely a DNS problem. Make sure to register all possible DNS names in the hosts file of each server involved. In my case, the registered name and FQDN was different. More info can be found here.
- If there are yellow or red alarms on most VM’s after Virtual Center 2 upgrade: The upgrade sometimes truncates records including the alarm thresholds. It will truncate 70% and 90% to 7% and 9%. VC looks like a Christmas tree the first time you log in. Your options are bad and worse in this case. I chose to wipe the old DB and create a new one. The stats were not critical to us. Doing this also affects rights, groups, and other things.
- “The virtual machine is not supported on the target datastore.” Rebooting solves lots of problems during the install. Especially this one.
- VMware Tools for Netware. I need to address this in a seperate post, but the answer is that the only instructions for this are old GSX 3.2 instructions. They work.
Sorry about the disorganized info, but this is just a braindump. Please let me know if you have any questions and I will get you more detailed info.
My company is installing a second SAN and VMware environment. Our integrator is a California based company that used to make their own storage hardware. They’re tight with EMC. I like them. They have worked hard for us, though there have been some rough spots.
We used these guys for our first VM-SAN installation. They did something dopey by forgetting to put SnapView in the Statement of Work (SOW). I have had to chase them down to finish the installation. It’s a year later and it’s still not done. One big reason is that we haven’t needed it and another is we were both procrastinating. So this new project has come around and everyone is being extremely thorough.
My company needs to begin installation of this new environment next week. My boss, their PM’s, and me have been trying to nail down the SOW all week. We were getting to late in the day Friday and they were sending messages saying that we needed to sign the SOW. My boss and I felt good about the SOW overall and we’re ready to sign. My boss signs the SOW and fax’s it to them. Three minutes later they send a message saying “I guess we scared him into signing. Another one for the good team.” Only problem for them was that they forgot to remove me from the “team” email.
I feel a little insulted by this. They didn’t scare us into anything. We were working hard to get it correct. We are getting everything we want; not out of fear and coercion. I know it’s naive, but I feel that they are our partner and that our success is their success. Taking care of us should be #1 for them.
Why am I sharing this? Probably because I’m annoyed. I’ll have to talk to my “teammates” on Monday.