Specing/Pricing out a home ESXi 5.5 box?

[Darren Dirt]

I understood every word in the above post.

Can't say that about most posts in this thread. 'Swhy I used Brick to express myself.


Carry on...

[Tom]

I understood every word in the above post.

Can't say that about most posts in this thread. 'Swhy I used Brick to express myself.


Carry on...

This is a secret, but I'm not a system admin, I'm a lowly code monkey.

Also, server pretty much purchased. It'll take a while for things to arrive however. The joys of ebay. At least a week till I have everything, if not two+. I'm hoping for 1 week.

I also decided to downgrade the mobo slightly to the H8DGi-F that is pretty much identical to the H8DG6-F, except it's missing the built in LSI chip. I ended up saving a couple hundred dollars with that decision. I have an IBM M1015 I can use for the main storage so it's not an issue at all.

Went with a Fractal Design XL R2 case, and a Seasonic X 850W psu. I /almost/ went with a nice seasonic case, they are so tempting, but I wanted a few more drive bays, and the only seasonic cases I could justify were limited to 8 Other options included el-cheapo 4U cases, but the reviews on Norco's and especially Rosewill's are just awful.

[Lazybones]

HyperV 2012 R2 is great actually either as part of a domain or workgroup.  But even if you strip the GUI out of it might as well attach it to a AD if you have one.

I found the trick was to think of them interms of Windows not VMware.  I've had to do the same thing with XenServer; think of it as a Linux distro rather than VMware or HyperV.  They are all very different in management interfacing and configuration.  In the Windows world workgroup means very little kerberos authentication so your management functions are limited to that which does not utilize kerberos - live migrations for example.  Similar problem to if you had VMware without a VMotion license and vCenter to manage.  And XenServer... well its a beast all on its own in that regard.

So yeah while I still say VMware is the #1 for sure, HyperV isn't that terrible anymore with 2012 R2... its actually production ready.  Anyone who started with HyperV 2008 rather... well I can see where the hatred would come from /badtasteinmouth

I couldn't even get it stood up reasonably in work-group mode in my home green field, that is why I asked... I found the permissions to be awkward without a domain...  The fast majority of my home devices are not domain join-able so I no longer put the effort into running a domain in my home network.

[Tom]

Here's a question, can ESXi deal with drives brought in as individual devices (jbod mode), and then have the main guest then build a sw raid5, and host images or logical volumes to the rest of the guests? Am I stuck with iSCSI when trying to do that? Or does ESXi allow the main guest to provide devices to other guests like virtually every other virtualization solution? I would prefer the data not go over the guest's internal networking.

Something makes me think that ESXi doesn't have the concept of a "main" guest, and that it has a very limited mini os built into the hypervisor instead of being a shim under a more capable OS (like Xen).

I'm googling, but so far haven't found a direct answer.

[Lazybones]

ESXi is as close to bare metal as you get... I haven't tried to setup a stand alone server without vCenter so I can't say... Maybe melbosa can...

In ESXi you can use RAW device maps... The guest VMs can own the disks... Hell you can put ESXi on a USB stick leaving ALL internal drives up to the VMs.

Things have changed in terms of management with 5.5 so that is why I am not sure what happens in a small lab setup.

[Tom]

Yeah, you can hand off disks as RDM, or as individual VMDK pools to a single guest, but then you have to somehow give access to those to other guests, which seems to be iSCSI or NFS over the guests networking stacks. That's a bit of overhead I'd rather not have.

[Lazybones]

Yeah, you can hand off disks as RDM, or as individual VMDK pools to a single guest, but then you have to somehow give access to those to other guests, which seems to be iSCSI or NFS over the guests networking stacks. That's a bit of overhead I'd rather not have.

The more proper way to do it is with a supported hardware RAID controller, then use virtual drives... This is one of the KEY reasons I made my cheap home setup under KVM.. I wanted software RAID.

I believe in Hyper-V free you would also get the option of FREE host software RAID.

[Tom]

Yeah, I'm starting to think I may just go with KVM in the end. I was looking at vSphere pricing and the only license I could conceivably see myself paying for is the Essentials. Though that is missing some useful bits that are only in Essentials Plus which is like 2-3x more expensive.

Hyper-V might be interesting to play with, but my desire to admin a windows box is pretty low.

My M1015 is a supported RAID controller afaik, but it's RAID5 is locked out with a hard to find "RAID5 KEY". It's annoying. It can do JBOD, 0, 1, and 10 without additional hardware. If I can find a better used RAID card i might go that route. We'll see.

And with KVM I get some basic (almost?) live migration to/from my older box, so long as I stick to the generic Qemu cpu config. Of course that needs iSCSI or NFS regardless. so maybe I wont make that a requirement hahaha.

[Melbosa]

ESXi does not have any type of software RAID solution in the host itself.  As Lazy said you can use a USB Stick or Flash Drive for the OS and save the disks completely for your RAID but you will need a RAID Controller to do so (and one that is compatible with the ESXi 5.5 build).  Your other solution is as you said, VM to RDM with Software RAID and then iSCSI or NFS... and just a side NOTE: latest version of ESXi 5.5 U1 which fixes the heartbleed bug has a known issue with random NFS disconnects.  Nice thing with the RDM solution for you is all the iSCSI or NFS traffic will never touch the Network Layer, it will all be handled by the hypervisor Disk and Memory bus speeds as the traffic will never leave the virtual switch.

Managing an ESXi 5.5 box without vCenter is still possible with just the vSphere Client (free) but you will find you will be doing a lot of esxcli or esx powershell commands still.

Hyper-V will give you a software RAID solution which is actually very well implemented now in the 2012 versions (even better in 2012 R2).  And again if you have to own a Retail version of Windows Server 2012 R2 I would just install that as your hypervisor (and then install your 1-2 Windows Server VMs you get with that license in Hyper-V guest) rather than the free Hyper-V if you go with a Microsoft Hyper V solution - which I believe you are NOT.  But thought I would point that out.  Only reason to go Hyper-V Free editiion is if you a) are not running any Windows Server guest OSs  OR  b) your Windows Server guest OSs are older than the current free version of Hyper-V.

[Tom]

ESXi does not have any type of software RAID solution in the host itself.  As Lazy said you can use a USB Stick or Flash Drive for the OS and save the disks completely for your RAID but you will need a RAID Controller to do so (and one that is compatible with the ESXi 5.5 build).  Your other solution is as you said, VM to RDM with Software RAID and then iSCSI or NFS... and just a side NOTE: latest version of ESXi 5.5 U1 which fixes the heartbleed bug has a known issue with random NFS disconnects.  Nice thing with the RDM solution for you is all the iSCSI or NFS traffic will never touch the Network Layer, it will all be handled by the hypervisor Disk and Memory bus speeds as the traffic will never leave the virtual switch.

It's still a switch, which means it hits the guest's own network stack. Or does vmware have a custom iSCSI driver that bypasses the network stack?

Managing an ESXi 5.5 box without vCenter is still possible with just the vSphere Client (free) but you will find you will be doing a lot of esxcli or esx powershell commands still.

I thought the free client was limited to 60 days?

Hyper-V will give you a software RAID solution which is actually very well implemented now in the 2012 versions (even better in 2012 R2).  And again if you have to own a Retail version of Windows Server 2012 R2 I would just install that as your hypervisor (and then install your 1-2 Windows Server VMs you get with that license in Hyper-V guest) rather than the free Hyper-V if you go with a Microsoft Hyper V solution - which I believe you are NOT.  But thought I would point that out.  Only reason to go Hyper-V Free editiion is if you a) are not running any Windows Server guest OSs  OR  b) your Windows Server guest OSs are older than the current free version of Hyper-V.

I have no expectation of running windows server instances. My preference is anything but Microsoft haha.

What would you say is the main plus going for Hyper-V over ESXi, or KVM? Especially for a linux person? I for sure will have one windows guest, maybe more, but no idea for sure yet.

The main positive I can see, is I get to learn something new. Get my fingers back into the windows server side of things (I took a course back in the day and learned all about NT 4! even though 2k+ was already out, what a crock that was).

[Lazybones]

What would you say is the main plus going for Hyper-V over ESXi, or KVM? Especially for a linux person?

I would honestly say for you KVM would be a better solution than Hyper-V or ESX...

If I was going to change one thing in my home KVM setup it would be to build it off of a Redhat based distro so that I could use one of the MANY Web management interfaces.

A stand alone Hyper-V free machine is going to require lots of windows specific power shell or a windows box to run the management tools.
ESXi 5.5 without a vCenter (web ui) will require running the windows management client, which for you also doesn't work.

[Tom]

It is starting to look like KVM is my best bet if I want something that "just works" for me. I typically use virt-manager to manage my existing KVM box, and it works rather well these days. IIRC its a standalone python GTK app. It used to be pretty buggy, but seems to work alirght now.

The two main reasons I wanted to use ESXi was that it should have better performance, and I wanted to learn and play with it. But it doesn't seem like they care much about that use case at all.

I can run windows apps in a VBox windows install I have, but it's a bit annoying to boot it up. Also I read that the standalone ESX app doesn't work on the new format virtual machines. You have to somehow create v8 or v9 vms.

[Melbosa]

ESXi does not have any type of software RAID solution in the host itself.  As Lazy said you can use a USB Stick or Flash Drive for the OS and save the disks completely for your RAID but you will need a RAID Controller to do so (and one that is compatible with the ESXi 5.5 build).  Your other solution is as you said, VM to RDM with Software RAID and then iSCSI or NFS... and just a side NOTE: latest version of ESXi 5.5 U1 which fixes the heartbleed bug has a known issue with random NFS disconnects.  Nice thing with the RDM solution for you is all the iSCSI or NFS traffic will never touch the Network Layer, it will all be handled by the hypervisor Disk and Memory bus speeds as the traffic will never leave the virtual switch.

It's still a switch, which means it hits the guest's own network stack. Or does vmware have a custom iSCSI driver that bypasses the network stack?

Yes but all the network IO is done in memory so it is still faster than an actual physical switch.  Performance proven if you have guest OSs talking to each other (no matter what the protocol) on the same vSwitch on the Same Host, over a physical device (at least VMware has proven it on their hypervisor).
Managing an ESXi 5.5 box without vCenter is still possible with just the vSphere Client (free) but you will find you will be doing a lot of esxcli or esx powershell commands still.

I thought the free client was limited to 60 days?

I have no expectation of running windows server instances. My preference is anything but Microsoft haha.

What would you say is the main plus going for Hyper-V over ESXi, or KVM? Especially for a linux person? I for sure will have one windows guest, maybe more, but no idea for sure yet.

Licensing costs if you were having to license Windows Guest OSs and heterogeneous compatibility between hypervisor and guest OS (in the Microsoft case).  As I said for you probably NOT the right choice.

[Melbosa]

Managing an ESXi 5.5 box without vCenter is still possible with just the vSphere Client (free) but you will find you will be doing a lot of esxcli or esx powershell commands still.

I thought the free client was limited to 60 days?

No ESXi Eval License is 60 days, so is the vCenter Eval.  The vSphere Management Client software is free, you just point it directly at the host IP and use root creds.  No vCenter or Web GUI required (yet - looks like the vSphere Client will be phased out in the next couple releases).

[Tom]

Yay. Server is built, and i'm running a cpu stress test. Things are going well so far.

Had to re-mount the cpu fans cause they were on backwards. And I added two fans on top, and one on the front cause temps were initially hitting 70c (not actually 70c, but the k10temp sensors arbitrary scale value, which is actually at or over these processor's max operating temp). Now that I have everything set up, temps are down to 40c so far. It seems to be rising ever so slowly, so i'm not sure where it'll top out quite yet.

It is VERY silent, especially for a dual socket server. It's amazing the difference slower spinning much larger fans make.