Righteous Wrath Online Community

General => Tech Chat => Topic started by: Tom on May 04, 2014, 04:00:17 PM

Title: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 04, 2014, 04:00:17 PM
Ok, so I'm in need/want of a decent ESXi box, mostly due to it being the only decent virtualization scheme that supports windows and doesn't require a Windows Server license just to run the host. Basically I expect to have at least one windows guest that needs decent io speeds (something that I haven't gotten from KVM/Qemu in the past).

I've looked into some older Core i5/7 based Xeon setups, like a xeon 5500 or 5600 based dual socket boards. They are getting pretty cheap for what you get. Also looked at some modern Opteron C32 based setups which look promising, and some Socket 2011 systems which are so far out of my price range my mind boggles.

Looking for a good price/performance ratio. Absolute modern hardware is not required. I was actually thinking of using this 2u box I bought off ebay for this purpose, as it should do everything I want just fine. But it has one problem. It's loud as all hell.

My current plan is to get some either inexpensive new components (talking in the 1k area for mb + 2x cpus), or some older/used components off ebay for 500-800$.

I know some of you know your stuff when it comes to this area, so I was hoping you'd have some insights.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Lazybones on May 04, 2014, 09:08:11 PM
Vmware compatibility guide
http://www.vmware.com/resources/compatibility/search.php

You might have issues with 5.5 since it really depends on a central server more than 5.1 did.
https://communities.vmware.com/thread/457843
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 12:58:36 PM
If you have to buy a windows server license anyway you might as well go HyperV. I could even set it up for you.  HyperV also does supportlinux distros... I run a few on my HyperV installations.  Your hardware options expand with the HyperV option as well.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 05, 2014, 01:12:23 PM
I don't actually need a windows server license. IIRC you can use a regular retail sku in a virtual env as long as you're not using it anywhere else.

The performance I am seeing in windows on kvm is a lot better than I have seen in the past, so I could go that way as well.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 01:13:11 PM
But what ever you do decide to go with here are my recommendations from both my enterprise and small business experience in the vm worls:
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 01:16:08 PM
Quote from: Tom on May 05, 2014, 01:12:23 PM
I don't actually need a windows server license. IIRC you can use a regular retail sku in a virtual env as long as you're not using it anywhere else.

The performance I am seeing in windows on kvm is a lot better than I have seen in the past, so I could go that way as well.

Yeah that is because your sku is married to your hardware if oem, and limited to moves every 90days if volume licensing.  Trust me, became a M$ licensing guru in the last year.  Why DC licensing can be attractive in large virtual environments.

So you do need one license of some type no matter how you run the windows server.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 05, 2014, 01:18:12 PM
8 nice?

But yes, I'll be slapping a ton of ram in the box. as much as I can without breaking the bank (32-64GB?). I have some older 1TB drives I plan on using for vm storage. 4-5TB is a good starting point.

Quote from: Melbosa on May 05, 2014, 01:16:08 PM
Quote from: Tom on May 05, 2014, 01:12:23 PM
I don't actually need a windows server license. IIRC you can use a regular retail sku in a virtual env as long as you're not using it anywhere else.

The performance I am seeing in windows on kvm is a lot better than I have seen in the past, so I could go that way as well.

Yeah that is because your sku is married to your hardware if oem, and limited to moves every 90days if volume licensing.  Trust me, became a M$ licensing guru in the last year.  Why DC licensing can be attractive in large virtual environments.
Yeah, I said retail for a reason ;) I'm not sure you can virtualize your OEM license... could be wrong.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 01:20:31 PM
Same vm licensing model applies to oem or volume licensing in terms of how many vms you can run.  Just the moving of the license is different between them.  Retail = oem in this regard.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 05, 2014, 01:24:23 PM
I don't actually need too many windows vms. maybe a couple. one for sure, it'll be doing automatic mingw and msvc builds for various projects.

but what did you mean by "8 nice" ?
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 01:28:43 PM
Stupid auto correct on this loaner phone.....


8 Nics
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 01:31:34 PM
You can get away with 4 but more often I find i need at least 6 for fault tolerance when doing management, iscsi, nfs, and lan access.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 05, 2014, 01:34:39 PM
Hm, that's something to think about. It seems a bit overkill for a home setup. I do actually have a switch capable of bonding at least 4 at a time though :D some boards I'm finding have 4 to begin with (sometimes with an extra for ipmi), and I'd only need to get a separate 4 port intel, or two two port intels...

My current vm host has a single GbE connection hooked up, even though it has two. I just haven't bothered. The actual load on it has historically been pretty low. But I should really think about hooking them both up since the mc server is on there, as well as the new backup raid5 array (that houses a copy of my nas, and "important" backups).
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 01:35:33 PM
As for storage get most spindle count you can for the storage you need.  Damn iops kills me every time.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 05, 2014, 01:38:24 PM
Yeah, acting as a compile cluster, it'll need more ioops than my current box thats for sure. question is how to handle that hmm, a raid5 array doesn't immediately help ioops I don't think (well I know the 300MB/s my current windows guest is seeing from the new 5x3TB raid5 helps with things :))

Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 04:01:59 PM
Raid 10 is best for your compiles, but definitely hefty on cost.

I run full small business workloads on RAID 5/6.  It just really means the more spindles the better as a rule of thumb.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 05, 2014, 04:36:49 PM
A decent sized ssd cache in front of the raid array could help. there are a few options there, assuming linux.

Getting the absolute highest speed/performance is not required, just "decent". I was not at all unhappy with what I was seeing in my win7+kvm test. But I imagine it could be better with esxi, or if I tried virt-io drivers for windows, but I don't know if those have been updated. I haven't looked into it lately.

When you set up your vm image storage, what size drives do you prefer? Lots of smaller ones? Or fewer larger ones? Could get a boat load of cheap 500GB drives for more spindles (assuming equal total space compared to using larger drives), or load up on more 1TB drives (I already have 5-6). Of course the main limiting factor will be the number of bays I have available...
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 04:41:36 PM
I've done 9x600GB SAS 10K and been somewhat happy with the performance when 15-20 various VMs running on it.  I typically spec my Space requirements, add 50%, then find the most spindles I can run in that total space and what my chassis can hold.   Best solution is of course shared storage arrays, but for that you need a synology or qnap or something similar.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 05, 2014, 04:53:54 PM
One sad thing, is one of the 1TB drives I had in my old array was clunking :( and it was yanked out of the old array before I tore it apart and rebuilt it in another machine. I think I have another one or two sitting /somewhere/ but that dead one will need to be RMAed if its warranty is still good.

How busy are your vms? do they end up using a lot of io bandwidth and ioops?
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Lazybones on May 05, 2014, 04:56:26 PM
It is really going to depend on what those VMs are doing...
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 04:59:34 PM
Yes Lazy is right.  But from an Small Business perspective the Hosts are doing everything because usually you don't have distributed loads.

For you I'm sure 6 spindles will be more than enough at 7200 RPM.  So maybe a raid 5 at 7 Spindles?
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 05, 2014, 05:04:29 PM
That's doable. That is what I was figuring on going with. I'd have 7 1TB drives, but one died :( I'll probably leave some room for growth if needed, but 6 1TB drives is a fair amount of space.

I'll try and do some actual testing sometime soon, spin up an install of the software I'm going to be using (buildbot), let it run for a while, and see if performance is adequate.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 05, 2014, 05:11:19 PM
I have a whole bunch of 1TBs SATAs doing nothing, so don't buy some new ones if you need more.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 05, 2014, 05:21:23 PM
Oh, I might take you up on that. I haven't decided on the case yet. probably some standard tower case, with lots of 3.5" bays. I imagine the board I get will of course have a bunch of sata ports, and potentially a built in lsi 2008 controller, and I have another IBM M1015 card (lsi 9211 like) I can use in it, so I am not in need of ports.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Lazybones on May 05, 2014, 09:02:14 PM
Quote from: Tom on May 05, 2014, 05:21:23 PM
Oh, I might take you up on that. I haven't decided on the case yet. probably some standard tower case, with lots of 3.5" bays. I imagine the board I get will of course have a bunch of sata ports, and potentially a built in lsi 2008 controller, and I have another IBM M1015 card (lsi 9211 like) I can use in it, so I am not in need of ports.

Finding a properly rated PSU with that many direct SATA power connectors, or clean output to enough molex connectors is the key when you start to go over 4 or so drives... Drives are the next most power hungry devices to high end GPUs...
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 05, 2014, 09:15:35 PM
Quote from: Lazybones on May 05, 2014, 09:02:14 PM
Quote from: Tom on May 05, 2014, 05:21:23 PM
Oh, I might take you up on that. I haven't decided on the case yet. probably some standard tower case, with lots of 3.5" bays. I imagine the board I get will of course have a bunch of sata ports, and potentially a built in lsi 2008 controller, and I have another IBM M1015 card (lsi 9211 like) I can use in it, so I am not in need of ports.

Finding a properly rated PSU with that many direct SATA power connectors, or clean output to enough molex connectors is the key when you start to go over 4 or so drives... Drives are the next most power hungry devices to high end GPUs...

I'm still flip flopping on components, including the case. there are some decent 4U cases that have a backplane, which makes it easier to manage power and data cables.

My last real concern is per thread performance vs number of cores... I can spec out a decent AMD based box with 16-24 cores, but relatively lower per thread performance vs a 8-12 core intel based system. I've found it hard to find cpu comparisons for many of the opteron and xeon cpus :(

The price premium you pay for intel hardware is incredible.

I have not yet decided on if per thread is more important than multi threaded, but much of the work it does would benefit from more cores, except in some cases, like the windows based jobs, where there's really only going to be one or two cores available for a single job anyhow.... so in that case per thread performance does matter, but not a HUGE amount. I suppose I have a habit of answering my own questions, but I'm still waffling on this a little.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 06, 2014, 02:19:07 PM
SO hw wise I've decided on a Supermicro H8DG6-F (http://www.supermicro.com/Aplus/motherboard/Opteron6100/SR56x0/H8DG6-F.cfm) motherboard, dual Opteron 6282 16-core 2.6Ghz processors, 64GB (8x8GB) DDR3 1333 ECC Registered ram. No definitive case yet. Might go supermicro, might go full tower case that has plenty of 3.5" bays, or I might go for a cheap 4u or tower server case that isn't a Norco. As mentioned previously, noise level is a definite concern, so a 1u or 2u case is right out.

It's looking like I'll end up spending 2k after all is said and done :( more than I wanted to, but I felt it was better to go with slightly better, more upgradable hardware with far more cores, than older less upgradable hw with fewer cores (lga1366 cpus, or 4-8core c32 amd's).

Probably overkill, but it's better to be prepared for the future imo. I really wish I had put 32GB ram in my current home server when I bought it. The price of ram right now is incredible. $400+ for 32GB ECC Unbuffered ram.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Lazybones on May 06, 2014, 03:15:14 PM
Quote from: Melbosa on May 05, 2014, 12:58:36 PM
If you have to buy a windows server license anyway you might as well go HyperV. I could even set it up for you.  HyperV also does supportlinux distros... I run a few on my HyperV installations.  Your hardware options expand with the HyperV option as well.

Are you running those in domains? I tried Hyper-V Server 2012 with local permissions and nearly tore my hair out... If you run it as a full windows server you can always RDP in but the free server edition seems to depend on domain permissions heavily.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 06, 2014, 04:59:09 PM
HyperV 2012 R2 is great actually either as part of a domain or workgroup.  But even if you strip the GUI out of it might as well attach it to a AD if you have one.

I found the trick was to think of them interms of Windows not VMware.  I've had to do the same thing with XenServer; think of it as a Linux distro rather than VMware or HyperV.  They are all very different in management interfacing and configuration.  In the Windows world workgroup means very little kerberos authentication so your management functions are limited to that which does not utilize kerberos - live migrations for example.  Similar problem to if you had VMware without a VMotion license and vCenter to manage.  And XenServer... well its a beast all on its own in that regard.

So yeah while I still say VMware is the #1 for sure, HyperV isn't that terrible anymore with 2012 R2... its actually production ready.  Anyone who started with HyperV 2008 rather... well I can see where the hatred would come from /badtasteinmouth
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Darren Dirt on May 06, 2014, 05:09:59 PM
reading this thread, and others like it over the last few years, I gotta say I feel like this:

(http://www.quickmeme.com/img/0b/0b346c6960292d687400cfa809f60cb2fcf92d3e485d57eef49f70697935d88b.jpg)



...carry on, you crazy admin-types, you...
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 06, 2014, 05:10:35 PM
But I love lamp too!
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Darren Dirt on May 06, 2014, 08:10:40 PM
I understood every word in the above post.

Can't say that about most posts in this thread. 'Swhy I used Brick to express myself.


Carry on...
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 06, 2014, 08:47:56 PM
Quote from: Darren Dirt on May 06, 2014, 08:10:40 PM
I understood every word in the above post.

Can't say that about most posts in this thread. 'Swhy I used Brick to express myself.


Carry on...

This is a secret, but I'm not a system admin, I'm a lowly code monkey. :o

Also, server pretty much purchased. It'll take a while for things to arrive however. The joys of ebay. At least a week till I have everything, if not two+. I'm hoping for 1 week.

I also decided to downgrade the mobo slightly to the H8DGi-F that is pretty much identical to the H8DG6-F, except it's missing the built in LSI chip. I ended up saving a couple hundred dollars with that decision. I have an IBM M1015 I can use for the main storage so it's not an issue at all.

Went with a Fractal Design XL R2 case, and a Seasonic X 850W psu. I /almost/ went with a nice seasonic case, they are so tempting, but I wanted a few more drive bays, and the only seasonic cases I could justify were limited to 8 :( Other options included el-cheapo 4U cases, but the reviews on Norco's and especially Rosewill's are just awful.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Lazybones on May 07, 2014, 08:33:41 AM
Quote from: Melbosa on May 06, 2014, 04:59:09 PM
HyperV 2012 R2 is great actually either as part of a domain or workgroup.  But even if you strip the GUI out of it might as well attach it to a AD if you have one.

I found the trick was to think of them interms of Windows not VMware.  I've had to do the same thing with XenServer; think of it as a Linux distro rather than VMware or HyperV.  They are all very different in management interfacing and configuration.  In the Windows world workgroup means very little kerberos authentication so your management functions are limited to that which does not utilize kerberos - live migrations for example.  Similar problem to if you had VMware without a VMotion license and vCenter to manage.  And XenServer... well its a beast all on its own in that regard.

So yeah while I still say VMware is the #1 for sure, HyperV isn't that terrible anymore with 2012 R2... its actually production ready.  Anyone who started with HyperV 2008 rather... well I can see where the hatred would come from /badtasteinmouth

I couldn't even get it stood up reasonably in work-group mode in my home green field, that is why I asked... I found the permissions to be awkward without a domain...  The fast majority of my home devices are not domain join-able so I no longer put the effort into running a domain in my home network.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 07, 2014, 10:23:47 AM
Here's a question, can ESXi deal with drives brought in as individual devices (jbod mode), and then have the main guest then build a sw raid5, and host images or logical volumes to the rest of the guests? Am I stuck with iSCSI when trying to do that? Or does ESXi allow the main guest to provide devices to other guests like virtually every other virtualization solution? I would prefer the data not go over the guest's internal networking.

Something makes me think that ESXi doesn't have the concept of a "main" guest, and that it has a very limited mini os built into the hypervisor instead of being a shim under a more capable OS (like Xen).

I'm googling, but so far haven't found a direct answer.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Lazybones on May 07, 2014, 10:51:53 AM
ESXi is as close to bare metal as you get... I haven't tried to setup a stand alone server without vCenter so I can't say... Maybe melbosa can...

In ESXi you can use RAW device maps... The guest VMs can own the disks... Hell you can put ESXi on a USB stick leaving ALL internal drives up to the VMs.

Things have changed in terms of management with 5.5 so that is why I am not sure what happens in a small lab setup.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 07, 2014, 10:55:52 AM
Yeah, you can hand off disks as RDM, or as individual VMDK pools to a single guest, but then you have to somehow give access to those to other guests, which seems to be iSCSI or NFS over the guests networking stacks. That's a bit of overhead I'd rather not have.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Lazybones on May 07, 2014, 11:02:48 AM
Quote from: Tom on May 07, 2014, 10:55:52 AM
Yeah, you can hand off disks as RDM, or as individual VMDK pools to a single guest, but then you have to somehow give access to those to other guests, which seems to be iSCSI or NFS over the guests networking stacks. That's a bit of overhead I'd rather not have.

The more proper way to do it is with a supported hardware RAID controller, then use virtual drives... This is one of the KEY reasons I made my cheap home setup under KVM.. I wanted software RAID.

I believe in Hyper-V free you would also get the option of FREE host software RAID.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 07, 2014, 11:13:49 AM
Yeah, I'm starting to think I may just go with KVM in the end. I was looking at vSphere pricing and the only license I could conceivably see myself paying for is the Essentials. Though that is missing some useful bits that are only in Essentials Plus which is like 2-3x more expensive.

Hyper-V might be interesting to play with, but my desire to admin a windows box is pretty low.

My M1015 is a supported RAID controller afaik, but it's RAID5 is locked out with a hard to find "RAID5 KEY". It's annoying. It can do JBOD, 0, 1, and 10 without additional hardware. If I can find a better used RAID card i might go that route. We'll see.

And with KVM I get some basic (almost?) live migration to/from my older box, so long as I stick to the generic Qemu cpu config. Of course that needs iSCSI or NFS regardless. so maybe I wont make that a requirement ;D hahaha.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 07, 2014, 11:16:59 AM
ESXi does not have any type of software RAID solution in the host itself.  As Lazy said you can use a USB Stick or Flash Drive for the OS and save the disks completely for your RAID but you will need a RAID Controller to do so (and one that is compatible with the ESXi 5.5 build).  Your other solution is as you said, VM to RDM with Software RAID and then iSCSI or NFS... and just a side NOTE: latest version of ESXi 5.5 U1 which fixes the heartbleed bug has a known issue with random NFS disconnects.  Nice thing with the RDM solution for you is all the iSCSI or NFS traffic will never touch the Network Layer, it will all be handled by the hypervisor Disk and Memory bus speeds as the traffic will never leave the virtual switch.

Managing an ESXi 5.5 box without vCenter is still possible with just the vSphere Client (free) but you will find you will be doing a lot of esxcli or esx powershell commands still.

Hyper-V will give you a software RAID solution which is actually very well implemented now in the 2012 versions (even better in 2012 R2).  And again if you have to own a Retail version of Windows Server 2012 R2 I would just install that as your hypervisor (and then install your 1-2 Windows Server VMs you get with that license in Hyper-V guest) rather than the free Hyper-V if you go with a Microsoft Hyper V solution - which I believe you are NOT.  But thought I would point that out.  Only reason to go Hyper-V Free editiion is if you a) are not running any Windows Server guest OSs  OR  b) your Windows Server guest OSs are older than the current free version of Hyper-V.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 07, 2014, 11:24:46 AM
Quote from: Melbosa on May 07, 2014, 11:16:59 AM
ESXi does not have any type of software RAID solution in the host itself.  As Lazy said you can use a USB Stick or Flash Drive for the OS and save the disks completely for your RAID but you will need a RAID Controller to do so (and one that is compatible with the ESXi 5.5 build).  Your other solution is as you said, VM to RDM with Software RAID and then iSCSI or NFS... and just a side NOTE: latest version of ESXi 5.5 U1 which fixes the heartbleed bug has a known issue with random NFS disconnects.  Nice thing with the RDM solution for you is all the iSCSI or NFS traffic will never touch the Network Layer, it will all be handled by the hypervisor Disk and Memory bus speeds as the traffic will never leave the virtual switch.
It's still a switch, which means it hits the guest's own network stack. Or does vmware have a custom iSCSI driver that bypasses the network stack?

Quote from: Melbosa on May 07, 2014, 11:16:59 AM
Managing an ESXi 5.5 box without vCenter is still possible with just the vSphere Client (free) but you will find you will be doing a lot of esxcli or esx powershell commands still.
I thought the free client was limited to 60 days?

Quote from: Melbosa on May 07, 2014, 11:16:59 AM
Hyper-V will give you a software RAID solution which is actually very well implemented now in the 2012 versions (even better in 2012 R2).  And again if you have to own a Retail version of Windows Server 2012 R2 I would just install that as your hypervisor (and then install your 1-2 Windows Server VMs you get with that license in Hyper-V guest) rather than the free Hyper-V if you go with a Microsoft Hyper V solution - which I believe you are NOT.  But thought I would point that out.  Only reason to go Hyper-V Free editiion is if you a) are not running any Windows Server guest OSs  OR  b) your Windows Server guest OSs are older than the current free version of Hyper-V.
I have no expectation of running windows server instances. My preference is anything but Microsoft ;) haha.

What would you say is the main plus going for Hyper-V over ESXi, or KVM? Especially for a linux person? I for sure will have one windows guest, maybe more, but no idea for sure yet.

The main positive I can see, is I get to learn something new. Get my fingers back into the windows server side of things (I took a course back in the day and learned all about NT 4! even though 2k+ was already out, what a crock that was).
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Lazybones on May 07, 2014, 11:48:26 AM
Quote from: Tom on May 07, 2014, 11:24:46 AM
What would you say is the main plus going for Hyper-V over ESXi, or KVM? Especially for a linux person?

I would honestly say for you KVM would be a better solution than Hyper-V or ESX...

If I was going to change one thing in my home KVM setup it would be to build it off of a Redhat based distro so that I could use one of the MANY Web management interfaces.

A stand alone Hyper-V free machine is going to require lots of windows specific power shell or a windows box to run the management tools.
ESXi 5.5 without a vCenter (web ui) will require running the windows management client, which for you also doesn't work.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 07, 2014, 11:52:25 AM
It is starting to look like KVM is my best bet if I want something that "just works" for me. I typically use virt-manager to manage my existing KVM box, and it works rather well these days. IIRC its a standalone python GTK app. It used to be pretty buggy, but seems to work alirght now.

The two main reasons I wanted to use ESXi was that it should have better performance, and I wanted to learn and play with it. But it doesn't seem like they care much about that use case at all.

I can run windows apps in a VBox windows install I have, but it's a bit annoying to boot it up. Also I read that the standalone ESX app doesn't work on the new format virtual machines. You have to somehow create v8 or v9 vms.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 07, 2014, 12:17:56 PM
Quote from: Tom on May 07, 2014, 11:24:46 AM
Quote from: Melbosa on May 07, 2014, 11:16:59 AM
ESXi does not have any type of software RAID solution in the host itself.  As Lazy said you can use a USB Stick or Flash Drive for the OS and save the disks completely for your RAID but you will need a RAID Controller to do so (and one that is compatible with the ESXi 5.5 build).  Your other solution is as you said, VM to RDM with Software RAID and then iSCSI or NFS... and just a side NOTE: latest version of ESXi 5.5 U1 which fixes the heartbleed bug has a known issue with random NFS disconnects.  Nice thing with the RDM solution for you is all the iSCSI or NFS traffic will never touch the Network Layer, it will all be handled by the hypervisor Disk and Memory bus speeds as the traffic will never leave the virtual switch.
It's still a switch, which means it hits the guest's own network stack. Or does vmware have a custom iSCSI driver that bypasses the network stack?

Quote from: Melbosa on May 07, 2014, 11:16:59 AM

Yes but all the network IO is done in memory so it is still faster than an actual physical switch.  Performance proven if you have guest OSs talking to each other (no matter what the protocol) on the same vSwitch on the Same Host, over a physical device (at least VMware has proven it on their hypervisor).
Managing an ESXi 5.5 box without vCenter is still possible with just the vSphere Client (free) but you will find you will be doing a lot of esxcli or esx powershell commands still.
I thought the free client was limited to 60 days?

Quote from: Tom on May 07, 2014, 11:24:46 AM
I have no expectation of running windows server instances. My preference is anything but Microsoft ;) haha.

What would you say is the main plus going for Hyper-V over ESXi, or KVM? Especially for a linux person? I for sure will have one windows guest, maybe more, but no idea for sure yet.

Licensing costs if you were having to license Windows Guest OSs and heterogeneous compatibility between hypervisor and guest OS (in the Microsoft case).  As I said for you probably NOT the right choice.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 07, 2014, 12:21:34 PM
Quote from: Tom on May 07, 2014, 11:24:46 AM
Quote from: Melbosa on May 07, 2014, 11:16:59 AM
Managing an ESXi 5.5 box without vCenter is still possible with just the vSphere Client (free) but you will find you will be doing a lot of esxcli or esx powershell commands still.
I thought the free client was limited to 60 days?

No ESXi Eval License is 60 days, so is the vCenter Eval.  The vSphere Management Client software is free, you just point it directly at the host IP and use root creds.  No vCenter or Web GUI required (yet - looks like the vSphere Client will be phased out in the next couple releases).
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 23, 2014, 11:37:45 PM
Yay. Server is built, and i'm running a cpu stress test. Things are going well so far.

Had to re-mount the cpu fans cause they were on backwards. And I added two fans on top, and one on the front cause temps were initially hitting 70c (not actually 70c, but the k10temp sensors arbitrary scale value, which is actually at or over these processor's max operating temp). Now that I have everything set up, temps are down to 40c so far. It seems to be rising ever so slowly, so i'm not sure where it'll top out quite yet.

It is VERY silent, especially for a dual socket server. It's amazing the difference slower spinning much larger fans make.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Mr. Analog on May 23, 2014, 11:43:26 PM
Nice!
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 24, 2014, 12:23:56 AM
Mel, any chance you could bring along some of those 1TB drives you said you had laying about? What kind are they? And how much do you want for em?


Quote from: Mr. Analog on May 23, 2014, 11:43:26 PM
Nice!
I'm liking it so far :) other than a slight problem with it not recognizing one stick of ram.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Mr. Analog on May 24, 2014, 01:00:58 AM
Quote from: Tom on May 24, 2014, 12:23:56 AM
Mel, any chance you could bring along some of those 1TB drives you said you had laying about? What kind are they? And how much do you want for em?


Quote from: Mr. Analog on May 23, 2014, 11:43:26 PM
Nice!
I'm liking it so far :) other than a slight problem with it not recognizing one stick of ram.

Nice! Man, I've had problems like that before, that sucks :C
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 24, 2014, 03:15:53 AM
Ugh, I might now have two dead 1TB drives. wtf seagate.

append: OMG. I just realized. If I had used my old array longer, I'd have lost the entire thing at some point. I know I had been running with one failed disk for a while, but at that point I had already moved all that data to the new NAS. sheesh. I hope my warranties are still good. but I bet they aren't.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 25, 2014, 04:42:46 PM
So it appears the 1TB seagates in my array have been "up" for 37359 hours, or about 4.2 years. Most of that was 24/7 uptime. So it's not a large surprise that they are starting to go.

I may need to steal some replacements sooner rather than later. In addition to those two very dead drives, one more is stuck on a long selftest, and another is taking seconds to respond to smartctl log requests.

This does not fill me with good feelings:

root@bender:~/build/whdd/build# ls /dev/sd{a,b,c,d,e} | xargs -n1 smartctl -a | grep -P \(Reallocated\|Bad\|Serial\|Power_On\)
Serial Number:    6VP063MF
  5 Reallocated_Sector_Ct   0x0033   099   099   036    Pre-fail  Always       -       48
  9 Power_On_Hours          0x0032   058   058   000    Old_age   Always       -       37359
183 Runtime_Bad_Block       0x0000   022   022   000    Old_age   Offline      -       78
Serial Number:    6VP061R1
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       5
  9 Power_On_Hours          0x0032   055   055   000    Old_age   Always       -       39662
183 Runtime_Bad_Block       0x0000   001   001   000    Old_age   Offline      -       121
Serial Number:    6VP055FL
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       21
  9 Power_On_Hours          0x0032   061   061   000    Old_age   Always       -       34645
183 Runtime_Bad_Block       0x0000   021   021   000    Old_age   Offline      -       79
Serial Number:    5VP8RRWZ
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   074   074   000    Old_age   Always       -       23126
183 Runtime_Bad_Block       0x0032   022   022   000    Old_age   Always       -       78
Serial Number:    6VP0658D
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       30
  9 Power_On_Hours          0x0032   057   057   000    Old_age   Always       -       37730
183 Runtime_Bad_Block       0x0000   100   100   000    Old_age   Offline      -       0


These are all from my older array. all been on for ages.

Melbosa, what brand/model are your spares? What kind of life have they seen?
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 26, 2014, 10:25:51 AM
Quote from: Tom on May 24, 2014, 12:23:56 AM
Mel, any chance you could bring along some of those 1TB drives you said you had laying about? What kind are they? And how much do you want for em?

Sorry missed this post last week.

Like I said Saturday, make me an offer.  I have 1TB 7200 RPM Seagates and 1TB 5900 RPM Seagates, and a few 2TB Seagate and WDs.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 26, 2014, 01:10:04 PM
Hmm, What condition are they in, do you know? Mine are a bit on the poor side...

I want to say $10 per? I guess it depends on how much use they've seen. I really don't want to touch any Green/LP drives, they are notorious for failing. They can't handle 24/7 uptime. Great little offline backup drives, but thats about it. WD Reds and Seagate NAS drives are fine though.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 26, 2014, 03:09:08 PM
WDs are Green.

Seagates are not NAS grade just workstation Grade.

They were in my file server for a few years.  I can still get $30 or so for them so $10 might be a bit low even with the Friend discount.  Maybe $20 per.
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 26, 2014, 03:14:19 PM
Quote from: Melbosa on May 26, 2014, 03:09:08 PM
WDs are Green.

Seagates are not NAS grade just workstation Grade.

They were in my file server for a few years.  I can still get $30 or so for them so $10 might be a bit low even with the Friend discount.  Maybe $20 per.
Hm, I can probably do $20. Regular drives being up for many years isn't the greatest though, as evidenced by the 20-30+ reallocated sectors on the 4 year uptime drives I have ;D

How many do you have? And I assume they are Plain old 7200.xx Baracudas?
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 26, 2014, 03:44:18 PM
Yep.  I have 2x7200 and 6x5900s
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 26, 2014, 04:03:30 PM
Allright, I'll take those off your hands, and any other drives that aren't Green/LP ;D
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Melbosa on May 27, 2014, 10:19:32 AM
You want all 8?
Title: Re: Specing/Pricing out a home ESXi 5.5 box?
Post by: Tom on May 27, 2014, 10:39:58 AM
Sure, might as well. I found a cheap 16 bay hotswap sas expander chassis. could whip up two arrays. one for the 7200, and one for the 5900s.