IT 2.0 :  Next Generation IT infrastructures

Virtualization is a disruptive technology and we all know that. With this post I want to share with you some scenarios about how server (and storage) virtualization can drastically change the landscape for "small IT shops" (aka SMB's) in the context of High-Availability and Disaster/Recovery. Up until today server "high availability" was not for everyone as it required a complexity and a cost that many IT shops could not sustain. I have already been talking about the change of paradigm that a virtual infrastructure brings in when it comes to make a service highly available. You can read this post for more info.

However that covers a small portion of the bigger picture. Particularly that post assumes that you have a number of physical servers connecting to a central storage repository so that you can restart your VM (i.e. your service) should one physical server fail. Fair enough but this obviously doesn't cover the other important subsystem which is the storage subsystem. In a scenario like this the central storage repository is a so called SPOF (or Single Point Of Failure). If you are a big Bank / Telco organization (or if you have enough money to spend anyway) you can get something decent using Midrange / Enterprise Storage arrays that support all sort of replication features so that you can literally create a fully redundant infrastructure with no SPOF. It must be noticed that while it is true that even Entry level Storage arrays might have no internal SPOF it is common referring to these boxes as a single entity (hence a potential single source of issues). This is even more evident when you think about being able to stretch your virtual infrastructure across two buildings in the same Metropolitan Area Network (such as 3 physical servers in Building A and 3 physical servers in Building B comprising a single cluster): this obviously leaves you with the only choice of putting your single Storage array in either Building A or Building B.

I don't want to get into all sort of discussions regarding how you would define a scenario like this. Someone refers to this as "DR", someone else refers to this as "Campus HA", someone else refers to this as "Continuous Availability". I am actually not interested in formal definitions. I am more interested in the fact that the vast majority of the customers I talk to (be them big Banks, big Telco's or the small SMB IT departments) would like to leverage virtualization technologies to be able to achieve this scenario in a much less complex way (typically a requirement of the big boys that have a very complex distributed IT infrastructure) or in a much less expensive way (typically a requirement of the not-so-big boys that have a tight budget). It is amazing to hear that one of the most appealing reasons for which all customers are virtualizing ... is not for consolidating the servers (sure this is important) but to provide better high availability and DR mechanisms. Virtualization is really a paradigm shift.

I have already said that for the bigs there are (expensive) technologies and products that allow you to achieve that. This is an example of a project I have been working on a few years ago. But where does this leave the "small" shops? I am talking about customers that have from 10 to 15 Windows / Linux servers and that do not have a budget that allows them to intercept Midrange Storage technologies with replication capabilities. These arrays are not enormously expensive (in fact I am assuming that who has more than 15 or 20 servers perhaps has a decent IT budget that allows them to buy these technologies). However IT departments that have up to 10 or 15 virtual instances which, by the way, could be deployed on as few as two 2-socket systems (for redundancy) based on these other assumptions I discussed in a previous post.... might not be keen on buying a Midrange Storage array just for the purpose of being able to replicate and protect the data. Don't get me wrong, the "geeks" working for IT would love to have that kit in their hands, it's the "buyer" within the organization that would question the value for the money.

That's why when I was first introduced to technologies such as those from Lefthand Networks and then Datacore I was intrigued. What these companies do in essence is storage virtualization. They do it differently and the product packaging itself is not identical but essentially, bottom line, they hide the layout of the storage arrays and the way it is presented to the hosts and their OS'es. As I said they have a different approach in how they achieve this.

Lefthand Networks sells SAN "building blocks" that are effectively x86 servers with their own software on-board. This software simply turns those network-connected x86 servers equipped with a certain amount of Direct Access Storage into a highly available and distributed SAN that the computational hosts and their OS'es can access via an iSCSI protocol.

Datacore uses a slightly different philosophy as it is sold as a software package that you would install on a Windows host (typically more Windows hosts for redundancy and high availability reasons). Storage administrators would then give visibility of heterogeneous and distributed storage resources to these hosts, typically via FC but not limited to that, so that the Datacore software could present, using various protocols such as iSCSI and FC, to the hosts and their OS'es, a virtual storage repository.

These two pictures should clarify this brief explanation of the two technologies.

Obviously this is not intended to be an exhaustive explanation of these vendors' technologies, features and strategies. Make sure you contact them directly should you be interested in knowing more about this.

In a typical scenario as the one I have outlined in the pictures above, separated physical "storage islands" would be aggregated into a single virtual SAN by the Lefthand/Datacore software that in turns provide a robust storage repository to (yet) other physical hosts running your production applications. That is how these products are usually positioned. How all this ties into the HA and DR for the masses I was referring to? Well it turned out that it is possible (and this is how both companies are now marketing their products as well) to install the software logic within virtual machines. Imagine a new innovative deployment scenario where you would create two special purpose virtual machines on two separate virtualized hosts and each virtual machine is associated with a vast amount of locally attached storage. At that point the Lefthand/Datacore software would turn those two VM's (with a bulk of local storage space associated to them) into your virtual SAN that is going to serve, through the iSCSI protocol, the other virtual machines (supporting your own production applications) running on the same two virtualized hosts. Confused? I think a picture is worth 1000 words.

On top of all the functionalities that these software provide the most interesting, in the context of this post, is the ability to mirror the local content of the Storage VM's in order to create an active/active fully redundant iSCSI SAN. As you can depict from the picture above the logical layout is quite different from the physical layout. Let me try to explain: logically the two Storage VM's create the virtual iSCSI SAN. The virtualization layer then maps the LUN's made available by the clustered Storage VM's and use these LUN's to create VMFS volumes to host the production virtual machines to support the business. The logical perspective differs from the physical perspective in the sense that, while logically the virtualization layers connect to an "external" iSCSI SAN and use it as an external service... the actual code that instantiates the virtual iSCSI SAN runs as redundant stand-alone virtual machines on top of the same virtualization layers.

One of the key advantages of such a setup is being able to tolerate the failure of one of the two systems and continue to be able to operate. The following pictures illustrates what happens should either one of the server or an entire building go off-line for some reason.

Since the Storage VM's replicate the local storage associated to them, either one of these entities can fail (be it the Storage VM itself, the virtualization layer, the physical server or the entire building) without affecting the availability of the VMFS volumes created on top of the mirror. This is transparent to the surviving virtualization layer as it could be compared to a failure of a Storage Controller within a redundant FC Storage Server. It is worth noticing that, in case of failure of the physical hosts or failure of the building, VM's can be either manually or automatically restarted on the surviving node depending on the virtualization layer being used and the feature set associated to it.

I am not getting into the details but consider that both these software support Synchronous and Asynchronous replication of the data so that you can even tune your solution based on the distance of the two buildings. In the simplest scenario both buildings are in the same Metropolitan Area Network so you would treat the two servers as if they were installed in the same rack of a single building. On the other hand if your buildings are far apart (or otherwise not LAN-like connected) you can tune it to use Asynchronous replication and build something that is closer to a Disaster/Recovery plan (well I am oversimplifying here but you get the point).

It is also worth noticing that since the Datacore software is an add-on that you (the customer or the integrator) would install on top of Windows, you can use any virtualization layer that allows you to create Microsoft Windows virtual machines making it very flexible in terms of deployment options. Lefthand on the other hand provides the Storage VM as a virtual appliance (thus making it more robust and easy to deploy in my opinion than a Windows add-on like Datacore) that is however, as of today at least, only available for VMware VI3 virtualized platforms.

This is clearly not something that you might want to look at in the context of a medim / big virtual infrastructure deployment where Midrange / Enterprise Storage arrays with their native virtualization and replication techniques offer a great deal in terms of performance, scalability and reliability. I don't want to downplay Lefthand and Datacore but I think there is a positioning that needs to be taken into account when comparing these products with Midrange / Enterprise class Storage arrays features. But in the context of the small IT shops, using the technologies described in this post you can achieve similar features at a small fraction of the costs and it might make sense doing so.

Let's try to do the math. A couple of System x 3650 configured with 2 x Intel Quad-Core processors, 16GB of RAM, "a bunch" of local disks and network adapters might cost around 10.000$ each (list price). This makes a 20.000$ total (list price) for the hardware.

I am not an expert on Lefthand and Datacore pricing (nor I want to become one) but as far as I have seen it would be fair to assume that, in a context and scope like this, the software to enable each Storage VM would cost around 5.000$ (list price). This makes a total of 10.000$ for the virtual SAN with remote replication capabilities.

Then it comes the virtualization layer. Here there are a number of options (both from a vendor perspective as well as from a feature set perspective within the same vendor). Clearly if you want to use MS technologies to enable the virtual infrastructure solution (i.e. Hyper-V with Systems Center Virtual Machine Manager) the cost would be pretty low. On the other hand if you want to use VI3 Enterprise to enable it the costs would be higher (so would be the feature set). Obviously one should also take into account lower costs VMware VI3 alternatives (such as VMware VI3 Foundation) as well as alternative virtualization vendors such as Citrix and VirtualIron. All in all I think it would be fair to assume that one could spend, on average, 5.000$ to enable both physical systems with a virtual infrastructure software (again it might be as low as 0$ or as high as 10.000+$ depending on what you want to achieve).

All numbers we have mentioned (and assumed) are list prices and I will let you do the math on the average discounts you can achieve on each of those items (for example I know Lefthand has some bundle offerings that lower considerably the price). In general it would be fair to assume that, for a "low 2 digit thousands of US dollars" (what a great way to not tell you what I think a potential discount could be) you can get the following:

• A physical Server and Storage infrastructure capable of supporting up to 10 or 15 virtual images
• Integrated Server and Storage high availability and redundancy
• Compatible with all server virtualization enterprise features (live partition mobility / high availability for virtual machines / etc)
• Without the costs associated to a SAN environment
• With acceptable performance and good-enough scalability (in the context of small IT shops)

All this at a fraction of the costs compared to achieving similar characteristics using high end Enterprise class components and products. As I said this is not meant to be for everyone; Midrange / Enterprise Storage Server arrays should continue to be intended as the preferred choice for High Availability and Disaster/Recovery scenarios in many circumstances. This is however a great way for customers with limited budgets to achieve similar levels of features at a fraction of the price. This is not about cannibalizing the Midrange and Enterprise products market, but it is rather making similar level of features available to the masses (masses that would not be able to get the same features otherwise).

In closing this thread I'd like also to point out that virtualization is not, as many still think, (just) the capability to carve a number of software partitions out of a single physical system (aka server consolidation). Virtualization is really a paradigm shift within the data center that allows to re-architect the entire stack (hardware, software, management) in a completely different (and better) way. It allows customers and vendors to look at the typical problems from a completely different angle, thinking out-of-the-box if you will. It allows to solve problems in a way that a few years ago one could not even think it would have been possible.

Massimo.

In the last few months I have been struggling to understand what is so different, in terms of mass adoption, between virtualizing server workloads and virtualizing desktop workloads (also known as "VDI" or "Virtual Desktop Infrastructure"). I have been exposed to this phenomenon of x86 virtualization since around 2000 where the idea was as simple as taking a high end server and miniaturizing it into many small virtual servers. Similarly I have been exposed for the last 3 years to the other big use-case for x86 virtualization which is "Desktop Virtualization" and I can tell you that the time it took for the first traditional use-case to take off (through seeding the market with the idea - piloting and proofs of concept - mass adoption) was way shorter than the time it is taking for VDI to take off (going through the same phases above). This doesn't mean that VDI is not taking off but there are no doubts that after 3 years from introduction I have seen so many more production implementation of VMware ESX than I have seen of VDI.

Why is that? Isn't VDI just virtualizing XP rather than Windows Server? Well not quite I would say. Let's dig into some of the details (not in strict order of importance).

- Desktop Virtualization alternatives. While I am focusing this discussion on the VDI concept there are some analysts that, for good reasons, are implying that desktop virtualization is not just VDI (i.e. virtualizing Windows XP and putting it on a server in the back). There are other alternative architectures  to "virtualize a desktop" such as Windows Terminal Services, Application Virtualization, OS streaming and many others. To complicate things further these technologies are sometimes complementary to each other and sometimes alternative to each other. So customers are challenged since the beginning of the potential desktop virtualization project with a great deal of input and information that they find hard to understand and digest. In the server space this has never been a great deal since "virtualizing a server" has always had a single meaning that was that of "hardware virtualization" (i.e. getting as many virtual hardware partitions as possible out of a single physical server). So in the server virtualization realm the confusion was far less than the one that is being created nowadays given all the potential architectures at the very different layers of the desktop software stack (and VDI is just one of these different architectures).

- VDI Products complexity. On top of the above complexity there is another one. In fact 8 years ago it was much easier to understand the products you needed to adopt a server virtualization model. If you used to buy 20 physical servers and install 20 Windows instances, now with server virtualization you would buy 2 physical servers, 2 VMware ESX 1.x licenses and install 20 Windows instances. As easy as it is. You couldn't do much differently and it worked great (so why bother?). VMware has since introduced new versions of the software and enriched their value proposition "linearly" with Virtual Center 1.x and eventually with VI3. On the other hand to adopt a desktop virtualization model you have to buy a virtualization platform, a connection broker, and you need to decide which access device you want to use etc etc. For every single layer of the architecture you have multiple implementations which translate into multiple different products that are supposed to do similar things (if you want to know more about the architecture of VDI have a look at this presentation). As a result in the last few years this desktop virtualization market has been very "foaming" with ISV's entering into this space and ISV's buying out other ISV's etc etc. Clearly it is much more difficult right now to understand what do to and which ISV to buy from a VDI solution than it was 8 years ago for a customer interested in entering the server virtualization space.

- Overall cost of the solution. In the desktop space there is a predominant metric that is "cost per seat" that you can hardly find in the server space. Sure customers understand that a server virtualization solution could cost slightly more than a traditional layout of a string of small physical servers but apparently they are more ready to discuss the benefits (in terms of TCO) of a virtualized solution and factor them in into the overall costs. This is especially true when these customers are considering high-availability solutions and disaster recovery that are either very expensive in the standard physical space or not achievable at all. On the other hand the "cost of the desktop" is a very strong metric that most customers are using when discussing the overall costs of a desktop virtualization solution. A couple of days ago I met with a customer that, as part of a very large bid, was buying (branded and good quality) desktops for 233€ (monitor and Windows license included). Needless to say that in a VDI solution which comprises the back end-servers, the virtualization software, the proper Microsoft licenses, the connection broker software, the thin clients and the miscellaneous utilities you might want to use to complement the scenario, the cost per user might be VERY WELL above that 233€. While for a server virtualization scenario the overall acquisition price of the solution can get close to what a customer would pay for a standard physical deployment (or at least within a reasonable range that is off-set by the tremendous advantages), to create a business case for VDI you have to include a detailed TCO analysis to get on pair with a standard desktop deployment. And we all know how difficult it is to "sell" on TCO (especially to desktops buyers).

- Microsoft licensing. Of particular importance is the issue of MS licensing. Historically, customers have always bought Windows PC's and historically these Windows PC's have come with a so called (very cheap) OEM Windows license (that is, when you buy a PC you get a Windows license tied to it). This OEM license CANNOT be used in a VDI scenario so you need to buy brand new licenses. And this is where the "fun" starts. This is a very bad story for customers both from a complexity perspective as well as from a cost perspective. At the time of this writing Windows licensing for virtual desktops is still pretty confusing: "should I buy a retail version of the OS?",  "Should I buy the VECD (Vista Enterprise Centrlized Desktop) license under Software Assurance?",  "What if I am not a customer with MS Software Assurance?" etc etc. All in all whatever you decide fits best your scenario as a customer, it's going to be more expensive than the cheap OEM Windows license you used to buy tied to your desktops purchase. We all hope MS will make this transition easier for our customers but so far ... not so good.

- End-user Experience. There is a big difference between virtualizing a server and virtualizing a desktop from an end-user perspective. You, as a CIO / Sys Admin, can virtualize a server or even the whole server farm and no one at your company would even notice it. It's just your own decision to do that or not to. In a desktop virtualization scenario, as soon as you start deploying the first thin client you are opening it up to the whole company. Immediately you have exposed your decision to dozens / hundreds / thousands of  other individuals that, for good reasons or political reasons, will start to challenge you. Good reasons might be technical limitations that you have to compromise with as of today, limitations for which a thin client can sometimes hardly cope, in terms of local device attachment support / multimedia video performance / flexibility / off-line capabilities etc etc, with a standard desktop deployment. I can assure you that no single "average end-user" would ever realize that their mail system in the back is now running on a vm whereas yesterday it was physical; however even the more "IT-candid end-user" would understand that he / she is using Outlook from a "little box where I cannot even attach my iPOD anymore" as opposed to the PC he / she was used to! And there is when political problems start. On this I have always said that a very happy Sys Admin has a frustrated end-user base and, viceversa, a very frustrated Sys Admin has a happy end-user base. It's a matter of compromising as usual: VDI technology advancements will allow the CIO / Sys Admin to provide the standard business requirements whereas end-users will need to understand that they can't just see their business access device as if it was their home PC.

I think these are some of the major road-blocks for VDI to become really true and start the massive deployment we have seen in the traditional server virtualization use-case. All in all I think that the root of the problems when trying to re-architect the desktop deployments is that, whatever you do, it's basically a "hack". If you think about that for a minute, the WHOLE industry only has one default that is "the end-user will be using a Windows desktop". Whatever you do with any technology that the industry is creating (be it an application, a physical USB device or whatever) to make it work in a different scenario... it is a "hack". We have implemented hacks with Terminal Servers and we are doing the same with VDI and any other technology such as Application Virtualization. As long as there is an industry that creates "stuff" for the PC and there is just a handful of people that try to make the "stuff" work differently in a different scenario ... it will always be an up-hill. I look forward to the day when the industy as a whole will embrace these non-PC deployments in a more structured way than the current "I'll do this assuming the PC and then someone will be able to hack it to make it work for alternative scenarios". I look forward to the day when the average "CIO Joe" that needs to create an IT infrastructure will not only think "I have 1000 users, I have to buy 1000 Windows PC's" but rather ... "I have 1000 users, I need to buy a VDI solution for them".

At that point all these things such as products and architecture complexity, end-user experience, licensing issues etc etc will fall apart... because it has become the "obvious / default" way to give end-users access to IT.

Massimo.

Hardware virtualization these days is a hot topic and we all know that. There are many customers looking into it for the first time and one of the problems they are facing right now is how they are going to size their new virtual infrastructure. Lately I have received lots of requests from many people in order to help them project the hardware investments (in terms of physical servers) that they need to jump onto the virtualization band wagon. In this post I'd like to try to provide you with a very quick and dirty method to do that.

Consider that there are many alternatives to get to a "decent and professional" technical result: you can either hire a consultant for a performance analysis of your current physical infrastructure and have him/her come out with the required hardware infrastructure to support your workload or you can do that on your own with professional tools available in the market (consultants can also leverage these tools and yet provide additional value). These are the best alternatives if you want to come out with a "professional" output that could help you to better present your internal hardware purchase request; please keep this in mind throughout the document. These approaches however could have a few drawbacks:

• They are time consuming. No matter what, it takes time to gather the data and analyze them to come out with a proper sizing (professional tools can help a lot here)
• They are expensive. If you want to use these professional tools and/or consultants to do this, it will cost you some dollars/euros to come out with that magic number.

There is no free lunch. The more professional you want to go... the more expensive it gets.

Having this said I'd like to offer you a "non-professional way out" so to speak. But before we get into the details of my methodology (ITIL practitioners might want to kill me for calling it a methodology) I want to set the stage for it. My sizing suggestions are going to be somewhat weak if you don't take into account the following concepts:

• Taking a "snapshot" of a physical environments is certainly time consuming but also difficult to achieve. By definition x86 server farms are very dynamic in nature so it could happen, especially for big deployments, that as soon as you are done with accounting the last server in your study, the whole picture has changed (sometimes dramatically) with new servers being introduced and old servers being removed. It's like taking a picture of 20 kids playing in a green field... you can be 100% sure that two snapshots are never going to be the same.
• Sometimes the performance effects that virtualization introduces are not predictable. There have been circumstances where a given application that was consuming very little resources on a physical system started to drag lots of CPU cycles once virtualized for no particular reasons. It is not so easy to develop an algorithm that will take as an input the physical resource utilization and translate them into virtual resources utilization simply because the effects of this thin layer are still very unpredictable in some cases.
• Another thing to keep in mind is that over-provisioning of hardware resources might turn to be cheaper than buying consultants and/or tools to calculate in a more professional way the exact size of the resources you need. The idea here is that if you take an educated (and conservative) guess on the number of systems and their configurations that you would require to support your own server farm (and you add a certain % of resources for contingency just in case) the total amount of money that will be spent on the infrastructure is (likely) to be less than what you would pay for a proper sizing consultancy + a more precise sizing of the infrastructure. Am I suggesting consultants are a waste of money? Not at all. They would provide you with a very professional study report that you can bring to your management for the budget approval of new servers and new infrastructure software.
• I have provided myself quick and dirty suggestions to customers and IBM business partners where I have taken educated guesses on the sizing of a target Virtual Infrastructure and where a proper professional study report was not strictly required. And in those situations, back to the over-provisioning discussed above, I have never come across a customer that was not happy because his/her systems were only used at 45% and not properly sized to run at 65/70% of resources utilization (especially in those circumstances where companies had to expand and could put those spare resources at good use in a short period of time).
• How much would you want to stress your physical systems anyway? Is 65/70% of CPU resource utilization a reasonable target number before "moving to the next" box? Or is 50% a better number? Or can you push it close to 90/95%? Also what are the effects of response time of applications running in virtual machines when the resource utilization of the host goes up? This CPU-centric measure of course assumes that you have enough/balanced memory in your system (you will never get to 65% of CPU utilization on a 4 socket systems with just 4GB of memory for example). These are however road-blocks during your professional studies: even if you take into account all the details regarding the actual resource utilization of your physical systems there is no magic rule at this point at least (and that I am aware of) that will be able to provide you with a definite answer on the target resource utilization for a given virtualized host before you start to see degraded performance.
• Most of the time one should concentrate around CPU and Memory configurations for sizing hosts that support virtual machines. This is not because other subsystems are not important obviously but sizing a storage subsystem is a complete other matter and certainly out of the scope of this post. By the way since a virtual infrastructure allows you to decouple physical computational resources (i.e. the servers) from the virtual machines hard drives (i.e. the centralized storage) you can size the two components separately. Sizing the storage will need to take into account things like overall space required, proper raid levels and size of the logical units (but as I said this discussion is not within the scope of this post). Typically two standard HBA's in each of the servers comprising the virtual infrastructure have enough bandwidth to support the traffic to and from the SAN (no matter what the sizing of the centralized storage has been). Similarly for the NIC's it shouldn't be a problem given the fact that most of the time the number of NIC's in a given system is a function of the complexity of the networks you need to connect your hosts to rather than a performance sizing exercise. That's the reason for which the methodology below will focus on CPU and Memory sizing: once you have determined the # of CPU's and total amount of RAM to be used you can stick with two SAN HBA's per server and as many network connections you need (as I said the choice of the network subsystem configuration is more related to the virtual infrastructure architectural design rather than a pure sizing discussion).

At this point you might consider this a weak argument and certainly not a very precise study on how to properly size your servers to support your VMware project. And that is exactly what it is: an educated guess based on "not actually measured data". So what is this all about? It is very simple and straight forward. Again, not very professional, but easy, quick and most of the time close to what a tool or a diligent study would tell you in a professional report.

So for those people that do not have time and money to get on the "professional band wagon" my suggestion is this: instead of starting from an actual measurement of your physical servers and then analyze your data to "engineer" a proper sizing, why not taking the other way around. Why not leveraging industry "rules of thumbs" reverse-engineering what other customers have already experienced world-wide and apply this to your own scenario? We are talking about two very simple and straightforward data-points here:

• Rule of thumb #1: Every brand new Intel/AMD core (or pCPU from now on) can support on average between 3 to 5 virtual CPU's (or vCPU from now on).
• Rule of thumb #2: Per every brand new Intel/AMD core configured you should have between 2 and 4 GB of RAM to obtain a "balanced system".

That's pretty much it. Most VMware customers using VI3 in production world-wide will probably tell you that, no matter how they got to their setup (i.e. through a pilot, an educated guess, a consultancy, a tool analysis) they will likely fall into the rules of thumbs above: they are all supporting 3 to 5 vCPU per physical core and their servers have between 2 and 4 GB or physical memory installed per physical core. Consider that the 3-5 is an average. I know customers in extreme situations that have 2 vCPU's per core and others that have 8 vCPU per core. 3 to 5 is pretty conservative and, unless you are in such an extreme situation, the worst that could happen is that you are over-provisioning your new server farm (see above).

So why not getting straight into an example which I am sure will clarify the whole thing. Say you have for example a physical server farm comprised of 60 physical servers. You are going to virtualize 55 of these 60 (this can be for multiple reasons). Of these 55 servers, 45 are going to be 1 vCPU virtual machines (this can be either because they were already 1 pCPU servers or because they were SMP servers but the resource utilization is so low that you can afford to migrate them into a 1 vCPU sand-box). 10 of the 55 are going to be 2 vCPU virtual machines.

So let's do some math. The total amount of vCPU's that you are going to activate is 45 x 1 vCPU + 10 x 2 vCPU = 45 vCPU + 20 vCPU = 65 vCPU's

Let's now try to see how many cores you need to support this workload applying the first rule of thumb: 65 vCPU / 3 = 22 cores (rounded). Consider that I wanted to be even more conservative and based on the generic "3 to 5 vCPU per core" I wanted to take the more conservative 3 vCPU per core. If I wanted to be more aggressive I would have used 5 vCPU per core so the math would have been 65 vCPU / 5 = 13 cores. This might be possible but we want to be conservative in such an "educated guess" so I would stick on the 22 cores.

Now, given that quad-core CPU's are widely available, we will calculate the number of actual "CPU packages" that we need to support this virtual infrastructure: 22 cores / 4 cores = 6 CPU packages (rounded). In terms of memory we would need 22 cores * 4GB = 88GB of memory. Again here we have taken a very conservative approach using 4GB per core Vs 2GB per core (see above rule of thumb #2).

So in the end you can assume that in order to support your new 55 virtual servers above you need 6 CPU packages with a total of (roughly) 88GB of memory. Mapping this exercise to actual physical systems is not that difficult: most likely you might want to use 3 x dual-socket rack based systems or blade systems with 2 x quad-core CPU's and around 32GB of memory.

A word of caution should be mentioned for the memory configuration. This methodology requires a bit of "business sense" and the numbers should not be treated as a given; for example due to high memory costs it might be even possible (and cheaper) to buy 4 x dual-socket rack / blade systems with 24GB or even 16GB in each. 16GB per each of the 4 systems is 64GB of memory which is anyway certainly above the original thought if we were to use a slightly more aggressive formula with 2GB of memory per core (22 cores * 2 GB = 44GB).

Another point of attention is the CPU "sku" (or model) to be used. There is typically a wide range of processor models within a given family (for example within the Intel 53xx family or the Intel 73xx family). There is usually also a big price fluctuation between the low(est)-end model and the high(est)-end model within the same family. Given the nature of this methodology it would be difficult to suggest the right model to be used for a specific scenario but a good business/technical practice would be to use the "n-1" or "n-2" models with "n" being the high-end sku. Usually the high-end model is optimal for "raw performance" while the n-1 and n-2 sku's are optimal for "price/performance" metrics.

Again it needs to be stressed that this is a high-level educated guess on how much computational power you would need to run your projected virtual servers. As I said at the beginning this methodology does NOT overlap with a more professional approach. Having this said however the same business sense should be used anyway when interpreting suggestions gathered from professional IT tools or consultants that might or might not have a deep understanding of the x86 hardware market dynamics (in terms of pricing).

Two more things before we close this topic.

You might want to consider having an additional "building block" (i.e. server) in case one of the systems that have been sized accordingly to the methodology fails. So in the example above, assuming to stick with the 3 x dual-socket with 32GB of memory, you might want to have a 4th server so that at any point in time you will always have at least 3 of them running even in the case one of them goes off-line. This is not mandatory but you need to consider that you would be running with fewer resources in case that happens.

The other thing worth mentioning is the fact that, as the number of the physical servers to be virtualized grows, the number of CPU packages and total amount of memory grow with it. It wouldn't be uncommon to require hundreds of CPU packages and TB of memory for a large scale project. If that is the case than another decision point is whether you want to scale out your new virtual infrastructure (i.e. with 2-socket rack / blades) or scale up (i.e. with 4-socket and above high-end systems). This is a pretty old document that I wrote on the subject that might help you with that decision. As I said it's a bit old but most of the considerations still apply.

And with this I think I am done. I just want to wrap up stressing again on the fact that:

• this is far from being a professional approach (I know it very well)
• yet it's a quick and dirty methodology that will get you "there" anyway.

Long live the sizing tools. Long live the sizing consultants.

Massimo.

"I need to setup a VI3 solution. Which CPU should I choose from a performance perspective? Intel or AMD?". I have heard this ad nauseam. If I was to get one single cent for every thread discussion I have seen on the net regarding the matter... I would be a billionaire. I have my opinion on the matter and I want to share it with you with this post.

Background

Intel and AMD are the two leading providers of x86 processors respectively with the Xeon brand and the Opteron brand (in the server space). This is not an introduction to what they do (Wikipedia might be your place if you are looking for that). At the time of this writing it must be noticed that Intel has introduced quad-core CPU support some 12 months ago more or less for the Xeon 5xxx family (previously Xeon DP for dual processor systems) and they have just announced quad-core CPU support for the Xeon 7xxx family (previously Xeon MP for multi processor systems). AMD has only been shipping dual-core CPU's so far and they have too just announced quad-core support across all their Opteron families (which includes 2P and 4P+ systems). So we will assume, throughout the discussion below, that they both provide quad-core CPU technologies.
 

Straight to the point, Xeons and Opterons have very radically different architecture implementations: Xeon has historically relied on the concept of the "Front Side Bus" where the sockets connect, through a bus, to the external memory controller. This picture should clarify the concept:

This architecture is referred to as a UMA architecture or Uniform Memory Access in the sense that each socket can get to any memory location in a uniform way (in terms of latency primarily). Consider that the latest chipsets now provide dedicated connections to each socket (as opposed to the shared connection that has been summarized in the picture), however it must be noticed that now 4 cores share the same socket thus increasing the level of contention.

On the other hand the Opteron processor has a very different architecture which is based on the concept of an integrated memory controller so that each socket (and memory controller) connects to the other sockets (and memory controllers) by means of a direct Hypertransport. Isn't it "cool"? Look at the picture below.

This architecture is referred to as a NUMA architecture or Non Uniform Memory Access meaning that, depending on where the data in memory is, a given CPU core could have a different latency to get to it. For example if the data that a CPU core on a given socket needs to access lives on a piece of memory directly connected to the integrated memory controller, access to it will be ideally very fast; if the data lives on a remote memory bank (i.e. one connected to the integrated memory controller of another socket) memory access will be slower.

You can imagine the memory access latency of a UMA system to be somewhat in the middle between the local memory access and the remote memory access of a NUMA system.

These are the facts. On top of this high level differences there are other important differences in the way Intel and AMD has implemented their solution. For example Intel has always been forced to use lately high capacity (dedicated and shared) caches among the cores to overcome the Front Side Bus contention inefficiencies. On the other hand the AMD architecture has not historically required high capacity caches due to their design to achieve comparable performance results.

How all this affects VMware ESX performance? Not too much in my opinion but more importantly not at the level for which you should bother. Let's see why.

Should I bother? I don't think so

My opinion, and my first rule of thumb, is...  "number of cores being equal, if you spend more than 5 minutes worrying about these details you are basically wasting your time". This is not because raw performance is not important but simply because these guys are pushing what you can get out of the silicon to the limits (no matter what the architecture is) and it is very difficult to determine which one is faster over the other nowadays. There are market niches where AMD is still king of the hill (particularly those HPC memory intensive / floating point type of applications - sorry not an expert on the matter) but in general we are talking about commercial workloads running on top of a virtualization software layer which introduces a very unknown effect on performance.

Which leads me to my rule of thumb #2 that is... "if you meet/hear someone with a very specific idea of AMD being definitely better than Intel in a VMware environment or Intel being definitely better than AMD, disregard his/her opinions". Virtualization has radically changed the way we build our IT and you need to understand that all x86 server hardware has been studied and developed with the simple paradigm of 1server-1OS-1Application in mind. Every single piece of the various subsystems (cache levels etc etc) has been tuned to achieve the maximum performance, over the last few years, for that legacy context. When you scramble all that with the new paradigm that is 1server-1hypervisor-nOSs-nApplications you are basically using a hardware platform (including the CPU) that has been tuned for the last decade to do something else. There are a very small set of people in this industry (I would say perhaps 30 or 40 and they typically work for CPU vendors, systems vendors and virtualization software vendors) that are working day and night to understand the effects that virtualization is having on the current generation of hw systems. And quite frankly the most extreme comment I have heard from them was a "for this particular workload running in a virtual machine, we have noticed that <put your processor of choice> has a slight lead over <the other one>". I have never heard one of them stating "Intel is definitely better than AMD" or "AMD is definitely better than Intel". So my rule of thumb #2 could also be read as "if you are not talking to one of these 30 - 40 guys .... don't waste your time".

Without getting into the problem of religious wars which I think are pretty ridiculous, the problem with people claiming very definitive statements of one being better than the other is that they are convinced of what they are saying. Most likely they have both technologies running in-house and based on their limited experience (not limited time wise - but limited projects-wise) they claim that "based on what they have seen" A is better than B. The problem with this is that it is ENORMOUSLY difficult to run a benchmark that allows an apple to apple comparison between two different systems to determine which one is the best. There are dozens (perhaps hundreds) of variables in such field experiences / benchmarks that are just not valid scientific method to valuate such a complex matter. And I don't even want to talk about people that benchmark their 4-socket enterprise servers running a file copy or using SiSoft Sandra (which would be like benchmarking the speed limit of a Ferrari with the test taking place in Milan downtown at 8am ... what would you expect? A Fiat could turn out to be faster than a Ferrari given this variable). I want to talk about subdle variables that might lead you to think something that is not real in facts. A good (real) example would be a customer of mine complaining about a slow application running within a vm. He was complaining about the fact that the batch job running inside a vm would take twice the time to complete compared to the same job running on a similarly spec'ed physical system. It turned out that running the job at different points in time during the day provided (very) different results. This was nailed down to be an environmental problem (such as SAN and network utilization at different hours) but let me tell you: 99% of the people testing an AMD based system at 8am in the morning with this batch (elapsed time: 2 hours) and an Intel based system (same config, same everything....) at 11am (elapsed time: 3 hours) would definitely point their browser onto the VMware forum with a nice post that says "I know for sure that an AMD system is 50% faster than an Intel based system". Missing the point.

However I am not saying that both AMD and Intel are equal performance-wise. What I am saying is that:

• We know so little about the effects that virtualization is posing on the hardware (we are just scratching the surface now of these implications)
• Both Intel and AMD are pushing the silicon to the extreme of laws of physics
• Both Intel and AMD are leapfrogging each other every 6 months

... that to me it is a waste of time discussing which one is better than the other. What you need to keep in mind is that they are damn good pieces of technology and you can't go wrong with either one or the other.

This leads me to the "IT artists" which is the extreme, in my opinion, of these "home-made experts". The IT artists are those that argue AMD being better than Intel because of its "native quad-core design". As an IT guy I am interested in three things CPU-wise:

• Absolute performance
• Price
• Power consumption

Quite frankly how AMD or Intel is giving me the above... is a technical detail I am not interested in. The IT artists however tend to articulate that AMD has no FSB and the way Intel can keep up in terms of performance is using big caches. This is very true but the real point is: as long as Intel is giving me that big cache at a fraction of the costs AMD would need to charge me for... should I care? I don't work for the fashion business so I am not interested in how "elegant" the AMD design is Vs the "cumbersome" Intel design. What I want is highest performance, at the lowest price, at the lowest power consumption. Period. I am not bashing AMD here to promote Intel. I think AMD is doing wonderful things to make this x86 platform progress as I tried to point out here but the problem is that it seems that "fashion" is easier to sell and market than the "real" stuff. 

And before you get me wrong I am not saying that an integrated memory controller is not a smart choice (Intel will get there anyway) and certainly those 30 - 40 people above could entertain us some 2 days talking about the performance implications of having a FSB Vs an integrated memory controller. But they are not IT artists, they start from the layout to articulate from a scientific perspective why and how it affects performance. IT artists just look at the layout and say "the layout is cool so it must be better".

Rule of thumb #3 is "when you meet an IT artist, don't pay too much attention to what she/he says". 

Conclusions

I like to close with an analogy here. Think at the next purchase of hardware as a journey from Milan to New York. Where the vendor you are buying from being the airline, the aircraft you are flying with being the system/server and the engines of the aircraft being the CPU's.

So my question is... when you need to buy a ticket... do you start from which engine provides the best performance to search then for aircrafts that use that engine to determine which airlines use those aircrafts... and you buy a ticket from them? Well... no offense but if you do, I think you need a doctor.

When I buy a ticket I care more about the airline that provides me the best service (at the right price obviously). I want to buy a ticket from an airline that appreciates my business, that is there to support me if I need it, that is able to make my life easier and my journey better if I go with them. This includes having chosen the right aircraft for that journey for my comfort. There might be instances however where, even a good airline, is using obsolete and odd aircrafts and, although they are alright and supportive as an organization, you might want to fly with another company just because they have better aircrafts. What I am pretty sure of though is that if the airline suites your need and they are using good and comfortable aircrafts that meet your standards... there is no way you don't want to fly with those just because you think they are not using the "coolest" engines. After all, assuming you can get to destination 20 minutes in advance with the cool engines, your journey might be a nightmare anyway (especially if your take-off gets delayed by 4 hours ... what are those 20 minutes going to buy you in the end?)

What I am trying to say here is that when you decide to buy a new server from a system vendor you should first look at the commitment that this system vendor has with regards to your business (that is virtualizing your system). A good system design is of course important and needs to meet your standards and requirements. However good management tools, an understanding of what your objective is and the capability to complement a hardware sale with the know-how to implement what you need are other important characteristics along with a vision on where this industry is going so that the vendor can help you take the right decisions. A vendor or partner that you trust basically.

Such systems vendors have done their diligent homework to understand what CPU to use in any given system and you should trust their choice (like you trust Airbus or Boeing for having chosen a specific engine for a specific aircraft). After all you need to remember that what a system vendor wants to do is to sell you something that performs well at the lowest cost and lowest power consumption. In the final analysis they want happy customers otherwise customers will not come back. 

Don't waste your time on the engines: it's a detail of a much bigger picture (and the detail right now is not even important given that the two options are excellent). Look at the forest not at the tree. When booking a journey, ask yourself which frequent flyer card you should apply to ... not which engine is faster.

Massimo.