Back to the "Resources" page on it20.info

Sample VDI project web log page  (last updated 28th Oct 2008)

This is intended to be a "notes from the field" document. At this point in time the customer name can't be disclosed but it will suffice to say it's a hospital. Before you read this document I strongly suggest that you go through this presentation for a background on the topic.

 

Background

The back-end is comprised of various Windows and Linux boxes running all sort of applications from infrastructure to vertical home grown applications going through mail servers etc. Most of these servers are already virtualized using VMware technologies.

There are around 3000 (more or less) PC's (and at a less extent laptops) distributed around the region (the hospital has a main central site and then they also have satellite sites throughout the territory). All of them are Windows XP personal computers. Other than standard productivity tools such as Office and the sort they do have lots of home-grown (and vertical to their "business") java-based applications. They are using Citrix Presentation Server to distribute some of these applications with limited success (CPS is seen as "too cumbersome" anyway and not strategically viable for a much larger deployment within the organization).

Currently every user has his/her own desktop (less often a laptop) assigned. While there is a central repository concept (i.e. file server) the user is entitled to save data on his/her own XP image. There are no roaming profiles / folder redirect concepts implemented at the moment. 

Another important thing to keep in mind is that, due to their legacies, each PC is configured with different hardware options. Specifically every PC being deployed is sort of unique in its configuration when it comes to locally attached printers. One PC might have a USB specific printer attached, another might have a USB and a parallel printer and yet another one might have a network (IP) printer configured that they share in an office comprised of 5/10 end-users. It is key to understand that these configurations are specific to the computer and do not follow the end-user (this is obvious since the printers are either attached or in proximity of the PC itself).

While they do have an active Windows Active Directory domain they logically attach their PC most of their vertical application packages have their own login. I have found this to be pretty common in the healthcare business to some extreme situations where the PC would autologon (or where users would logon with an "office id") and then they would login with their personal user-id directly into the application. 

 

Challenges / Requirements

The customer was looking for an alternative to their standard PC deployments. They are having multiple issues with their regular desktops. Some of them are:

These are some of the facts. More generally they are under pressure to find a solution to the problem of the rising costs of managing such a big desktop deployment. Hardware and Software maintenance has become a real burden for them and the complexity of the software stack being used at the moment on their client devices is not going to be simplified soon, perhaps it might become even more complex in the future (thus making it even more challenging to implement / support).

 

The idea

It has become very clear immediately that what they need to do is to simplify their distributed environment centralizing, as much as possible, their complexities. Years ago they tried to fix this problem with Citrix Presentation Server but they found a few problems/limitations. In fact it is a good tool to centralize the management of some applications (not even all of them) but it has limitations when it comes to centralize and "present" to the end-user the entire desktops. This is a limitation on multiple fronts (i.e. some applications requires an XP client machine for example). Also Citrix Presentation Server, as I said, has always been perceived as a "heavy tool". As a result of this alternative technologies have been taken into account including workstation blades (aka PC blades). In this case the cost to centralize roughly 3000 units resulted prohibitive from a preliminary business case. On the other hand it appeared clear that a hybrid solution like "hosted desktop virtualization" (aka VDI - Virtual Desktop Infrastructure) could fit better the customer requirements and objectives.

 

The philosophy (phase I)

In Phase I the customer purchased a solution for deploying 150 new end-user seats in a VDI scenario. Basically instead of buying 150 new standard Windows XP Desktops the customer bought a solution comprised of various technologies to provide the same service to their end-users in a completely different and innovative manner. We can summarize these technologies into 3 major components:

  1. Back-end servers (hosting platform)

  2. Connection Broker (access logic)

  3. Thin Client (access device)

Regarding the hosting platform we opted for a Bladecenter solution with VMware ESX. We decided that a scale-out solution using a blade form factor was the best price/performance/feature compromise for such a (potential) massive deployment. In fact VDI deployments tend to be very resource-hungry in terms of CPU and Memory (not because a desktop consumes a lot of resources ..... but because when you have 3000 of them ... they do!). Blades are in this case a good choice because they allow you to have a great deal of hardware resources at a very appealing price and built into an easy to manage package (i.e. the Bladecenter chassis). Consider that for VDI deployments, although the TCO savings associated to it are enormous, acquisition price still play an important role in the business case so with a scale-out approach a customer might be able to cut some premium hardware expenses (this is different from a server virtualization project where the price per virtual server is not as important or evident as the price-per-user is in a VDI project). This of course doesn't mean that high-end servers are not a good fit for VDI deployments (every situation is different). One word on the virtualization software: while VMware VI3 at the moment is positioned as the best option, this market is getting very fluid so it will be interesting to evaluate the best platforms in terms of feature/cost/maturity to evolve this deployment in future phases.

Regarding the "access logic" we opted to use a VDI connection broker. Using a one-to-one mapping (i.e. an access device pointing statically to a Windows XP virtual image) was not an option given the management burdens of dealing with (potentially) 3000 individual desktops (even though centralized, which is better anyway than dealing with 3000 individual desktops distributed). The choice here has been to implement a VDI connection broker to leverage the pools capabilities. The idea is to segment a number of homogeneous Windows XP images (i.e. deployed from the very same template) and assign them dynamically to an Active Directory group. Users checking in with the connection broker would be temporarily assigned a virtual desktop from the pool they are entitled to use and at logoff the virtual desktop would be put back again into the pool for the next user checking in. This is an important step for the customer since it involves moving from a concept of "personal desktops" onto a new concept of "shared desktops". This involves a bit of re-working of their Microsoft infrastructure in terms of Windows user profiles and the like. The product we have used for this layer is the Leostream connection broker. It is to be noticed that we have started the preliminary discussions and pilot in the summer of 2006 and at that time there were not too many choices. Having this said the customer immediately liked the fact that the Leostream broker was a very "agile" piece of software (as opposed to the feeling they had about the Citrix Presentation Server). Sure they do different things but in the final analysis Leostream for this customer specifically was exactly what they were looking for (i.e. a very slim component/sw that would allow to create a pool of desktops and assign them to an AD group). While there is no plan at the moment to change Leostream with another broker we will keep an eye on the market to evaluate alternatives. 

Regarding the access device, the philosophy behind this project that is driving the customer (and me as an external consultant at an extent) is that once you decide to centralize your desktop infrastructure you can't afford to centralize the complexity of the software stack YET having to deal with distributed intelligent devices. Let me rephrase: if you decide to bring into your datacenter (potentially) 3000 XP images, you'd better have 3000 "dumb" devices in your distributed locations otherwise instead of managing 3000 very complex objects (i.e. the distributed computers) you end up managing 6000 somewhat complex objects (i.e. the 3000 centralized virtual XP images + 3000 access devices). They key here is to get rid of the management associated with the 3000 access devices and thus the choice was to use one of the thinnest thin clients available in the market (i.e. the Wyse S10). The Wyse S10 uses a "proprietary" Wyse operating system called ThinOS that, as the name implies, it's a very thin piece of software that gets loaded on the device (it used to be in the 1-2MB range now getting into the 3+ MB as more functionalities gets added (especially when it comes to the new TCX Wyse extensions - please refer to the Wyse web site for more info on these). Consider that the S10 is not the only ThinOS based device available: Wyse has just announced a brand new model called V10L which uses the same base software but can take full advantage of these TCX functionalities.

Back on track though, the point is that we wanted to implement access devices that were not drastically more difficult to manage than a power-let or than a Cisco IP phone. You plug it in, it will download any necessary firmware update and specific configuration information from a central repository and "it just works". Notice that you could implement something like this even with CE-based, Linux-based or XPe-based thin clients but my personal rule of thumb is that, as soon as the size of the logic you get on the access device increases, the management burdens increase (not even proportionally but exponentially). For what it matters Microsoft would also argue that you can do this (i.e. plug&play and "it just works") with standard Windows deployments tuning properly AD policies and leveraging other technologies. However we all know where we ended up with Windows XP deployments in terms of management issues.

Well, anyway we went for the S10 as we found it a good compromise and that is what we are using for this deployment. We might consider alternatives in the future.

 

The implementation (phase I)

We are currently deploying 150 seats based on the solution and components mentioned above.

Regarding the hosting platform we opted for 2 x Bladecenter chassis with 5 x HS21 blades in each for a total of 10 blades. Each HS21 blade is configured as follows:

This is a generic picture of the chassis we have used:

We decided NOT to use any shared / external storage in this configuration. Since the architecture would be designed around the concept of "pooled XP virtual desktops" this allowed us to use alternative methods to provide high availability and scalability. We have designed the solution to be "logically" fault tolerant rather than being "physically" fault tolerant. While the customer does have an Enterprise Storage Area Network as well as a mid-range NAS server device available it was a thought decision not to use any of those. One of the key reasons for this was because the customer wanted to create a self-contained environment for this project without leveraging any other external existing technology in their datacenter. This also allows for a complete independence in terms of costs and performance of the virtual desktop environments (connecting it to the SAN would have had higher costs implications and connecting to the NAS would have had <potential> latency implications). I am not implying that this is the right way to implement a VDI solution but it certainly was for this customer given their organizational requirements.

This picture should outline the relationships between the blades, the storage, the vm's and the various VDI pools:

Particularly the problem of latency and performance is important. Since there are no baselines it would be difficult to determine the effects of this solution in terms of I/O requirement. It is certainly true that a standard desktop does not generate a lot of disk and network I/O generally speaking but the variables here are multiple. Some of them are:

In short it was decided to start this deployment with local hard disk drives for this initial phase (phase I). Based on the results we will be gathering in terms of usage patterns and resources utilization we might be able to "adjust" the architecture for future deployments. While I am not ruling out the usage of their existing enterprise SAN or NAS based devices, future alternatives that we have already discussed with the customer include attaching either a low-end SAS based SAN or a low-end NAS device to each Bladecenter chassis. This would allow to keep acquisition costs low as well as to keep the element concept we have used here which can be treated as a sort of self-contained deployment unit to support "n" VDI end-users.

Another thing worth noticing is that currently each Bladecenter chassis is configured with 5 blades for a total of estimated 75 XP guests running. It was determined with a preliminary assessment that a single (yet redundant) gigabit uplink could be used to connect the chassis to the network backbone.

VMware VI3 Standard has been activated on all 10 blades described above. A Virtual Center instance mapping these 10 servers has also been installed and activated. This VC is running as a virtual machine on a separate (existing) VI3 Enterprise infrastructure.

Regarding the "access logic" we have decided to use the Leostream Connection Broker in a high-availability configuration. The broker is a very important piece of this solution both from a functional and non-functional requirement (if the broker service is not available the end-users won't get any desktop assigned). The broker in subject comes as a preconfigured VMware virtual appliance which includes the broker logic as well as a local mySql database for persistency. Now Leostream supports multi-tiers configurations with an external database (SQL 2005) and many connection broker instances attached to it. So the steps we went through were:

  1. installed a SQL 2005 instance in a dedicated Windows 2003 virtual machine

  2. installed the first Leostream CB instance and externalized the database onto the SQL 2005 above

  3. installed the second Leostream CB instance and attached to the same database above

  4. created a cluster/unique IP address on a hardware load balancer to include the two IP addresses of the broker instances

The three virtual machines above are hosted on a separated VMware VI3 Enterprise infrastructure. The two connection broker instances are highly available by design (i.e. if one fails the other can continue to serve requests). As per the SQL database we decided to leverage the HA feature of the VI3 infrastructure it's running on. This feature along with regular backups of the SQL database was considered the best approach to take. Also, having started with a multi-tiers setup, this allows us to scale ("scale out" with the CB instances and "scale up" with the SQL backend database) without changing the architecture of the solution.

There have been discussions underway with Leostream to understand whether the Connection Broker service could catalog directly the standalone ESX hosts by-passing VirtualCenter which is a single point of failure. The problem is that, while the functionalities of a VMware Virtual Infrastructure could tolerate VirtualCenter to be down for hours / days, in VDI scenario VC becomes an incredibly critical piece of technology and if it's not available the whole round-trip that occurs from the end-users requesting a desktop all the way to the broker assigning one.... won't work. Unfortunately it seems (based on Leostream feedbacks) that VirtualCenter cannot be bypassed in this situation so this is the way it needs to be done. This of course assumes that VC is used to manage the ESX hosts. Since there is the possibility to catalog ESX hosts as "Centers" within the Leostream interface I will take that it is possible to do that when these ESX hosts are not managed by any VirtualCenter server.

This picture outlines the broker architecture we have implemented:

Regarding the "access device" the preferred choice for this initial 150 seats (phase I) has been the Wyse S10. As I have mentioned the S10 is a very thin device which simplifies the management of the distributed environment. An important characteristic of the Wyse S10 is that it gets configured via DHCP/FTP using a text base file. The flow seems complicated but it's not. Your DHCP server will provide each S10 with standard information such as the IP address, DNS server etc etc. On top of the standard options the DHCP should also provide the following information:

This is how you would deploy a number of similarly configured Wyse S10 devices. The problem is that in such a situation all the devices would get the same configuration (which includes video resolution, printer configurations etc etc). This customer has a constraint for which all devices (or almost all of them) need to be configured "ad-hoc". So how do you do that? In order to allow different S10's to get different configurations you could use another option on the DHCP server which is Option 162 which is the "FTP Root Path". Essentially what this option does is it will add to the standard directory where the default WNOS.ini file is an additional directory for the ad-hoc file. So if I set Option 162 to "Building1" this means that during the boot process the S10 will contact the FTP server but instead of looking into the standard Wyse root directory it will search for a "Building1" directory where it will download the WNOS.ini file (as well as the firmware if it exists). This is a standard manner for deploying "groups" of similarly configured S10 devices (i.e. for example Conf-A in Building1, Conf-B in Building2 etc etc). You would typically work in this case at a subnet level where each building (or floor) has its own subnet so that you can configure the DHCP server to use a specific value for option 162 per each subnet. However this customer required that each S10 had to be configured differently so what we initially thought to do was this:

Fortunately the customer was already using the reservation mechanism so this was not a big issue. Find below an example of the directory structure we have setup for this potential implementation:

\wyse\10.1.1.1\wnos\wnos.ini

\wyse\10.1.1.2\wnos\wnos.ini

\wyse\10.1.1.3\wnos\wnos.ini

Each of the wnos.ini files are different and pertains to each of the specific device (in this example there are 3 devices considered with IP addresses respectively 10.1.1.1, 10.1.1.2, 10.1.1.3). It must also be considered that device upgrades (ThinOS upgrades) are performed placing the new firmware image (called RCA_wnos) into the same directory where the wnos.ini file exists. The device at boot time always check if there is a file called RCA_wnos in the directory and, if it contains a firmware that is newer than the one on-board, it will download and installs it automatically. This is one of the great things in terms of TCO and thin clients. An upgrade of 1000 distributed devices only requires that you copy the firmware update file into the proper directory and the thin client will self-update it at the next reboot. However in our situation this has a negative side effect: having a directory per each device the administrator is supposed to copy the RCA_wnos file in each of these directories. This is not only a waste of hard disk space but it could lead to management problems. To overcome this we have tested the usage of hard links. Basically you maintain the file once in a "template" directory and you create "hard links" to that file in each of the directories where the wnos.ini file exists. This way, when there is a new S10 firmware update, you don't have to copy it into each directory, you will only have to copy the new file in the template directory and all links will point to the new file. We have tested, on a Windows host (using the fsutil hardlink create command) and it worked flawlessly.

Update: We later on found out that an alternative method to provide customized parameters to each of the S10 could be used. The latest S10 firmware versions support the variables include=$IP and include=$MAC in the WNOS.ini file. The idea is to have a single WNOS.ini file with global parameters that each S10 would read. Underneath the global "wnos" directory which contains the single WNOS.ini file you can create an "include" directory which contains a file named after either the MAC address or the IP address of each S10 device with additional wnos-like parameters. This is the new structure that we currently decided to implement:

/wyse/wnos/wnos.ini                                         <- this is the main WNOS.ini file that each S10 will read at check-in
/wyse/wnos/RCA_wnos                                    <- this is the firmware update file that each S10 will use for updates
/wyse/wnos/inc/10.196.86.13                           <- this is the file to further customize the S10 device with IP 10.196.86.13
/wyse/wnos/inc/10.196.86.14                           <- this is the file to further customize the S10 device with IP 10.196.86.14

The main wnos.ini file would look like this:

PRIVILEGE=High LockDown=no
VNCPrompt=Yes Reject=30
EnableLocal=yes
Language=it
include=$IP

(notice the "include" statement that makes the S10 checking in pointing to an additional file for further configuration parameters on top of the list of global settings above)

The file 10.196.86.13 (for example) would look like this:

Printer=LPT1 Name=STAMPANTE12 PrinterID="HP Universal Printing PCL 5" Enabled=yes EnabledLPD=yes
Printer=LPD1 LocalName=STP_PCL5 Host=hp4250-2-it.print Queue=LPT1 PrinterID="HP Universal Printing PCL 5" Class=PCL5 Enabled=yes
Printer=LPD2 LocalName=STP_PCL6 Host=hp4250-2-it.print Queue=LPT1 PrinterID="HP Universal Printing PCL 6" Class=PCL5 Enabled=yes
Printer=LPD3 LocalName=STP_PS Host=hp4250-2-it.print Queue=LPT1 PrinterID="HP Universal Printing PS" Class=PS Enabled=yes

While we still need to create additional customization files per each S10 we think that this is a better/easier manner to deal with these personalization compared to the usage of the Option 162 (FTPRoot) as described above. This is especially true for maintenance purposes where the single global wnos.ini file can help. Due to the fact that we reserve the IP address to each S10 in the DHCP scope we determined that the usage of the $IP variable fits more into our environment than the usage of the $MAC variable (i.e. if a device needs to be changed we don't have to touch the directory structure as we only change the MAC-IP reservation in the DHCP scope). 

It is also important to notice that Wyse does provide a management software for their devices. It's called Wyse Device Manager (aka WDM or Rapport). I should state clearly upfront that I am not a fan of this software for a number of reasons. Technically you could configure WDM to respond to S10 requests for WNOS.ini configuration files. Basically what WDM does is leveraging the FTP service underneath to serve the S10 with the same parameters (coming from the same WNOS.ini file) but with a slightly different interface. In order to do this you can configure Option 186 (which is the "Rapport Server") on the DHCP server. The problem with this scenario is that you have to create a DefaultDeviceConfiguration (DDC) for each of the S10's to be deployed given the customer's constraints (i.e. every S10 needs to have an ad-hoc configuration). Basically when you create / import a DDC configuration a new folder on the FTP server gets created. This time the association is done within the WDM database (based on a drag and drop of the specific S10 onto the specific DDC you want to use). So in a "raw" FTP scenario the flow would be:

While in a WDM scenario the flow would be:

To make a very long story short it was decided that while the "raw" FTP scenario seems to be more cumbersome (due to the Option 162 that needs to be specified for every MAC-IP reservation) it in reality more "streamlined" than using an additional layer of complexity provided by the WDM software. Basically the S10 devices get configured as in the "raw FTP" scenario (Options 161 / 162) but they would check-in into WDM service as well where they show up as registered devices (in this case WDM needs to be configured so that it doesn't push down the WNOS.ini file onto the device at check-in. See this article). Having the Wyse S10 devices checking-in on the WDM would allow the helpdesk to remote VNC into the S10 should there is any problem. However a direct VNC connection could be opened against the S10 device using the standard VNC client so the value of WDM used in this manner is questionable at the very least. As a matter of fact, due to this, the customer decided to build their own front-end for the help-desk for which they developed a number of simple front-end web pages that query the back-end WDM database (out-of-the-box SQL views has been used) and the Leostream database (custom views had to be created) and present a simple layout to help-desk where they can easily determine user name logged into the broker, IP / MAC of the device being used etc etc and obviously initiate VNC sessions.

Regarding the configuration of the ini files discussed above the Business Partner involved in this project created a simple but yet powerful web interface that allows the system administrator to create new configuration files or modify existing ones. This is a screenshot of the tool:

 

On a side note it is interesting to notice that I spent the last few months trying to understand how Wyse can solve this situation (i.e. dissimilar S10 configurations) by design. I came across other customers with different local configurations requirements (specifically when it comes to printers). Printers in fact are such a widely used asset and are also so bound to being locally configured (i.e. "per device" or "per office") that either S10 users do not typically print or these problems get solved in different ways every time.

The following picture outlines the overall architecture of the implementation:

The result (phase I)

As of October 2008 the customer has implemented the infrastructure described above with a great deal of satisfaction. They have been able to experience all the benefits briefly described in the overview at the beginning of this document ("Challenges / Requirements" section). The most interesting part of this is that, as they started to deploy this innovative platform, the customer found out other use cases where this would have made an excellent fit. Particularly they found this architecture was extremely useful to solve one the issues they had with providing (and maintaining) specific applications to family doctors that were using their own Desktops or Laptops. So this is was not so much about replacing internally owned Desktops with Wyse S10 terminals (the main idea driving this project) but it was more about providing specific applications to users that were using their own PC or Laptop. In this context they historically had to either install applications locally on the physical remote desktops/laptops (with all the challenges you might imagine) or providing these applications through alternative mechanisms such as Citrix Presentation Server which have always been seen as something that adds complexity (plus they have had some application compatibility issues that made them a bit nervous about this deployment method). What they found out was that, integrating Leostream with their Juniper firewall, they could allow the family doctors, working from their own Windows Desktops or Laptops, to VPN into their network, login into the Leostream broker and have the broker assign them a temporary VM from the pool equipped with the application(s) they need to provide the external users with. This has immediately solved the burden of installing the application locally on the users' physical devices as well as the burden of dealing with compatibility issues when virtualizing the applications with third party software.. Last but not least they have appreciated the fact that, with this architecture, they don't have to install ANYTHING on the external doctors' devices since the whole thing is agent-less and what they need to be available on the end-user's device is a browser and an RDP client, something that all Windows PCs have out-of-the-box. So the big picture now looks something like this:

Notice the philosophy behind this architecture: whatever falls inside my management boundary is (assumed to be the firewall boundaries) needs to be easy to manage and maintain (i.e. thin clients). Whatever falls outside of my management boundary could be whatever as I am not responsible for it (i.e. as I only provide "a service" to "a device").

A word on sizing and capacity planning. We have originally estimated that a blade configured the way it was configured at the beginning (see above the specs) could support an average of 15 VMs. It turned out that CPU was not the problem but memory was. As a consequence to that the customer upgraded each 4-cores blade (2 x dual-core) from 8GB of memory to 16GB of memory to create a more balanced system. We took a screenshot of one of these blades and this is how it looks like:

 

Note: all these measurements were taken at around 11am so they are pretty realistic regarding actual usage.

This blade specifically is running 25 VMs and, as you can see, CPU is very far from being the bottleneck. All these VMs have been assigned 512MB of memory so there is a bit of memory savings as the host is only using 10GB of physical memory. Memory saving here is not as big as one would expect although I am pretty sure I have seen other hosts in the same cluster doing slightly better than this. The current "bottleneck" at this point is storage space because with local DAS storage you can only support so many VMs before running out of physical space. This is a well known problem and in fact the next infrastructure refresh will account for a small iSCSI SAN to overcome this inconvenient getting rid of the DAS storage. Once we get passed this problem we assume we could get around, at minimum, some 50 VMs out of a single 2-way blade given the next wave of hardware they are going to use will likely have 2 x quad-core CPUs and a total of 32GB of memory. This is in general extremely interesting because, as the technology progresses and the prices remain substantially the same, the business case and the economics for setting up an infrastructure like this becomes more and more viable. The assumption here is that while the hardware requirements to run software will increase overtime, the hardware evolution and enhancement will increase at a (much) faster pace thus allowing always more workloads to be consolidated on a single platform, hence a (much) lower price per user.

Other than this the customer got excited about what could be done with this and started to customize some open-source tools and adapted them to manage their VDI infrastructure:

This specific example is a customization of MRTG (a tool originally conceived to graphically outline the activity of Routers); the customer is using its graphic engine to visualize the VDI pools statistics in terms of total # of VMs comprising a pool (the green area) as well as the # of VMs actually assigned to users at anu given point in time (the blue line). This in the picture is a 24 hours view.

Back to the "Resources" page on it20.info