High Performance Compute as a Service (HPCaaS)

The mainframe days of yore, punch cards and time sharing. An ACE station with twin Control Data computers. (NASA photo 107-KSC-67C-919)

Continuing on my random ideas post. Here is another item that can be applied to High Performance Compute (HPC) scenarios. With research institutions and the like it is often necessary to wait to use physical HPC clusters to perform research. (It hearkens back to the days of mainframes and the stories I heard of having to get up at 2am for your slot on the main frame.) What if it could be virtualized and the clusters built out on demand? Maybe a deliver HPC as a Service (HPCaaS)? This would speed delivery time, make HPC resources more accessible, and facilitate more dynamic HPC environments.

Hopefully if you’re this far down you’re thinking HPCaaS would be pretty cool. How would one go about putting this together?

There are a few ways to do this and I’m going to use ArcGIS in my example as its the one I am most familiar with. It also fits nicely into my wheel house.

One way to deliver HPCaaS is to build a cloud service around it using vRealize Automation and Orchestration. For example Enterprise Hybrid Cloud (EHC) which is an offering from a federation of companies, one of which I work for (EMC). This can be a very powerful solution, allowing the deployment and configuration of storage, compute, etc. It can stand up ArcGIS clusters and the other components needed. You can even leverage VMware NSX to dynamically instantiate your load balancers.

This makes it possible to standup and configure clusters like the one below for an ArcGIS deployment. You can standup the NLBs, Web Adapters, cluster A or B, etc. Essentially you build out the components once, create blueprints and service catalog(s) around it. Then enable users to request the clusters they need.

Multiple-machine deployment with GIS server clusters (ArcGIS Pro)

Side note: Now it should be noted that, its often necessary to think about this outside of how virtualization is traditionally thought of. Instead of consolidating and “sharing” performance capabilities of x86 hardware we are just creating a bigger pool of resources. In an HPC environment The goal is not to squeeze every last drop of computing power into as many servers/desktops as possible. Instead its to provide the resources needed. So if its only possible to get a couple of HPC systems on a single ESXi host that should be fine and expected.

Creating HPC clusters this way works pretty well. Only it can become complicated when you start adding requirements like GPU’s to the mix as well as those pesky end user devices. VMware Horizon View can take care of the end user devices. Now what about the GPUs?

You could also build clusters leveraging GPUs with Horizon View. Now it should be noted that “Windows 10, 8.1, 8, and 7 are supported for basic testing and application development use only. They are not recommended for deployment in a production environment.” (see note here) I haven’t tried deploying server based OSs with GPUs from Horizon View. It may or may not work. As for a Desktop OS, deployment is a breeze and it can be incorporated into vRealize Automation and Orchestration. This provides the expanded capabilities of leveraging GPUs for HPC.

How is automating this done? There are two methods to deliver desktop OSs as part of an automation solution. First you can leverage the vRealize Orchestration Plugin for Horizon View. It has several workflows in it and can enable a service catalog for desktops. The second option is to use PowerCLI and create your own set of Cmdlets that you can use with or without vRealize. You can find more about this in the Horizon View documentation here.

This brings together all the parts needed to create a HPCaaS system allowing users to visit through a self service catalog and request a solution that is “tailor made” just for their HPC needs and can be stood up in a fraction of the time that is needed to manually build all of the components.

If you would like to know more about this or have questions please leave a note in the comments section.

Hope you enjoyed this random musing,

Tony