Category: Tech Field Day


This is a blog post that I’ve had at the back of my mind for a good 6 months or so. The pieces of the puzzle have come together after the Gestalt IT Tech Field Day event in Boston. After spending the best part of a week with some very very clever virtualisation pro’s I think I’ve managed to marshal the ideas that have been trying to make the cerebral cortex to wordpress migration for some time !

Managing an environment , be it physical or virtual for capacity & performance requires tools that can provide you with a view along the timeline. Often the key difference between dedicated “capacity management” offerings and performance management tools is the very scale of that timeline.

clip_image002

Short Term : Performance & Availability

There we are looking at timings within a few seconds / minutes ( or less ) this is where a toolset is going to be focused for current performance on any particular metric , be it the response time to load a web application , Utilisation of a processor core or command operations rate on a disk array. The tools that are best placed to give us that information need to be capable of processing a large volume of data very quickly due to the requirement to pull in a given metric on a very frequent interval. The more frequently you can sample the data , the better quality output the tool can give. This can present a problem in large scale deployments due to a requirement that many tools have to write this data out to a table in a database – this potentially tethers the performance of a monitoring tool to the underlying storage available for that tools , which of course can be increased but sometimes at quite a significant cost. As a result you many want to scope the use of such tools only to the workloads that require that short term , high resolution monitoring. In a production environment with a known baseline workload , tools that use a dynamic threshold / profile for alerting on a metric can be very useful here ( for example Xangati or vCenter Operations ) If you don’t have a workload that can be suitably base lined ( and note that the baseline can vary on your business cycle , so may well take 12 months to establish ! ) then the dynamic thresholds are not of as much use.

Availability tools have less of a reliance on a high performance data layer as they are essentially storing a single bit of data on a given metric. This means the toolset can scale pretty well. The key part of availability monitoring is the visualisation and reporting layer. There is no point only displaying that data to a beautiful and elegant dashboard if no-one is there to see that dashboard ( and according to the Zen theory of network operations , would it change if there was no one there to watch it ! ) The data needs to be fed into a system that best allow an action to be made – even if it’s an SMS / Page to someone who is asleep. In this kind of case , having suitable thresholds are important – you don’t want to be setting fire alarms off for a blip in a system that does not affect the end service. Know the dependencies on the service and try to ensure that the root cause alert is the first one sent out. You do need to know that the router that affects 10,000 websites is out long before you have alerts for those individual websites.

Medium Term : Trending & Optimisation

Where the timeline goes beyond “what’s wrong now” , you can start to look at what’s going to go wrong soon. This is edge of the crystal ball stuff , where predictions are looking to be made in the order of days / weeks. Based on collected utilisation data in a given period , we can assess if we have sufficient capacity to be able to provide an acceptable service level in the near future. At this stage , adjustments can be made to the infrastructure in the form of resource balancing ( by storage or traditional load ) – tweaks can also be made to virtual machine configuration to “rightsize” an environment. By using these techniques it is possible to reclaim over allocated space and delay potential hardware expansions. This is especially valid where there may be a long lead time on a hardware order. The types of recommendations generated by the capacity optimisation components of VKernel , NetApp ( Akorri ) and Solarwinds products are great examples of rightsizing calculations.  As the environment scales up , not only are we looking for optimisations , but potential automated remediation ( within the bounds of a change controlled environment ) would save time and therefore money.

Long Term capacity analysis : When do we need to migrate Data centers ?

Trying to predict what is going to happen to an IT infrastructure in the long term is a little like trying to predict the weather in 5 years time , you know roughly what might happen but you don’t really know when. Taking a tangent away from the technology side of things , this is where the IT strategy comes in – knowing what applications are likely to come into the pipeline. Without this knowledge you can only guess how much capacity you will need in the long term. The process can be bidirectional though , with the information from a capacity management function being fed back into the wider picture for architectural strategy for example should a lack of physical space be discovered , this may combine with a strategy to refresh existing servers with blades. Larger Enterprises will often deploy dedicated capacity management software to do this ( for example Metron’s Athene product which will model capacity for not only the virtual but the physical environment )  Long term trending is a key part of a capacity management strategy but this will need to be blended with a solution to allow environmental modeling and what if scenarios. Within the virtual environment the scheduled modeling feature of VKernel’s vOperations Suite is possibly the best example of this that I’ve come across so far – all that is missing is an API to link to any particular enterprise architecture applications. When planning for growth not only must the growth of the application set be considered but the expansion in the management framework around it , including but not limited to backup and the short-medium term monitoring solutions.  Unless you are consuming your it infrastructure as a service , you will not be able to get away with a suite that only looks at the Virtual Piece of the puzzle – Power / Cooling & Available space need to be considered – look far enough into the future and you may want to look at some new premises !

We’re going to need a bigger house to fit the one pane of glass into…

“one pane of glass” – is a phrase I hear very often but not something I’ve really seen so far. Given the many facets of a management solution I have touched on above , that single pane of glass is going to need to display a lot ! So many metrics and visualisations to put together , you’d have a very cluttered single pane. Consolidating data from many systems into a mash-up portal is about the best that can occur , but yet there isn’t a single framework to date that can really tick all the boxes. Given the lack of a “savior” product you may feel disheartened , but have faith!. As the ecosystem begins to realise that no single vendor can give you everything and that an integrated management platform that can not only display consolidated data , but act as a databus to facilitate sharing between those discrete facets is very high on the enterprise wishlist , we may see something yet.

I’d like to leave you with some of the inspiration for this post – as seen on a recent “Demotivational Poster” –a quick reminder of perfection being in the eye of the beholder.

“No matter how good she looks, some other guy is sick and tired of putting up with her s***”

I’ve spent more than my usual amount of time in an around airports this week – Travelling to and from the Tech Field Day event in Boston , then hung around with a few of the other delegates before their flights back.

 

It seems that one of the other delegates flights had a pretty severe delay on it due to the incoming flight being late. We realised the only way his flight would be on time would be if they used a different aircraft. My mind immediately went off on a bit of a tangent to Cisco UCS ( as you do ! )

 

The flight plan consists of a number  , a given size and model of Aircraft and a source / destination. With me so far ? The flight plan is given to a particular aircraft , so plane #767-4001 becomes Delta Flight DL270 going from Boston to Heathrow – and will be known as DL270 while that Flight is in use. If for some reason there is a problem with 767-4001 , the airline can opt to use a different plane , for example 777-4002 , which is not quite the same model and in fact has a few more seats & flies a little faster. The plane is still able to take of and land under the identity of DL270.

 

This is very much like a service profile – its not fixed to the hardware ( plane ) and can be associated with different hardware ( which may not be of quite the same specification ) should you require. Its is purely a definition of what would make up that profile just in the same way that we have defined flight DL270 to fly me from Boston back to London.

 

Now if I could only persuade my UCS Chassis to serve complimentary drinks…..

I’ve been lucky enough to be selected again to attend one of Gestalt IT’s Tech Field Day events. These place a selection of IT community members with a selection of Vendors for a series of sessions that go beyond the usual sales pitch you might get a user group event. The are also a lot more interactive , with a roundtable discussion before , after & sometimes during a session. The events are recorded and streamed live , you can also keep with with what the kids at the back of the class are whispering to each other by following the #TechFieldDay hashtag on twitter.

 

This Event is to be held in Boston in just over 2 weeks time and has a particular focus on Virtualisation technology. Other events have been based around Networking & Wireless technology, or just general datacenter technologies. The delegates have been selected for their work within the Virtualisation community , featuring more than its fair share of VMware vExperts and of course the whole vSoup Podcast crew! We are aiming to be able to record & publish an episode of the show live from the event.

 

The Presenters

Solarwinds :

I have seen Solarwinds present before and I’m looking forward to their deep dive style – as veteran TFD Sponsors they know that talking geeky is going to get a good response from us. I would imagine there will be some good detail on the product that is the fruit of the Hyper9 acquisition.

Vkernel:

I’ve enjoyed a good relationship with Vkernel over the last couple of years , both as an end user and as a blogger. Its not their first appearance at a Tech Field Day event so I’m sure that we’ll see something new around their infrastructure optimisation product set.

VMware:

I’ve heard good things about this little start-up , they have something called a Hypervisor , which could go far Smile Is what I’d have said man years ago , but like an ageing relative I’m going to have to say “look how they’ve grown!” I shall be looking forward to meeting up with the Wookie of Virtualisation , John Troyer and seeing what VMware have to show us beyond the press release!

Symantec:

Tech Field Day usually attracts a mix of sponsors , from the very fresh start-up ( in fact there will be a start-up coming out of “stealth mode” at the event ) to the established company. Symantec will sit firmly in the latter of those two and In my opinion have a harder task at these events because they have a PR/Marketing/Community machine that is more used to higher level , PowerPoint rich communication ; which is something that Tech Field Day just isn’t about. I’d love to see a “big” sponsor present with the passion and in depth knowledge of a start-up.

Embotics:

I was lucky enough to meet up with a few of the Embotics guys in the last year and while I like their policy based Virtualisation management product its been something that’s been quite a hard sell back to management. I’ve heard they might have something in the pipeline that will really emphasise its value. Watch this space for more details….

 

There is one extra vendor to be announced in addition to the “stealth mode” start-up launching itself , which I’m particularly looking forward to.  I think its going to be the perfect mixture of catching up with friends within the community , meeting some new ones and submersing myself in some seriously good technology. For more details, check out www.techfieldday.com

 

image

SolarWinds have just launched their latest product in their Admin focussed line of tools and while the underlying focus and data is bread and butter to the network admin , the usage seems focussed towards  a more security admin / HR role.

So , why the departure from the more conventional monitoring and management toolset? The first point Josh Stevens , SolarWinds “Head Geek” made was that the Community asked for it. One of the things that impressed me about SolarWinds when I saw them as a presenter at Tech Field Day was their attention to the IT community – through programs like thwack.com they keep a finger close to the pulse of what admins and other IT Pro’s are actually using their toolsets for and more importantly , what they would like to use it for. I’m a big fan of end user driven development – give the community some option / say in how you evolve after all without users/customers where will you be ? The best example of this would be the user driven feature set of dropbox – see www.dropbox.com/votebox for more details ! Much of the work of a system admin is driven by compliance requirements , be it within a security or regulatory context. The Device Tracker fills a niche currently only occupied by CiscoWorks in terms of tracking connected devices , but adds historical data tracking.

UDT Constantly scans your network devices for connected MAC and IP addresses, keeping a record of what is connected to what – this allows the tool to do a few neat things such as.

– Tracking down problem users / machines : you know the IP of the machine that violated your AUP ,  but not sure where in the building they were at the time ( handy if you hotdesk ) , if you know where your switch ports go to , then this tool could help you match up who was sat at what desk.

– Capacity Planning : So you’ve cabled all of your switch ports to your structured cabling , but how many are you really using over a given period ? Identify unused ports and spare capacity at the access layer.

 

The product seems pretty well integrated with the Orion NPM toolset – using the same interface and familiar look & feel so Admins will be able to pick it up and play right away. The only missing piece of the puzzle is a closer look at clients connected via a wireless network but I’m assured that will be in future version.

Image[3]

SolarWinds are offering some hefty pre-release discounts on User Device Tracker – which starts at $1700. For more details , head over to the SolarWinds Website

image

 

As the tweet above proves I’m about to out scoop Eric “Scoop” Sloof of ntpro.nl fame and would like to be the first to break the news on the innovative Pork Product Delivery system (PPDS)  from your favourite real time monitoring provider , Xangati.

 

In a recent briefing on the new VDI/VI Dashboards I was able to grab a screen shot as the present flicked to a preview screen that proves this to be the case.

 

image

 

Not only is Xangati able to provide role based dash board of real time data about your VI environment that reflect the real health issues within a system , but they are able to monitor the Saltiness Levels for Admins ( SLA’s ) and trigger off an Automated Bacon Delivery Service (ABDS) provided via a network of bacon resellers ( ButcherNet). This was already been successfully beta  tested at Tech Field Day. Turkey based Bacon Substitute (TBBS) is available for environments that don’t dig on swine.

Reading , wRighting and Recording – Measure how your applications hit your disks!

I’ve spent the last week thinking more about storage than I usually would, particularly in the light of some of the conversations I’ve been having over Tech Field Day with the other delegates & sponsors who have had varying levels of interest & expertise within the storage world. If, like me you have a basic appreciation of storage but want to get in that little bit deeper , a good primer would be Joe Onisick’s storage protocols guide at DefinetheCloud.net

Admins working in smaller shops probably have a little closer control over the storage they buy as they are likely to be the ones specifying , configuring and crying over it when it goes wrong ; It’s one of the con’s of working for a large enterprise is that the storage team tends to be separate – they guard their skills and disk shelves quite closely , sometimes a little too closely – I do wonder if their school reports used to say “does not play well with others” . The SAN is seen as a bit of a black box by the rest of the department and generally as long as the required capacity is available to someone when they ask for it , be it a lun or VMware datastore , then everyone is happy to let them get on with it.

As soon as there is a performance issue however , that happy boat starts to rock .The storage team starts to get defensive, casting forth whitepaper & best practice guide as if they were a World of Warcraft character holding a last stand. At some point you may well find that you hit the underlying best performance of the SAN , no matter how well tuned it is. You are then left in a bit of a quandary of what to do, in the worst case you have to bite that bullet and move that application which looked like the lowest of the low hanging fruit back onto a physical server with direct attached storage , where it’ll smugly idle at 5% utilisation for the rest of its life , ever causing reproachful looks when you walk past it in the datacenter.

How do you avoid the sorry tale above ? In a nutshell, “Know your Workload!” When you start to map what your applications are actually using you can start to size your environment accordingly. One of the bigger shocks that I’ve come across when doing such an exercise is a much heavier proportion of writes than the industry would have us expect. This causes a big problem for storage vendors who rely on flash based cache to be able to hit their headline performance figures. When reading from a cache , of course the performance will be great , but under a heavy write intensive load the performance of the underlying disk starts to be exposed and it seems to come down to number and speed of spindles. Running a system that uses intelligent tiering to write hot blocks in the fastest way then cascade them down the array as they get cooler could help in this instance. Depending on your preference for File or Block level storage , there are a number of vendors who could help you with this, for example Avere Systems or 3PAR or the next Generation of EMC’s FAST technology.

At Tech Field Day , NetApp , VMware and Cisco presented on their flexpod solution for a scalable and secure multi tenant virtualised infrastructure. If you’d like to watch the recording of the presentation, its available here . What would appear to differentiate the flexpod from other products is that is a not a blackbox device , designed to drop into a data centre to provide X number of VM’s , when you have X+1 VM’s, you just go out and buy another device.

While you can approach a VAR and order a flexpod as a single unit , the design and architecture is what makes it a “flexpod” – being a single bill of materials that can be put together to give a known configuration. The view of this being that it offers a greater agility of design , for example using a NetApp VServer head to present storage from another vendor to the solution.

To me , this seems a little bit like buying a kit car.

imageYou get a known design and list of components you have to source – although the design may well recommend where you source the components. Sometimes you can get them part built or pre built, but if you want to run it with a different engine , you can drop one in should you so desire.

 

The VBlock from the VCE guys is a different kettle of fish – its not a design guide , its a product. You chose the VBlock that suits the size of deployment that you want to do , order it and sit back and wait for the ready built solution to arrive on the back of a lorry ( truck to our US friends 😉 ) This is like ordering a car from a dealership.

image

Of course you could just go to any reseller and buy a bunch of servers , network & hardware and then install ESX on it. The Stack vendors might compare this to trying to hand cast your car from a single block of metal !

image 

At the moment many of us who can already design a solution from scratch are at that hand casting level , and while I wont deny we’ve been through a few pain points , we’ve usually been able to fix them. Its part of the skill that keeps us employed. By going for an “off the shelf product” the pain of that part of a system design is divorced from the solution and perhaps it would allow focus on what may be the next part of the design at the service and application level –don’t worry about build a car , worry about driving it! . If you need a car to drive to work and do the weekly shopping in, you buy one from a dealership – but if you have a specific need , then you may have to get into the workshop and build a car that meets those needs.If you want to concentrate

When a prebuilt solution  develops a problem that requires support , the offerings from the major vendors seem to differ a little. If you have a VBlock, you have one throat to choke ( presumably not your own , its only a computer problem , don’t let it get to you 😉 ) and one number to call. They will let the engineers from the different divisions fight it out and fix your problem , which is ultimately the only thing of concern to you as an owner.

The situation with a flexpod seems a little less intuitive. As its not a single SKU – you would require a separate support contact with each vendor ( of course this may be marshalled by the VAR you purchase through ) , You would initiate contact with the vendor of your choice – they then have a channel under the skin to be able to work with engineering functions of the other partners at the network, storage , compute & hypervisor arms as required. I would like to think this does not mean the the buck gets passed for a couple of rounds before anyone takes ownership of the problem , but I’ve yet to hear of anyone requiring this level of support. If you have and had a positive or negative experience , please get in contact.

If you have “rolled your own” solution , then support is up to you ! make sure that you have a similar SLA across the stack , or you could find yourself in a situation where you have a very fast response from your hypervisor people , but when they work out its your storage at fault , they might make you wait till the next day / end of the week. If this does happen to you , then I’m sure you’ll have plenty of time to clear your desk….

 

image

I’ve almost recovered from my Hectic week of Jet-setting for this year , starting with the VCAP-DCD Beta Exam in Amsterdam and culminating in a few days of visiting vendors for talks and roundtables in Silicon Valley. It was my first visit to the west coast , so I was initially star struck by it all – names you only ever see as a URL on buildings really pushes home how close you are to the technology and its not hard to get caught up with the buzz of it – I lost count of the number of startup ideas I heard over the course of the event!

For those of you who haven’t heard of the Tech Field Day concept before , here is a brief guide.  Following on from a concept launched by HP , the field day brings a number of delegates from the user community together with a vendor or vendors for a session that should be a little bit more in depth that your average marketing pitch. The delegates are not there to buy anything , and are no way obliged to write about their experiences, although Food & Drink , Travel & Accommodation expenses are covered by the sponsoring vendors.

This particular event marked a new direction for TFD in that it was streamed live over the web via ustream.tv . This potentially changed things in a couple of ways – The cameras were far form hidden and I wonder if the fact that they were being broadcast affected some peoples candour and in a couple of circumstances the sponsors where prepared to say some things off camera that they were not prepared to when they were rolling. That said , the greater audience did mean that a few questions were asked that may have not been bought up had it not for being mentioned on twitter by someone watching the stream. I would like to think that I was as honest as I’d have been on and off camera!

I think the event is possibly better suited to the smaller vendors with a less refined marketing function – Of the larger vendors that we saw , the sessions felt a little pre-canned with PowerPoint hitting a critical mass at one particular site. Making use of an “Executive Briefing Centre” , while it gives you access to nice comfy rooms with wireless internet access does nudge conversations to wards that more marketing side of things. Just using a regular conference room facilitated a more in depth discussion and 2 way communication.Perhaps there is a case for presentations to be done “in the round” to use a theatrical example , with delegates sitting in a "”doughnut” around the presenter.Presenters that had a real passion about their product held the audience much better , a prime example of which was Dave Hitz – founder of NetApp. He was only booked in for a 15 minute slot , but stayed for most of the 4 hours session , which is a lot of time to dedicate for a guy in his position. Outside of his own slides he was active in the discussions around the topics. It was a shame he wasn’t able to stay for lunch, where I believe the best dialog with the NetApp guys occurred.

I my next few blog posts I’m going to try and write about subjects that came up during the sessions , rather than a summary of each session , which you would better off getting from watching the excellent recordings made by the PrimeImage Media guys.

 

For those that missed it , have a look at the following video from the day (my wonderful piece to camera is at about 1:41 )

Tech Field Day 4 – Day 1 – NetApp 2 from Stephen Foskett on Vimeo.

 

One last thing – you may well have noticed my fledgling upper lip furniture – I’m growing a moustache this month as part of Movember – donating my face to men’s health. If you would like to donate to help men who have problems growing good facial hair like myself , then my MoSpace page is at http://uk.movember.com/mospace/1067584/