Despite the fact my role is pre-sales – I spent a lot of time digging into client performance statistics with them. I wanted to document some of the more common ways to collect and review performance data.
vCenter:
I view vCenter as a great performance trending tool only. Real-time statistics are not stored in the database and can only be viewed for the past hour – sadly. That said, when a client is trying to troubleshoot an issue I recommend they temporarily change the logging level for the 5 minute interval from the default level of 1 to 4. While this will increase your database size, some other interesting counters (memory, cpu, disk and network) are captured and can looked at with some historical accuracy for the day. See this vCenter Performance Counters page for what’s available in each level. Organizations should really consider augmenting the vCenter tool with something that will keep granular historical data.
esxtop/resxtop:
esxtop (or resxtop if you’re using the vMA appliance) is my hardcore tactical troubleshooting tool. The real-time information it displays is extremely useful in diagnosis though its biggest issue is that you can only look at one host at a time. I often ask for esxtop data samples using the following command:
esxtop -b -a -d 2 -i 150 >outputfile.csv
This will collect data using esxtop ‘b’atch mode, ‘a’ll counters, every ’2′ seconds for 150 ‘i’terations (or a total of 5 minutes). The resulting .CSV file can be opened directly with PerfMon (use Win7 64bit edition) or esxplot (cheers to Geoffrey for his development effort).
vm-support:
Last, the vm-support script can be called with some parameters to not only collect the diagnostic bundle but to also collect performance data from the VSI nodes that can be re-played with esxtop (or extracted – see the command below). This combo deal is also very useful since it allows us to not only look at the performance data, but some of the logging to check things like vSwitch config, flow control and monitor modes, etc. The size can be daunting, potentially GB’s when extracted and so typically must be transported outside of email.
vm-support -n -s -i 2 -d 150
(again for samples every 2 seconds for a total of 5 minutes with no core dump). To extract esxtop data use the following command:
esxtop -R / |esxtop -b >outputfile.csv
Anyone else have any cool tools or tips for data collection and analysis?
Happy Holidays Everyone!
