• Linked Applications
    Loading…

Customer use case: analysis of a completely overloaded system

We started an analysis on customer side because of pending decisions how to extend the available capacities.
these are the results after the first look (1h) - the most interesting thing here is that the system is not looking to bad on the SCSI level - but when one looks deeper ...

The Customer has performance issues since a longer time and there is a need to clarify the root causes before spending money.

There are several ideas starting from newer Nodes (it is a 2145-8G4 node) or higher performance storage classes

The result might be - get rid of the old N6060 storage with the performance issues as planned and invest in medium performance capacities
because there is no real need for highest performance classes.
The Cache on the 2145-8G4 nodes is small - we have to go deeper to find whether it is a technical need to invest in newer SVC nodes.

Customer Session on 4. April 2003Click on pictures to enlarge

The IO Heat Analysis Overlay in the Treemap shows
that the customer does not use the performance capabilities
of his backed storage.


This shows - that enough performance capability is
available. (IN THEORY)

In Praxis - if the systems in the storage backend cannot
deliver the performance they are built for than all these
calculations are for nothing.

The result is then, that the performance of storage
backend systems is not enough beause of several
reasons.

The processor of the system is too slow. It doesn't help
when we build in hundreds of disks for performance when
the processor is not able to serve all theses IOS.

Something is speeding down the performance like
a WAFL file system which is too full to perform striping.
(which might be the case here)

The coloring is based on calculations we did for the Arrays.

  • blue - does only use less than 20% of the
    performance capabilities
  • Green between 20% and 80%
  • Yellow between 80% and 100%
  • Red more than 100%

 

 

The CPU Load and RW Cache is in a normal range.


The Global Write Cache max is much too high -
it should normally be from 0% up to less than 80%.

In this situation the write Cache max is mainly mote
than 75% which is much too high.


This is a normal loaded System

  • (red) CPU << 70%
  • (blue) RW Cache SVC = 78
  • (blue) RW Cache Storwize = 72
  • (green) Global W - Cache max < 80%
  • (dark green) (not shown here)
    Global W - Cache min often 0

This is one of the managed disk groups from
the customer.

The Max Write Cache fullness is nearly always
more than 80%.

 

It looks like the SVC cannot get rid of the cache
content because of restricted performance in
the storage back end.


  • (green) MDG W cache max - sometimes 100%
    for small timeframes. Normally should be
    at 80% or less. Long times on 100% is overload
    especially when MDG W Cache min is over 80%.
    More than 100% only single peaks. In this case
    watch out for min value - has to be <80%

  • (dark green) (not shown here)
    MDG W Cache min - should often be 0 sometimes
    for smaller periods 80% seldom more than 80%
    90% and more is alarm signal

The SAN does not show any problems.
In the time period of cache overflow no heavy
transfer activity can be measured on the SAN
ports and no Buffer Credit Wait % problems occor.

 

  • (Yellow) Buffer Credit Wait % - always measure this
    on the single port (never on aggregate).
    less than 10% is OK as long as there are no other
    performance complaintsin this moment.

  • All other codes from the debug section have to be 0

Another Sign of congestion is that the response time
of the SAP volumes go in parallel with their IOP demand.

 

More IOPS lead automatically to a higher response time.




Popular content:

Page: Performance bottleneck analysis on IBM SVC and IBM Storwize V7000 , Page: BVQ use cases and experiences , Page: Downloads and BVQ releases , Page: Solved buffer credit wait situation at a customer side , Page: Use BVQ to optimize SVC and Storwize multi IO group clusters


General links

Return on invest 

Performance analysis whitepapers


The BVQ Blog
SVC / Storwize monitoring, reporting and performance analysis solution. BVQ is the fastest and complete performance analysis tool for IBM storage virtualization.
BVQ Version 3.4 adds Storwize drive statistics, point context menus and easy tier monitoring
We just released BVQ Version 3.4 with some very strong enhancement. Some of the major highlights are:   Drive statistics for Storwize systems! BVQ 3.4 adds drive statistics below the mdisk layer You now have a detailed look...
SVC Storwize performance analysis video with BVQ version 3.4
With BVQ version 3.4, we will achieve great improvements in the handling of Performance analysis.   BVQ 3.4 makes it easy to do performance analysis. Ideal for people starting with it. BVQ 3.4 speeds up up performance analysis for...
BVQ Version 3.3.3 download page in English language is now available
You can now find this page here http://bvqwiki.sva.de/x/BQEz   The improvements of BVQ 3.3.3 are Loading and drawing processes of the performance view were significantly improved. Long lasting data load processes...
New BVQ offline scanner now scans Storwize drive performance information
We just released the new offline scanner which will now also scan the drive information from any Storwize product. Direct Link to BVQ offline scanner http://bvqwiki.sva.de/x/3IBdAQ With this we can make more accurate statements about the...
BVQ V3.3.3 with great improvments for performance analysis released
We just released the new BVQ V3.3.3 with great enhancements in performance and usage of the performance analysis methods. We are in progress to translate the release notes to English. If you want to download V3.3.3 now you can do this from the German...
Thoughts about storage monitoring or monitoring or analysis?
  I had a discussion about storage monitoring and analysis. This discussion ended in the following understanding: Monitoring with a simple product help you to understand when things go wrong. They show the symptoms but not the...
Sneak preview BVQ dashboard will greatly enhance storage monitoring capabilities
The BVQ development team is currently working on the new BVQ dashboard functionality. This new dashboard should become available in 1H2015. The dashboard is great for monitoring, you can open several dashboards and put them onto different screens, this...
BVQ is so incredibly fast! Leads to SVC / Storwize performance analysis results in seconds instead of minutes or hours
This is just a new experience with the newest BVQ Beta code which will become available as BVQ Version 3.4 in some weeks. BVQ Version 3.4 will have a completely refurnished internal data handling which you will not recognize as a benefit today but this...
Analysis of SVC remote copy performance problem
This is an example of an successful latency peak analysis were we found the reason of a performance problem in the remote copy connection.  It is very complex to analyze RC performance problems but the BVQ Copy Services Package adds all tools needed to...
New SVC / Storwize /Flash and BVQ customer reference Bitmarck
  Find more details in the BVQ reference pages in English in German   TASK Expansion and optimization of SAN infrastructure for a higher performance and an...

(.)