click to expand ...

Home
About SVA ... Contact us
Overview (all documents)
Product description
Services with BVQ
BVQ advantages
New functions and future plans
White papers
Downloads and releases
Users Manual and users guide
Further WIKI documents

 


Test the BVQ Risk Report with the offline scanner ... no install needed for this!
Today most health checks are done with the BVQ Offline Scanner. This means no installation is needed and minimal risk for the environment.
The Risk Report is based on the BVQ offline-scan methodology.  Find more information about this here:
Avoid storage availability problems - BVQ offline scanner and risk analysis

 

About this BVQ success story


This is a customer success story where BVQ helped to fix performance issues in a Red Hat Enterprise Virtualization (RHEV) environment with IBM Storwize V7000 storage. This document describes how the BVQ team solved a performance issue in a specific customer situation.

The single steps which led to the solution had been chosen based on this specific environment.
These steps can therefore not be used in other environments without prior root cause analysis.

 

Content


About the BVQ success story
   Unacceptable performance of the RHEV
   Performance optimization project start 09/20/2013

The optimization project starts
   Step by Step moving the volumes to uncompressed
   The result of intervention: improved response times
   The performance also benefits from Easy Tier

Resume from the BVQ team
Resume from the customer

BVQ web pages

Unacceptable performance of the RHEV

The Red Hat environment is set up with six storage volumes which had been thin provisioned and compressed to optimize the capacity efficiency.
The achieved capacity efficiency is high which results in a reduction from 16TB down to 5TB. The space reduction with the compression algorithm was approximately 50%! The space reduction with using Thin Provisioning was depending on the age of the volume - between 50% and 70%.

 


Pict.1: The IBM Storwize monitor displayed very bad volume latencies. This gave the first impression that the bad RHEV performance was caused from the storage. But this monitor cannot show more helpful details to solve the issue. (picture was taken later)

The situation in the beginning of the project was that the performance of the RHEV environment was reduced due to very high latencies from the IBM Storwize volumes. The RHEV as a result was running below acceptable performance levels – the users experienced phases of long response times which are not necessarily connected to the actual workload of the system. The Storwize system-monitoring showed very bad volume response times. The first step was to eliminate this most obvious bottleneck to find out whether this was the only cause for the bad RHEV performance.
The customer already experienced a long analysis phase before to find out the root causes for the bad volume performance. The target of this customer intervention was to figure out what the reason was and finally find a way to improve the situation.

Performance optimization project start 09/20/2013

The whole performance optimization project lasted a bit more than one week.

  • Friday 09/20/2013:
    The customer contacted the BVQ team and requested a BVQ PoC installation. Some mails have been exchanged and we sent back a BVQ PoC license enabling the customer to install BVQ by himself the same day. A call with the BVQ team was scheduled for Wednesday the following week..

  • Wednesday 09/25/2013:
    At this time the BVQ team was able to take a first look into the customer's storage environment. The team found that the cache in the managed disk group serving the RHEV was totally exhausted overfilled. This led to situations where volumes were withdrawn from cache and started to respond with very bad latencies. The more intense the BVQ team analyzed the environment the clearer the picture evolved, that compression has to be the performance killer in the customer's environment.

    Together with the customer a step by step plan was established to eliminate compression of the system. Additionally it was decided to also activate the Easy Tier mechanism within the IBM Storwize, as the workload of the system is large and high frequent swap areas on the volumes could be best addressed with SSD storage.

    The decision for Easy Tier was also taken due to the fact that the workload underneath the V7000 cache the customer should use more than 50 disks to deliver the needed performance. With Easy Tier, the BVQ team saw the fair chance that SSDs could handle a large amount of the high frequent swap workload and the number of disks could be reduced. The decision to use Easy Tier again confirmed the need to switch of compression because Easy Tier is not optimizing writes on compressed volumes in the Storwize code V7.1.0.3...
  • Friday 09/27/2013 until Monday 09/30/2013: The compression has been replaced step by step using volume copies and adding Easy Tier managed disks.

  • On Monday morning the project had been completed.

The optimization project starts


The RHEV Users were experiencing high latencies independent of high or low workloads. This was reflected in the measurements when monitoring IOPS against latency from the RHEV disks.


 
Pict. 2: These graphs show the high latencies (red) the moments the Red Hat Enterprise Virtualization (RHEV) accesses the Storwize system. The workload is high with more than 3500 IOPS (green) which are mostly write IOPS in this specific customer situation. The maximum latency (red) measured was 575 ms on Wednesday.


Analyzing the workload it has to be noticed that the RHEV volumes are doing far more writes than reads. The Storwize cache efficiency in this environment is high for read and write but the cache availability itself is not stable. The transfer sizes are very different and can change every moment. In phases of higher write transfer sizes (higher data rate) more data is transferred from the RHEV into the Storwize which often causes a cache collapse due to a cache overflow.

 
Pict.3: This picture shows a typical reaction on MDG (managed disk group)
"max cache fullness" situation. The MDG Cache (upper picture) is fully used and will now start to recover. The cache capacity, which belongs to this volume (lower picture), will be reduced to 0. Now the volume does no longer own write cache and the response time (red) explodes from below 1ms up to 60ms for more than one hour. In between we discovered a short time period (in yellow dashed lines) where the system was able to lower minimal cache fullness clear below 80%. This was the first time, when the volume received some cache for its operation.


Looking at the Managed Disk Group Cache Partition it became obvious that the cache was completely overfilled. One reason for this can be that Storwize with the 7.x code is using cache memory to calculate the compression. An additional indicator for this is that the cache fullness min. indicator never went below a certain level (in this case 55%). With compression this minimum level of cache fullness would be shifted to a higher value and therefore less cache is available for the operations. This underlines the situation that cache max is reaching the point of cache overflow earlier.
Something we could not measure yet in the actual BVQ code is the CPU power needed for compression. This will be available with BVQ version 3.0.

 
Pict. 4: The managed disk group cache is completely overfilled. Cache fullness max is on 100% or even more for very long times. Even worse is
that the cache never recovers to a low percentage. It is very typical for compression that cache fullness min percentage is never below a certain limit. This needs to be checked before one can say that this is a problem which has a compression root cause.

Step by Step moving the volumes to uncompressed

With some additional storage it was easy to use the volume mirror operation to create uncompressed copies of the volumes. In our case we had the good situation that some additional capacities were available for the copy operations.

 
Pict.5: The only way to switch off compression is to create an uncompressed copy of the volume. Afterwards you only have to delete the compressed side of the mirror.

 

This enabled us to create a new managed disk group being target for the copy operations. In any way one has to be careful when the volumes will be decompressed. More capacity is needed with uncompressed volumes afterwards. Use the BVQ property sheets to get exact numbers how much additional capacity is needed for the uncompressed volumes. (Pict.6)
After the start of the copy operation (Pict.5) the system displayed a volume synch time with estimated more than 500 hours! This number looked frightening in the beginning but it improved very soon after the process started.


 
Pict.6: The BVQ properties provide a good guess concerning the capacity needed to move a volume from compressed to de-compressed. In this case the 4TB volume is compressed and thin provisioned and needs 721GB of capacity. After de-compressing this volume will need 1.16TB of storage capacity.


To further speed up the copy operation we increased the mirror sync rate to 100 which was no problem in this situation as the storage is less used over the weekend.
At the end the copy of one volume needed five to eight hours.
In this specific situation we decided to decompress all volumes. The base for this decision was that we had enough storage available and all volumes are designed to deliver the same performance. RHEV is using these volumes like a stripe-set what further underlined this decision. In other situations one will only decide to decompress some volumes based on their performance and cache consumption. One can spare a lot of capacity (and money) with proper analysis before starting this process.

Success - The result of intervention are heavily improved response times


The following two pictures show the difference in performance. They have been taken in the same day with one week difference.

Pict.7: Wednesday 09/25 before we started the optimization. The worst volume latencies measured have been 575 ms as mean value across all RHEV volumes. This means far more latency for the one or other of the 6 volumes. The latencies are very dependent from the volume transfer sizes, which is a good proof for cache overflow issues. (Higher data rates lead to cache overflow)



Pict.8: Wednesday 10/02 after we finished the first phase of optimization (rebalance mdisk group is still running in this moment). The worst volume latencies measured have are been 1.5 ms and 2.5ms. The scaling for latency and IOPS are the same as in Picture 7.

Easy Tier Analysis "The performance also benefits from Easy Tier"


The following two pictures show the contribution of the two 200GB SSD mdisks to the performance of the managed disk group.

Pict.9: When we compare the total backend IOPS against the IOPS from the SSDs we find that the SSDs took over nearly half of the complete IOPS. This is a good result when we keep in mind that most of the IOPS are write IOPS. In other cases which are more read oriented the easy tier effect could be much higher.



Pict.10: When we compare the medium response time of the mdg to the response time of the single SAS drives we find out that easy tier improved the mdg latency from 14.11 ms down to 7,89ms.


In this case the performance improvement from easy tier is more or less a reduction of needed disks to achieve the performance. When we calculate the need to run on 2000 IOPS in the storage backend we would need 41 10k disks. With easy Tier the SSDs cover nearly 50% of this load. In other words we only need to plan for 1000 backend IOPS for the SAS drives, which can be handled with 20 spindles, which is more than 50% less of needed capacity

Resume from the BVQ team

It was a great pleasure to work with this customer – we have learned a lot from this performance situation and we will for sure add some of the new aspects into our software. One great benefit was that we could assist the customer in the single steps and see the improvements.

Resume from the customer

We had been looking at the V7000 performance page and the heat maps for a few months while trying to figure out what the cause of our performance issues were across all the V7000 volumes. We contacted BVQ team to see if their tool could help us identify the issues. What we got was contact with a team who knew exactly what to look for and where to find it. They gave us more insight into our platform during a two hour walkthrough than we have had since we deployed the V7000 in March. Through phone calls and remote access sessions, we believe our V7000 is now performing to the best of its abilities, given our particular environment/requirements. If you have or are planning to deploy a V7000, you should seriously consider deploying BVQ along side it!
Eamon, IT Architect.

 

BVQ web pages


BVQ in the Internet



If you are interested in further information or you want to become a partner to distribute BVQ to your customers,
please contact us at: https://www.bvq-software.de/en/contact/


BVQ is a product of the SVA System Vertrieb Alexander GmbH