One Factor At a Time
Many modern environments are complex, utilizing multiple technologies, products from many vendors, and multi-tier architectures. The testers are the eyes and ears of the performance specialists (typically sysadmins, DBAs, network engineers and developers) who diagnose and fix problems. Testers may have to play the role of a general contractor, coordinating the efforts of these different specialists with their deep narrow areas of expertise and their own vocabularies. The Unix guys do not understand what the Windows guys are doing within the same overall shared technical infrastructure and how it may impact them, and neither talk to the DBAs or the security guys.
Even the determination of what performance characteristics to observe in these environments may not be straightforward: choices of response time, throughput
, memory and / or processors, error rates, voice or video quality (if the network provides VoIP
), user satisfaction, etc.
Other complications are that the sysadmins may see testers as meddling in their business, and developers often fail to realize that performance testing, unlike black-box feature testing, requires close continuous development support.
Within this complicated context, we would to like to use OFAT (tuning one factor at a time) if we could. A modern database of moderate size and complexity typically has dozens of adjustable parameters or factors (?tunables?), and a complete infrastructure can easily have thousands. We would like to change only one factor within each iterative cycle of retune-and-retest, so that we can see the cause-and-effect relationship of the one change clearly by comparing the before-and-after performance test results. Let?s say that in a particular situation the tuners a have mere 43 factors available to adjust (a more typical number would be ten times higher). They leave 36 unchanged, increase 4 by various amounts and decrease the other 7 factors. In consequence, the system response time improves by lets say 3%, but the tuners and testers do not which adjustments helped improve response time and which impeded it.
OFAT (tuning one factor at a time) does not work, though, for two main reasons. First, OFAT takes too long. If it takes 2 to 3 tuning cycles on average to find the optimum setting for each factor, then tuning 43 will take 90 to 130 iterations. We generally do not have the time for more than a few cycles.
Second, and far more importantly, OFAT would not work even if we were willing to invest the time to undertake 130 iterations in the situation above. OFAT assumes the tunable factors are mutually independent. Once we find the optimal setting for one factor, we can lock it and move on to tune the next factor. In reality, the first factor becomes de-optimized as soon as we change any other factor because of the dependencies. Usually there are many complex, poorly understood and hidden interdependencies among the factors. But other methods (typically multi-factor-at-once adjustments) are hard to apply well.
Contributed by Ross Collard∞