Multi-Vari Studies, How to Quickly Find 85% of the Variation in a Product or Service

The name “Multi-Vari” was given to this form of analysis by L.A. Seder in his classic paper “Diagnosis with Diagrams,” which appeared in Industrial Quality Control in January and March 1950.  The premise is to utilize graphics to understand where the variation in a process exists.  Is it excessive variation within a single piece, excessive variation from piece to piece, or is the variation excessive from time to time.  If we are relating this to service delivery substitute “service delivery to customers”, “service delivery from customer to customer”, and “service delivery from time to time” in the previous sentence.

Multi-Vari Studies are often classified as either a “Nested Design” or a “Crossed Design.”  In the Nested Design the data is collected without making changes to the process to investigate where the variation is coming from.  It could be positional which is within piece variation, it could be cyclical which is consecutive piece-to-piece variation, or it could be temporal which is time-to-time variation such as day-to-day, or week-to-week.  The following graphic is an example where we are trying to find the source of process variation with regard to warp in a glass container.

From this Nested Design Multi-Vari Chart we can clearly see that Machine Section 7 is very different than any other section on the machine.  Section 7 becomes the target for variation reduction.  The question becomes, “Why is it so different than the rest of the machine?”

In the Crossed Design the plan is to test changes to the process in a balanced manner following an on or off strategy.  In the Crossed Design either 2 or 3 potential variation contributor process variables are studied at 2 different settings.  Analysis of Variance is often added as part of the study to provide detailed statistics that support what the graphic analysis portrays.  The ANOVA provides the verdict of “guilty beyond a shadow of a doubt” to support what we see graphically.  The following graphic is an example where we are trying to minimize the time it takes to boil a cup of water in a microwave oven.

From this Crossed Design Multi-Vari Chart it is clear that to minimize the time to boil a cup of water in a microwave oven the container should be rotated, located 4 inches off center, and covered.  To add further proof to this graphic finding an ANOVA (analysis of variance) was conducted with the following results.

The sources of variation are Cover, Rotate, and Location.  Each are significant with p values that are less than .0009 (assume worst case for unknown digit of 9) which equates to a confidence level of at least of 99.91%.

Multi-Vari Studies provide a graphic means to quickly find 85% of the variation in a product or service.  I think you will find this technique to be useful.

Choosing Hypothesis Tests

A question my students often ask is, “Which hypothesis test should I use and when?” In this article we will address some guidelines to answer the question. The available hypothesis tests are:

• Continuous Variable Outcomes
• T Test
• Paired T Test
• ANOVA (Analysis of Variance)
• Test for Equal Variances
• Discrete Variable Outcomes
• Chi-Square

The following examples will address which test to use given a certain set of circumstances. In hypothesis testing we are faced with answering the question, “Do the variables in my process make a difference, or not, if they are changed?”

Continuous Variable Outcomes

The output, or outcome, in the process is measured on a continuous scale. We will refer to the outcomes as the “Y”. The input variables, or the things we will be changing, are varied between discrete settings, or levels. The variable could be continuous, but the settings are specific and can be considered discrete.

Case 1: T Test

The T Test allows testing of two items only, or two level settings only. Let’s say we want to improve our gas mileage. The output Y is miles per gallon. The inputs for the T Test are gasoline additives. The level settings could be Yes (use the additive) and No (plain gasoline without additives). The sample size can be small using the T Test. Run 5 tanks of fuel under each condition and measure the miles per gallon. The null hypothesis for this test is regardless of whether or not we use the additive the gas mileage will remain the same evidenced by p values much greater than 0.05. The alternative hypothesis is that there is a difference between Yes and No which is evidenced by p values that are less than or equal to 0.05.

Case 2: Paired T Test

In the Paired T Test only two items can be tested, but the tests are run concurrently, or in pairs of both items. We use the pairing technique when environmental factors may influence the outcomes. We want that “noise” to have an equal chance to affect the test subjects so running the test concurrently assures this equality of noise distribution. In this case, we will test two hull designs for nautical speed. Testing will be carried out over several days so the conditions in the ocean will definitely be changing such as wind speed, wind direction, wave height, and currents. Both of the hull designs will be subjected to the same conditions when we conduct the tests simultaneously in pairs. The plan is to conduct 5 races over the course of one week. If the p values in the Paired T Test are less than or equal to 0.05 than the hull design with the greatest nautical speed can be declared the winner because the test shows a significant difference. If the p value is much greater than 0.05 then we need to go back to the drawing board because there is no difference in the hull designs.

Case 3: ANOVA

Analysis of Variance, or ANOVA, is very powerful because there is essentially no limit to the number of items, or level settings that can be evaluated during the testing. We are limited only by practicality. In this case we want to determine if there is a difference in the distance a golf ball can travel. The outcome Y is the distance in yards. We will test Pinnacle, Nike, Titleist, Srixon, Bridgestone, and Callaway. A robot with one type of golf club will be used to launch the golf balls. Swing speed and force will be the same for each test subject. Twenty of each ball will be launched and the driving distance will be measured. As in all of these hypothesis tests, the p value is the measuring stick for declaring if a difference exists or not. When the p value is < or =" to" style="mso-spacerun:yes"> When the p value is much, much greater than 0.05 we declare that no significant difference exists between the test subjects.

Case 4: Test for Equal Variances

In the three previous cases the concern was a difference in the average value of the outcome based upon the level setting of the input variable. With Test for Equal Variances the evaluation is the variability of the outcomes about the average. The standard deviations are evaluated to test for differences in variation. In this case we will use the data from Case 3, the driving distance of the golf balls. Which golf ball is most consistent in driving distance? If I buy a dozen of these golf balls can I expect the same results? The Test for Equal Variances provides the answer. If the p value is low than the null must go, but if the p value is high the null applies. The null hypothesis is always “There is no difference.” Two tests are used, one is called Bartlett’s Test which requires the distributions to be normally distributed and the other is Levene’s Test which requires only that the data is continuous.

Discrete Variable Outcomes

The output, or outcome, in the process is measured by counting occurrences which is a discrete variable. We will refer to the outcomes as the “Y”. The input variables, or the things we will be changing, are varied between discrete settings, or levels.

Case 5: Chi Square

Chi Square testing compares discrete Y’s and discrete X’s. In this type of analysis categories, or groups, are compared to other categories, or groups. For example, “Which cruise line had the highest customer satisfaction?” The discrete X variables are (RCI, Carnival, and Princess Cruise Lines). The discrete Y variables are the frequency of responses from passengers on their satisfaction surveys by category (poor, fair, good, very good, and excellent) that relate to their vacation experience.

Conduct a cross tab table analysis, or Chi Square analysis, to evaluate if there were differences in levels of satisfaction by passengers based upon the cruise line they vacationed on. Percentages are used for the evaluation and the Chi Square analysis provides a p-value to further quantify whether or not the differences are significant. The overall p-value associated with the Chi Square analysis should be 0.05 or less. The variables that have the largest contribution to the Chi Square statistic drive the observed differences.

Now you should have a good understanding of which hypothesis test to use and when it is most appropriate. Remember that it is just as important to determine that there is no difference as well as that there is a difference. Sound business decisions depend on making choices based on significance.

Where do we apply Statistical Tools?

Before starting any type of analysis classify the data set as either continuous or attribute, and in many cases it is a blend of both types. Continuous data is characterized by variables that can be measured on a continuous scale such as time, temperature, strength, or monetary value. A test is to divide the value in half and see if it still makes sense.

Attribute, or discrete, data can be associated with a defined grouping and then counted. Examples are classifications of good and bad, location, vendors’ materials, product or process types, and scales of satisfaction such as poor, fair, good, and excellent. Once an item is classified it can be counted and the frequency of occurrence can be determined.

The next determination to make is whether the data is an input variable or an output variable. Output variables are often called the CTQs (critical to quality characteristics) or performance measures. Input variables are what drive the resultant outcomes. We generally characterize a product, process, or service delivery outcome (the Y) by some function of the input variables X₁,X₂,X₃,…X_n. The Y’s are driven by the X’s.

The Y outcomes can be either continuous or discrete data. Examples of continuous Y’s are cycle time, cost, and productivity. Examples of discrete Y’s are delivery performance (late or on time), invoice accuracy (accurate, not accurate), and application errors (wrong address, misspelled name, missing age, etc.).

The X inputs can also be either continuous or discrete. Examples of continuous X’s are temperature, pressure, speed, and volume. Examples of discrete X’s are process (intake, examination, treatment, and discharge), product type (A, B, C, and D), and vendor material (A, B, C, and D).

Another set of X inputs to always consider are the stratification factors. These are variables that may influence the product, process, or service delivery performance and should not be overlooked. If we capture this information during data collection we can study it to determine if it makes a difference or not. Examples are time of day, day of the week, month of the year, season, location, region, or shift.

Now that the inputs can be sorted from the outputs and the data can be classified as either continuous or discrete the selection of the statistical tool to apply boils down to answering the question, “What is it that we want to know?” The following is a list of common questions and we’ll address each one separately.

• What is the baseline performance?
• Did the adjustments made to the process, product, or service delivery make a difference?
• Are there any relationships between the multiple input X’s and the output Y’s? If there are relationships do they make a significant difference?

That’s enough questions to be statistically dangerous so let’s begin by tackling them one at a time.

What is baseline performance?

• Continuous Data

Plot the data in a time based sequence using an X-MR (individuals and moving range control charts) or subgroup the data using an Xbar-R (averages and range control charts). The centerline of the chart provides an estimate of the average of the data overtime, thus establishing the baseline. The MR or R charts provide estimates of the variation over time and establish the upper and lower 3 standard deviation control limits for the X or Xbar charts. Create a Histogram of the data to view a graphic representation of the distribution of the data, test it for normality (p-value should be much greater than 0.05), and compare it to specifications to assess capability.

Minitab Statistical Software Tools are Variables Control Charts, Histograms, Graphical Summary, Normality Test, and Capability Study between and within.

• Discrete Data

Plot the data in a time based sequence using a P Chart (percent defective chart), C Chart (count of defects chart), nP Chart (Sample n times percent defective chart), or a U Chart (defectives per unit chart). The centerline provides the baseline average performance. The upper and lower control limits estimate 3 standard deviations of performance above and below the average, which accounts for 99.73% of all expected activity over time. You will have an estimate of the worst and best case scenarios before any improvements are administered. Create a Pareto Chart to view a distribution of the categories and their frequencies of occurrence. If the control charts exhibit only normal natural patterns of variation over time (only common cause variation, no special causes) the centerline, or average value, establishes the capability.

Minitab Statistical Software Tools are Attributes Control Charts and Pareto Analysis.

Did the adjustments made to the process, product, or service delivery make a difference?

• Discrete X – Continuous Y

To test if two group averages (5W-30 vs. Synthetic Oil) impact gas mileage, use a T-Test. If there are potential environmental concerns that may influence the test results use a Paired T-Test. Plot the results on a Boxplot and evaluate the T statistics with the p-values to make a decision (p-values less than or equal to 0.05 signify that a difference exists with at least a 95% confidence that it is true). If there is a difference choose the group with the best overall average to meet the goal.

To test if two or more group averages (5W-30, 5W-40, 10W-30, 10W-40, or Synthetic) impact gas mileage use ANOVA (analysis of variance). Randomize the order of the testing to minimize any time dependent environmental influences on the test results. Plot the results on a Boxplot or Histogram and evaluate the F statistics with the p-values to make a decision (p-values less than or equal to 0.05 signify that a difference exists with at least a 95% confidence that it is true). If there is a difference choose the group with the best overall average to meet the goal.

In either of the above cases to test to see if there is a difference in the variation caused by the inputs as they impact the output use a Test for Equal Variances (homogeneity of variance). Use the p-values to make a decision (p-values less than or equal to 0.05 signify that a difference exists with at least a 95% confidence that it is true). If there is a difference choose the group with the lowest standard deviation.

Minitab Statistical Software Tools are 2 Sample T-Test, Paired T-Test, ANOVA, and Test for Equal Variances, Boxplot, Histogram, and Graphical Summary.

• Continuous X – Continuous Y

Plot the input X versus the output Y using a Scatter Plot or if there are multiple input X variables use a Matrix Plot. The plot provides a graphical representation of the relationship between the variables. If it appears that a relationship may exist, between one or more of the X input variables and the output Y variable, conduct a Linear Regression of one input X versus one output Y. Repeat as necessary for each X – Y relationship.

The Linear Regression Model provides an R² statistic, an F statistic, and the p-value. To be significant for a single X-Y relationship the R² should be greater than 0.36 (36% of the variation in the output Y is explained by the observed changes in the input X), the F should be much greater than 1, and the p-value should be 0.05 or less.

Minitab Statistical Software Tools are Scatter Plot, Matrix Plot, and Fitted Line Plot.

• Discrete X – Discrete Y

In this type of analysis categories, or groups, are compared to other categories, or groups. For example, “Which cruise line had the highest customer satisfaction?” The discrete X variables are (RCI, Carnival, and Princess Cruise Lines). The discrete Y variables are the frequency of responses from passengers on their satisfaction surveys by category (poor, fair, good, very good, and excellent) that relate to their vacation experience.

Conduct a cross tab table analysis, or Chi Square analysis, to evaluate if there were differences in levels of satisfaction by passengers based upon the cruise line they vacationed on. Percentages are used for the evaluation and the Chi Square analysis provides a p-value to further quantify whether or not the differences are significant. The overall p-value associated with the Chi Square analysis should be 0.05 or less. The variables that have the largest contribution to the Chi Square statistic drive the observed differences.

Minitab Statistical Software Tools are Table Analysis, Matrix Analysis, and Chi Square Analysis.

• Continuous X – Discrete Y

Does the cost per gallon of fuel influence consumer satisfaction? The continuous X is the cost per gallon of fuel. The discrete Y is the consumer satisfaction rating (unhappy, indifferent, or happy). Plot the data using Dot Plots stratified on Y. The statistical method is a Logistic Regression. Once again the p-values are used to validate that a significant difference either exists, or it doesn’t. P-values that are 0.05 or less mean that we have at least a 95% confidence that a significant difference exists. Use the most frequently occurring ratings to make your determination.

Minitab Statistical Software Tools are Dot Plots stratified on Y and Logistic Regression Analysis.

Are there any relationships between the multiple input X’s and the output Y’s? If there are relationships do they make a difference?

• Continuous X – Continuous Y

The graphical analysis is a Matrix Scatter Plot where multiple input X’s can be evaluated against the output Y characteristic. The statistical analysis method is multiple regression. Evaluate the scatter plots to look for relationships between the X input variables and the output Y. Also, look for multicolinearity where one input X variable is correlated with another input X variable. This is analogous to double dipping so we identify those conflicting inputs and systematically remove them from the model.

Multiple regression is a powerful tool, but requires proceeding with caution. Run the model with all variables included then review the T statistics (T absolute value <=1 is not significant) and F statistics (F <=1 is not significant) to identify the first set of insignificant variables to remove from the model. During the second iteration of the regression model turn on the variance inflation factors, or VIFs, which are used to quantify potential multicolinearity issues (VIFs <5> 5 to 10 are issues). Review the Matrix Plot to identify X’s related to other X’s. Remove the variables with the high VIFs and the largest p-values, but only remove one of the related X variables within a questionable pair. Review the remaining p-values and remove variables with large p-values >>0.05 from the model. Don’t be surprised if this process requires a few more iterations.

When the multiple regression model is finalized all VIFs will be less than 5 and all p-values will be less than 0.05. The R² value should be 90% or greater. This is a significant model and the regression equation can now be used for making predictions as long as we keep the input variables within the min and max range values that were used to create the model.

Minitab Statistical Software Tools are Regression Analysis, Step Wise Regression Analysis, Scatter Plots, Matrix Plots, Fitted Line Plots, Graphical Summary, and Histograms.

• Discrete X and Continuous X – Continuous Y

This situation requires the use of designed experiments. Discrete and continuous X’s can be used as the input variables, but the settings for them are predetermined in the design of the experiment. The analysis method is ANOVA which was previously mentioned.

Here is an example. The goal is to reduce the number of unpopped kernels of popping corn in a bag of popped pop corn (the output Y). Discrete X’s could be the brand of popping corn, type of oil, and shape of the popping vessel. Continuous X’s could be amount of oil, amount of popping corn, cooking time, and cooking temperature. Specific settings for each of the input X’s are selected and incorporated into the statistical experiment.

Minitab Statistical Software Tools are DOE, Factorial Plots, Pareto Effect Plots, ANOVA, Histograms, and Response Optimizer.

You are now ready to tackle some data, answer some questions, and become statistically dangerous.