T-Test? ANOVA? Chi-Square? Statistical analysis of survey results is a critical step that many armchair researchers overlook when conducting analysis. Since surveys typically involve collecting data from only a portion of the total population, a statistical significance test should be used to help determine whether a difference between two numbers is likely to reflect a meaningful difference. Factors such as sample size, difference in shares, and magnitude of preference all contribute to whether differences are meaningful or due to chance. However, there are many different options when it comes to analyzing the data and identifying results that translate to actionable plans. Below, we’re offering a quick primer on some of these options – T-Test, Z-Test, ANOVA, Chi-Square – just a few of the tools that our statisticians and analysts frequently use for clients’ research projects.
First, A Note About Confidence Levels
Before deciding what significance test to use, you need to determine a confidence level. In market research a common confidence level used is 95 percent. This would mean that if the research were repeated 100 times, you would see the same (or similar) results 95 of those times. A confidence level as rigorous at 99 percent is usually only used for medical research, where very precise results are required. A confidence level of 90 percent is often considered too low and can result in more measurement errors, where a survey result is considered a real difference when in fact it is not.
Using the following tests with our recommended 95% confidence level can help you measure real differences in your collected data. Here are some of the most common statistical tests used for research data:
The t-test compares two measures (such as share of preference, mean, etc.) and tells you the statistical difference between two measures. Statistical significance is determined by the size of the difference between the group measures, the sample size, and the standard deviations of the groups. In a t-test, the sample size is generally small (usually less than n=30), and researchers do not need to know the standard deviation ahead of time*.
When to use: Use a t-test when you have two measures you want to compare to determine if they are different from each other and you don’t know the standard deviation.
When to use a 1-tail or 2-tail test
The t and z distributions are typically symmetrical – that is most of the observed results occur in the middle of the curve (the mean) and fewer results occur as you get farther away either on the left side of the curve (results are less than the mean) or the right side of the curve (results are greater than the mean).
Different formulas for either a one or two tail analysis will be used. The specific usage of a one- or two- tailed test will depend on your desired output. The “tail” refers to the end of the distribution of the test statistic for the specific analysis that you are using. Essentially, a one-tail tests allow for the possibility of a difference in just one direction, while a two-tail test looks at the difference in both directions.
Fig.1 The area under the curve (95%) represents your confidence level. This means that 95% of all results will fall within these bounds. In a one-tailed test (at 95% confidence) you are only concerned with the high or low values, while in the two tailed test you are worried about both.
If you are using a t-test or z-test, which has a two-tailed distribution, you will first have to determine what test will work best for your specific analysis.
When to use
- A two-tailed test is appropriate if you want to determine if there is any difference between the groups you are comparing. For example, to see if Group 1 scored higher or lower than Group 2, you would use a two-tailed test. A two-tailed test uses both the positive and negative tails of the distribution and it tests for the possibility of positive or negative differences.
- A one-tailed test is appropriate if you only want to determine if there is a difference between groups in a specific direction. If you are only interested in determining if Group 1 scored higher than Group 2, and you are completely uninterested in the possibility of Group 1 scoring lower than Group 2, then you want to use a one-tailed test.
For market research, it is almost always more appropriate to use a two-tailed test. A one-tailed test is justified if you have a very specific prediction about the direction of the difference like Group 1 scoring higher than Group 2, and you are completely uninterested in the possibility that the opposite result could be true.
ANOVA or the “Analysis of Variance” is a statistical method used to compare the measures of more than two populations or groups. It tests whether the measures of multiple groups are equal or not. For example, in the table below, the ANOVA will test if the means of the four groups are essentially equal.
While the ANOVA will be able to tell you overall if any of the means are statistically different, it will not be able to tell you what specific group or groups are different.
When to use: Use ANOVA testing when you want to see if any of the means (averages) of specific groups are different from each other.
Research data is often collected by various segments or categories (gender, income, region of the country, etc.). The Chi-Square test informs researchers about if there is a statistically significant difference between how the various segments or categories answered a given question. The test is appropriate for nominal data (the ANOVA test uses continuous data).
When to use: Use Chi-Square to determine whether or not there is a statistically significant difference between how various (usually two) groups answered a given question.
Using the appropriate test for your analysis can help you understand what your results really mean. On the other hand, using statistical tests inappropriately can lead to invalid results that are not replicable and highly questionable. You should know what test you plan to use as you’re designing your research project. The above guidelines give you a very brief overview of what technique to use and when. However, if you’re still wondering what analysis might be best for the data you’ve collected, partnering with a research supplier such as the Stevenson company can help you better grasp why or why not your study did or did not achieve statistical significance.