What to Do with a Significant Interaction in Two-Way ANOVA?

Posted on
ANOVA f test Sum of Squares

We discussed how to conduct a 2-way factorial ANOVA in this tutorial, and we talked about the three different types of Sum of Squares here. We will build on these and discuss how to run post hoc analyses when you have a significant interaction. We will use the Moore dataset from the carData package in R. This data frame consists of subjects in a “social-psychological experiment who were faced with manipulated disagreement from a partner of either of low or high status. The subjects could either conform to the partner’s judgment or stick with their own judgment.” (John Fox, Sanford Weisberg and Brad Price (2018). carData: Companion to Applied Regression Data Sets. R package version 3.0-2. https://CRAN.R-project.org/package=carData)

The variables we will use in our analyses are:

  • partner.status: Partner’s status. A factor with levels high and low.
  • fcategory: F-Scale Categorized. A factor with levels low, medium, and high.
  • conformity (outcome): Number of conforming responses in 40 critical trials.

First, we visualize the data. The following boxplot shows the distribution of scores on the conformity variable within each combination of partner.status and fcategory.

We can also get a sense of whether an interaction is present by looking at an interaction plot.

Recall from the two-way ANOVA tutorial that if there is an interaction, the difference in means between treatment levels will be different depending on the level of the other factor, i.e. the plots of means will not be parallel. Looking at this plot, there appears to be an interaction between partner.status and fcategory. The difference in means in the two partner status levels is small when F-score category is low but larger when the F-score category is medium or high.

Next, we will run our 2-way ANOVA, and get the following results (Note that we are using type III Sum of Squares):

Source Type III Sum of Squares df Mean Square F Sig.
partner.status 239.562 1 239.562 11.425 .002
fcategory 36.019 2 18.009 .859 .431
partner.status*fcategory 175.489 2 87.744 4.185 .023
Error 817.764 39 20.968

From these results, we can conclude that, based on a significance level of \(\alpha = 0.05\):

  • The main effect of Partner Status (\(p = 0.002\)) is significant
  • The main effect of F-Score Category (\(p = 0.431\)) is not significant
  • The interaction effect (\(p = 0.023\)) is significant

The interaction effect is significant in the overall ANOVA, but that knowledge is not meaningful unless you look at the pairwise comparisons. Within each level of fcategory (“low”, “medium”, and “high”) we will perform pairwise comparisons to partner.status.

Partner Status \(df_1\) \(df_2\) F-ratio p-value
Low 2 39 2.323 0.111
High 2 39 2.138 0.132

The effect of F-Score appears to not be significant in either case. That is, the difference in means between F-score categories are not significantly different from each when partner status is low, nor are they significantly different when partner status is high.

Next, we will repeat this for partner status in different levels of F-score category:

F-Score Category \(df_1\) \(df_2\) F-ratio p-value
Low 1 39 11.486 0.002
Medium 1 39 6.899 0.012
High 1 39 0.105 0.748

At an \(\alpha = 0.05\) level, the effect of partner status within the low and medium F-score categories are significant (\(p = 0.002\) and \(p = 0.012\) respectively). Partner status does not appear to have a significant effect on conformity for subjects with high F-scores.

Finally, we will do pairwise comparisons for levels of partner status within the F-score categories that yielded significant results. That is, we’ll compare high versus low status among subjects with low F-scores, and then we’ll do the same among subjects with medium F-scores. We ignore the high F-score subjects because we did not find a significant effect. Note that, because partner status only has two categories, we only perform a single pairwise comparison within each F-score category.

F-Score Category Contrast Estimate SE \(df\) \(t\)-ratio p-value
Low Low - High -8.500 2.508 39 -3.389 0.002
Medium Low - High -7.023 2.674 39 -2.627 0.012