---
title: "discovr repeated measures designs as multilevel models"
author: "Andy Field"
output:
  learnr::tutorial:
    progressive: true
    allow_skip: true
    theme: "united"
    highlight: "kate"
    css: ./css/discovr_style_future.css
runtime: shiny_prerendered
description: "Repeated measures designs. Fitting models for one- and two-way repeated measures designs using a multilevel model framework using `glmmTMB`. (All code will also work with `lme4`.) The examples match `discovr_15` but the modelling approach differs. This tutorial builds on previous ones to show how models can be fit with one or two categorical predictors when these variables have been manipulated within the same entities. We look at fitting the models, setting contrasts for the categorical predictors and diagnostic plots."
bibliography: [discovr_15_mlm.bib, packages.bib]
---
<html lang="en">


```{r setup, include=FALSE}
knitr::opts_chunk$set(
	echo = TRUE,
	message = FALSE,
	warning = FALSE
)

#necessary to render tutorial correctly
library(learnr) 
library(htmltools)
#easystats
library(datawizard)
library(insight)
library(modelbased)
library(parameters)
library(performance)
#tidyverse
library(dplyr)
library(ggplot2)
#non tidyverse/easystats
library(glmmTMB)
library(marginaleffects)
#students don't use
library(knitr)



source("./www/discovr_helpers.R")

#Read data files needed for the tutorial

sniff_tib <- discovr::sniffer_dogs
scent_tib <- discovr::alien_scents
```



# discovr repeated measures designs as multilevel models (GLM 4)

```{r, child = "./docs/intro.Rmd"}

```

## Packages {data-progressive=FALSE}

```{r, child = "./docs/packages.Rmd"}

```


## Data

```{r, child = "./docs/data.Rmd"}

```


## Fitting models

```{r, child = "./docs/fit_models.Rmd"}

```


## `r bmu()` Aliens and sniffer dogs [(B)]{.lbl}

The main examples in this tutorial are from [@fielddsr22026]. When the alien invasion comes well need spaniels (or possibly other dogs, but lets hope its mainly spaniels because spaniels are cool) to help us to identify the space lizards. Having got wind of a potential invasion from alien space lizards, some of whom could shapeshift into humanoid form, the top-secret government agency for Training Extra-terrestrial Reptile Detection (TERD) met to come up with a plan for detecting the invading space lizards. They decided to test the plausibility of training sniffer dogs to detect aliens. Over many trials 8 of their best dogs (Milton, Woofy, Ramsey, Mr. Snifficus III, Willock, The Venerable Dr. Waggy, Lord Scenticle, and Professor Nose) were recruited for a pilot study. During training, these dogs were rewarded for making vocalizations while sniffing alien space lizards (which they happened to have a few of in Hangar 18). On the test trial, the 8 dogs were allowed to sniff 4 entities for 1-minute each: an alien space lizard, a shapeshifting alien space lizard who had taken on humanoid form and worked undetected as a statistics lecturer, a human, and a human mannequin). The number of vocalizations made during each 1-minute sniffing session was recorded. So, this is a [repeated measures design]{.alt}: each dog has four scores representing the number of vocalizations they made while sniffing each of the four entities.


> - H~1~: If training has been successful the dogs should vocalise more when sniffing space lizards compared to when sniffing other things


## `r bmu()` Step 1: summarize [(A)]{.lbl}

<div class="stepbox">
  `r step()` **Step 1**

Get your data into `r rproj()` and pre-process using [tidyverse]{.pkg} packages or the [datawizard]{.pkg} package from [easystats]{.pkg}.
</div>


#### `r alien()` Alien coding challenge

View the data in [sniff_tib]{.alt}.

```{r sniff_tib, exercise = TRUE, exercise.lines = 2}

```

```{r sniff_tib-solution}
sniff_tib
```

Note that there are three variables:

- `dog_name`, which is a character variable (note the `<chr>` under the name),
- `entity` (alien, shapeshifter, human, mannequin), which is a factor (note the `<fct>` under the name)
- `vocalizations` (the number of vocalizations made during 1 minute of sniffing), which is numeric and has the data type 'double' (note the `<dbl>` under the name).

The data are in tidy format, which means that each row represents an instance of the outcome variable and the columns code information about each instance, for example, which dog the instance is related to and what they sniffed. Consequently each dog occupies 4 rows of the tibble (because each dog contributes four instances of the outcome variable, `vocalizations`).

The variable `entity` is a factor (categorical variable), so having read the data file and converted this variable to a factor it's a good idea to check the order of the levels of this variables.

#### `r alien()` Alien coding challenge

Using what you've learnt in previous tutorials check the order of the levels of the variables `entity`.

```{r chk_levels, exercise = TRUE, exercise.lines = 3}

```

```{r chk_levels-hint-1}
# use this function:
levels()
```

```{r chk_levels-hint-2}
# Remember that to access a variable you use:
name_of_tibble$name_of_variable
```

```{r chk_levels-hint-3}
# solution:
levels(sniff_tib$entity)
```

You should find that the levels are ordered as alien, human, mannequin, and shapeshifter.

#### `r alien()` Alien coding challenge

Use what you already know to create an object called [sniff_sum]{.alt} that is a table of descriptive statistics for vocalizations grouped by the entity sniffed and display it.

<div class="tip">
  `r cat_space()` **Tip**

When we group data and use `describe_distribution()` the resulting table has a column called [Variable]{.opt} that tells you which variable is being described. If we select a single variable to be described this column is redundant. We can remove it using `data_remove()` from [datawizard]{.pkg} and specifying ["Variable"]{.alt} as the variable to be removed. For example, to remove this column from our summary table ([sniff_sum]{.alt}) before rendering we can use:

```{r, eval = F, class.source = '.panel_alt'}
sniff_sum |> 
  data_remove("Variable") |>
  display()
```
</div>


```{r sniff_sum, exercise = TRUE, exercise.lines = 10}

```

```{r sniff_sum-hint-1}
# Start by piping the tibble into the group_by function to group output by entity:
sniff_sum <- sniff_tib |> 
  group_by(entity)
# Now pipe the results into the describe_distribution() function
```

```{r sniff_sum-hint-2}
# Pipe the results into the summarize() function
sniff_sum <- sniff_tib |> 
  group_by(entity) |> 
  describe_distribution(select = "vocalizations") 
# Now use display() to display the table (pay attention to the tip)
```


```{r sniff_sum-solution}
# Solution
sniff_sum <- sniff_tib |> 
  group_by(entity) |> 
  describe_distribution(select = "vocalizations") 

sniff_sum |> 
  data_remove("Variable") |> # optional to remove redundant column
  display()
```

It looks like the dogs made the most vocalizations when sniffing the alien and shapeshifting alien, which is what we would expect if training was successful.

## `r bmu()` Step 2: Visualize [(A)]{.lbl}

<div class="stepbox">
  `r step()` **Step 2**

Visualise the data using the [ggplot2]{.pkg} package from [tidyverse]{.pkg}.
</div>

#### `r alien()` Alien coding challenge

Use what you already know to create a violin plot of vocalization scores split by the entity (*x*-axis).


```{r sniff_gg, exercise = TRUE, exercise.lines = 7}

```

```{r sniff_gg-hint-1}
# Start by setting up the plot (replace the xs):
ggplot(sniff_tib, aes(x = xxxx, y = xxxx)) 

```

```{r sniff_gg-hint-2}
# Now use geom_violin() to add the distributions. Feel free to change the colour and fill by replacing #999933 and #DDCC77 with different HEX codes. 
ggplot(sniff_tib, aes(x = entity, y = vocalizations)) +
  geom_violin(colour = "#999933", fill = "#DDCC77", alpha = 0.3) 
```

```{r sniff_gg-hint-3}
# Now use stat_summary() to add the confidence intervals (replace the xs)
ggplot(sniff_tib, aes(x = entity, y = vocalizations)) +
  geom_violin(colour = "#999933", fill = "#DDCC77", alpha = 0.3) +
  stat_summary(fun.data = "xxxx", colour = "#999933")
```

```{r sniff_gg-hint-4}
# Use scale_y_continuous() to set the breaks for the y-axis to be whole numbers between 0 and 12 (replace the xs):
ggplot(sniff_tib, aes(x = entity, y = vocalizations)) +
  geom_violin(colour = "#999933", fill = "#DDCC77", alpha = 0.3) +
  stat_summary(fun.data = "mean_cl_normal", colour = "#999933") +
  scale_y_continuous(xxxx = xxxx) +
```

```{r sniff_gg-hint-5}
# use labs() to add axis labels to the x, y and colour legend (replace xxxs):
ggplot(sniff_tib, aes(x = entity, y = vocalizations)) +
  geom_violin(colour = "#999933", fill = "#DDCC77", alpha = 0.3) +
  stat_summary(fun.data = "mean_cl_normal", colour = "#999933") +
  scale_y_continuous(breaks = 0:12) +
  labs(y = "xxxx", x = "xxxx")
```

```{r sniff_gg-hint-6}
# add a theme (replace the xs):
ggplot(sniff_tib, aes(x = entity, y = vocalizations)) +
  geom_violin(colour = "#999933", fill = "#DDCC77", alpha = 0.3) +
  stat_summary(fun.data = "mean_cl_normal", colour = "#999933") +
  scale_y_continuous(breaks = 0:12) +
  labs(y = "Vocalizations (in 1 minute)", x = "Entity sniffed") +
  xxxxxx()
```

```{r sniff_gg-solution}
# Solution
ggplot(sniff_tib, aes(x = entity, y = vocalizations)) +
  geom_violin(colour = "#999933", fill = "#DDCC77", alpha = 0.3) +
  stat_summary(fun.data = "mean_cl_normal", colour = "#999933") +
  scale_y_continuous(breaks = 0:12) +
  labs(x = "Entity sniffed", y = "Vocalizations (in 1 minute)") +
  theme_minimal()
```

The plot reiterates what we know from the means: dogs made the most vocalizations when sniffing the alien and shapeshifting alien, which is what we would expect if training was successful.

## `r user_astronaut()` Step 3: Fit the model [(C)]{.lbl}

We can get a conceptual grasp of the model we're fitting by thinking about the predictor variable `entity` as a single variable

$$
\begin{aligned}
\text{vocalizations}_{ij} &= \left[\beta_{0j} + \beta_{1j}\text{entity}_{ij} \right]+ \left[u_{0j} + \varepsilon_{ij}\right] \\
\end{aligned}
$$

However, because `entity` is a categorical variable with four categories it will be represented by three dummy variables in the actual fitted model. We could use standard dummy coding (perhaps comparing each category to the mannequin because it is the only entity that is made of synthetic material), but instead we are going to set contrasts that specifically address our hypothesis.

> Hypothesis:
>
> - If training has been successful the dogs should vocalise more when sniffing space lizards compared to when sniffing other things
> 
> Associated contrasts:
> 
> - **Contrast 1**: {alien, shapeshifter} vs. {human, mannequin}
>
> We have two 'chunks' in contrast 1 that would then need to be decomposed:
>
> - **Contrast 2**: {alien} vs. {shapeshifter}
> - **Contrast 3**: {human} vs. {mannequin}

The resulting model is, therefore

$$
\begin{aligned}
\text{vocalizations}_{ij} &= \Big[\hat{\beta}_{0} + \hat{\beta}_{1}\text{alien vs. non}_{ij} + \hat{\beta}_{2}\text{alien vs. shapeshifter}_{ij} \\
&\quad + \hat{\beta}_{3}\text{human vs. mannequin}_{ij}\Big] + \Big[u_{0j} + \varepsilon_{ij}\Big] \\
\end{aligned}
$$

The main difference to models we have seen before is that we have added a term that reflects the deviation of each dog's intercept from the overall intercept ($u_{0j}$), and this term will have an associated variance that is estimated from the data. Put another way, we model the fact that individual dogs will vary in the overall number of vocalizations they make.


### `r user_visor()` Planned contrasts [(B)]{.lbl}

To code the contrasts from the previous section, we follow the rules for contrast coding (see `discovr_11`):

- Use $k-1$ contrasts. With 4 groups that means 3 contrasts. We have already defined these contrasts above.
- Each contrast compares two 'chunks' (which contain 1 or more categories). Assign one 'chunk' positive weights and the other negative.
- For each contrast, assign an initial weight to each group that is equal to the number of groups in the opposite chunk
- For each contrast, assign a final weight by dividing the initial weight by the number of groups in the contrast that have non-zero weights

Following these rules we'd end up with the contrasts in Table 1. (It might help you to figure out from where the values come to note that 2/4 = 1/2.)

```{r con_tbl, echo = FALSE}
con_tbl <- tibble(
  `Group` = c("Alien", "Human", "Mannequin", "Shapeshifter"),
  `Contrast 1 (aliens vs. non-aliens)` = c("1/2", "-1/2", "-1/2", "1/2"),
  `Contrast 2 (alien vs. shapeshifter)` = c("1/2", 0, 0, "-1/2"),
  `Contrast 3 (human vs. mannequin)` = c(0, "1/2", "-1/2", 0),
  )

knitr::kable(con_tbl, caption = "Table 1: Contrast coding for the entity variable")
```


#### `r alien()` Alien coding challenge

To set these contrasts, we first need to remember that the categories of `entity` are ordered (we checked earlier):

```{r, echo = F}
levels(sniff_tib$entity)
```

Remembering this order, to set the contrasts we create a list using `list()`. We give this list a name (for example, [sniff_cons]{.alt}) and within the list we specify each contrast in turn. On the left-hand side of the equals sign we give the contrast a name. For example, I have called the first contrast `aliens_vs_non`. You can call them what you like but the names can't contain spaces. On the right-hand side we specify the contrast weights in Table 1. For example, for contrast 1 we specify [c(-1/2, 1/2, 1/2, -1/2)]{.alt}, which, remembering the order of the levels of `entity`,  means that the alien and shapeshifter conditions are assigned weights of $\frac{1}{2}$ whereas the human and mannequin conditions are assigned weights of $-\frac{1}{2}$.

```{r, eval = F}
sniff_cons <- list(
  aliens_vs_non = c(1/2, -1/2, -1/2, 1/2),
  alien_vs_shape = c(1/2, 0, 0, -1/2),
  human_vs_manquin = c(0, 1/2, -1/2, 0)
  )
```

Having specified these contrasts, we can use the `contrast()` function to apply the contrasts as we have done in previous tutorials.

#### `r alien()` Alien coding challenge

Try assigning the contrasts to the variable `entity`.

```{r sniff_con, exercise = TRUE, exercise.lines = 8}

```

```{r sniff_con-solution}
sniff_cons <- cbind(
  aliens_vs_non = c(1/2, -1/2, -1/2, 1/2),
  alien_vs_shape = c(1/2, 0, 0, -1/2),
  human_vs_manquin = c(0, 1/2, -1/2, 0)
  )

contrasts(sniff_tib$entity) <-  sniff_cons
contrasts(sniff_tib$entity) # this line is to show us the contrasts to check we've set them correctly
```


Now, let's fit the model.

### `r user_visor()` Fitting a repeated measures model [(B)]{.lbl}

We can fit an overall model of type of entity predicting the number of dog vocalizations using the `glmmTMB()` function from the [glmmTMB package]{.pkg} [@R-glmmTMB; @glmmTMB2017; @glmmTMB2025], which we met in `discovr_14`. (All of the code will also work with the `glmmTMB()` function from [lme4]{.pkg}.)  As a reminder, the `glmmTMB()` function takes the following form:

```{r, eval = F}
my_model <- glmmTMB:glmmTMB(outcome ~ predictor(s) + (random effects), 
                          data = tibble,
                          family = gaussian()
                          na.action = an action,
                          REML = FALSE,
                          control = glmmTMBControl(),
                          subset = NULL)
```

Essentially it is similar to `lm()` in that we specify a formula that describes the model, and then there are some other arguments that we can use. By default maximum likelihood estimation is used but we can change to restricted maximum likelihood (REML) estimation by specifying [REML = TRUE]{.alt}, we can also fit the model to a subset of the data using the [subset]{.alt} argument. The [control]{.alt} argument allows us to change aspects of the fitting process. A common use of this is to specify a different optimizer if the model can't be fit. Perhaps most important is that we can specify a distribution family for the model. By default, this family is set to Gaussian, which is another name for the normal distribution. This default is fine for the models in this tutorial, but this argument allows us to fit a much wider range of models for different response distributions. For example, we can fit a multilevel version of logistic regression (`discovr_19`) by changing the argument to [family = binomial()]{.alt}.

Our model will have a random intercept for dogs, which we can specify using `(1|dog_name)` (see `discovr_14`).

#### `r robot()` Code example

To specify the current model we could execute

```{r, eval = F}
sniff_mlm <- glmmTMB::glmmTMB(vocalizations ~ entity + (1|dog_name), data = sniff_tib)
```

This code creates a model called [sniff_mlm]{.alt} that was specified in the equation above. 


#### `r alien()` Alien coding challenge

Use the `glmmTMB()` function to fit the model and store it as `sniff_mlm`.

```{r sniff_set_contrasts}
sniff_cons <- cbind(
  aliens_vs_non = c(1/2, -1/2, -1/2, 1/2),
  alien_vs_shape = c(1/2, 0, 0, -1/2),
  human_vs_manquin = c(0, 1/2, -1/2, 0)
  )

contrasts(sniff_tib$entity) <-  sniff_cons
```

```{r fit_sniff_mlm}
sniff_cons <- cbind(
  aliens_vs_non = c(1/2, -1/2, -1/2, 1/2),
  alien_vs_shape = c(1/2, 0, 0, -1/2),
  human_vs_manquin = c(0, 1/2, -1/2, 0)
  )

contrasts(sniff_tib$entity) <-  sniff_cons
sniff_mlm <- glmmTMB::glmmTMB(vocalizations ~ entity + (1|dog_name), data = sniff_tib)
```


```{r sniff_glmmTMB, exercise = TRUE, exercise.lines = 5, exercise.setup = "sniff_set_contrasts"}

```

```{r sniff_glmmTMB-hint-1}
# fit the model (replace the xs):
sniff_mlm <- glmmTMB::glmmTMB(xxxxx ~ xxxxx*xxxxx + (xxxx|xxxxx), data = xxxxx)
```

```{r sniff_glmmTMB-solution}
# fit the model:
sniff_mlm <- glmmTMB::glmmTMB(vocalizations ~ entity + (1|dog_name), data = sniff_tib)
```


```{r echo = F}
sniff_cons <- cbind(
  aliens_vs_non = c(1/2, -1/2, -1/2, 1/2),
  alien_vs_shape = c(1/2, 0, 0, -1/2),
  human_vs_manquin = c(0, 1/2, -1/2, 0)
  )

contrasts(sniff_tib$entity) <-  sniff_cons
sniff_mlm <- glmmTMB::glmmTMB(vocalizations ~ entity + (1|dog_name), data = sniff_tib)
sniff_lrt <- test_lrt(sniff_mlm)
sniff_fit <- model_performance(sniff_mlm)
sniff_pars <- model_parameters(sniff_mlm)
```

<div class="bug">
  `r bug()` **De-bug**

This code creates the models but doesn't display them so it will appear as though nothing has happened, but it has.
You will know this because you'll get this message:

```{r, eval = F, class.source = '.panel_alt'}
Random effect variances not available. Returned R2 does not account for random effects.
```

This is because we're fitting the model to such a small sample.
</div>

#### `r alien()` Alien coding challenge

We can get an overall test of the fit of the model using the `test_lrt()` function from [performance]{.pkg} to perform a likelihood ratio test. We put the name of the model we just fitted into the function. Try this out below (I've also used `display()` to tabulate the results).

```{r sniff_fit, exercise = TRUE, exercise.lines = 4, exercise.setup = "fit_sniff_mlm"}
test_lrt(sniff_mlm) |> 
  display()
```

Adding `entity` to the intercept only model significantly improved the fit, `r report_lrt(sniff_lrt, row = 2)`


```{r qn_sphericity_ex1, echo = F}
quiz(caption = "One-way repeated measures design quiz (level 2)",
question("How would you interpret the effect of `entity` in the output?",
         answer("The number of vocalizations was *not* significantly different across the entities sniffed because the *p*-value associated with the $\\chi^2$-statistic is 0.01, which is greater than than the criterion value of 0.05.", message = "This answer is incorrect, notwithstanding how pointless the 0.05 cutoff is, 0.01 is less than 0.05 making this effect significant."),
         answer("The number of vocalizations was significantly different across the entities sniffed because the *p*-value associated with the $\\chi^2$-statistic is 0.005, which is less than than than the criterion value of 0.05.", correct = T, message = "This answer is correct."),
         correct = "",
         allow_retry = T,
         random_answer_order = T)
)
```



## `r user_visor()` Step 4: Evaluate the model [(B)]{.lbl}

<div class="stepbox">
  `r step()` **Step 4**

Evaluate the model using a small number of functions from the [performance]{.pkg} package from [easystats]{.pkg}. Specially, we evaluate

- How well the model fits the data
- Whether the underlying assumptions of the model are met. If not, we refit the model using robust methods (see Figure 3).
 
</div>


As with previous models we can get fit statistics for our final model by placing it within `model_performance()`.

#### `r alien()` Alien coding challenge

Use the code box below to get fit statistics for the final model.

```{r sniff_perf, exercise = TRUE, exercise.lines = 4, exercise.setup = "fit_sniff_mlm"}
          
```


```{r sniff_perf-solution}
model_performance(sniff_mlm) |> 
  display()
```


<div class="bug">
  `r bug()` **De-bug**

Yep, it's that message we saw earlier again:

```{r, eval = F, class.source = '.panel_alt'}
Random effect variances not available. Returned R2 does not account for random effects.
```

Don't fit these models to small samples.

</div>


The marginal $R^2$ tells us that the proportion of variance attributable to only the fixed effects is `r value_from_ez(sniff_fit, value = "R2_marginal")` (`r percent_from_ez(sniff_fit, value = "R2_marginal")`), which is a sizeable amount.

#### `r alien()` Alien coding challenge

As with any linear model, we can use the `check_model()` function to produce diagnostic plots. Use the code box to do this.

```{r sniff_chk, exercise = TRUE, exercise.lines = 4, exercise.setup = "fit_sniff_mlm", fig.height=10, fig.width=8}
          
```


```{r sniff_chk-solution}
check_model(sniff_mlm)
```

With such a small sample these plots are hard to interpret but reference lines in the linearity and homoscedasticity plots are not flat and there's a lot of variance in the model predicted lines. The middle left Q-Q plot implies non-normality residuals because they deviate from the reference line at the extremes. The headline is that these models aren't really designed for teaching examples based on a tiny sample!

## `r bmu()` Step 5: Interpret the model [(A)]{.lbl}

<div class="stepbox">
  `r step()` **Step 5**

Interpret the model

- Interpret the values that define the model (the [parameter estimates]{.alt}) using the [parameters]{.pkg} package from [easystats]{.pkg}.
- Make specific predictions from our model using the [modelbased]{.pkg} package from [easystats]{.pkg}.
 
</div>


#### `r alien()` Alien coding challenge

To display the parameter estimates of the model in a nice table we use our old friend `model_parameters()`.


```{r show_sniff_pars, exercise = TRUE, exercise.lines = 4, exercise.setup = "fit_sniff_mlm"}
model_parameters(sniff_mlm) |> 
  display()
```


<div class="reportbox">
  `r pencil()` **Report**`r rproj()`

There was a significant effect of the type of entity on sniffer dog's vocalizations when approaching them, `r report_lrt(lrt = sniff_lrt)`. Contrasts revealed that vocalizations were significantly higher when sniffing aliens compared to non-aliens, `r report_pe(sniff_pars, row = 2)`), but vocalizations were not significantly different when sniffing different types of aliens, `r report_pe(sniff_pars, row = 3)` or when sniffing a human compared to a mannequin, `r report_pe(sniff_pars, row = 4)`).


</div>


Also note in the random effects that the variance for the intercept was `r value_from_ez(sniff_pars, row = 5)`, which you might think suggests that dogs did not differ at all in overall levels of vocalizations. However, this value relates to the error message in that there was insufficient data to estimate this parameter.


### `r user_visor()` *Post hoc* tests [(B)]{.lbl}

An alternative to contrasts is to compare all means to each other with *post hoc* tests. This procedure tends to be used when you have no specific *a priori* hypotheses (although why you'd be doing research without prior hypotheses is anyone's guess).

<div class="warning">
  `r warning()` **Advisory**

You would generally not conduct *post hoc* tests and contrasts, you would do one or the other, not both. 

</div>

#### `r robot()` Code example

We can obtain *post hoc* tests with a Bonferroni correction by placing our model into the `estimate_contrasts()` function (see `discovr_11`), using this code

```{r, eval = F}
estimate_contrasts(model = sniff_mlm,
                   p_adjust = "bonferroni",
                   ci = 0.95) |> 
  display()
```


#### `r alien()` Alien coding challenge

Get *post hoc* comparisons between the mean vocalizations across all combinations of entities. Use a Holm adjustment for multiple comparisons.

```{r sniff_ph, exercise = TRUE, exercise.lines = 5, exercise.setup = "fit_sniff_mlm"}

```

```{r sniff_ph-hint-1}
# Replace the xs
estimate_contrasts(model = xxxx,
                   p_adjust = "xxxx",
                   ci = xxxx)
# now display the table
```

```{r sniff_ph-solution}
estimate_contrasts(model = sniff_mlm,
                   p_adjust = "bonferroni",
                   ci = 0.95) |> 
  display()
```

```{r qn_ph_ex1, echo = F}
quiz(caption = "Post hoc test quiz (level 2)",
  question("Assuming an a priori alpha of 0.05, complete the following statement by ticking all responses that are correct. \"There were significantly more dog vocalizations when sniffing ...\"",
         answer("... an alien compared to a human.", correct = T, message = "The *p*-value of 0.005 is less than 0.05 and the mean is higher for aliens than humans so this statement is correct."),
         answer("... an alien compared to a mannequin.", correct = T, message = "The *p*-value of 0.003 is less than 0.05 and the mean is higher for aliens than mannequins so this statement is correct."),
         answer("... an alien compared to a shapeshifter.", message = "This statement is incorrect: the *p*-value of 0.239 is larger than 0.05 implying a non-significant difference between mean vocalizations."),
         answer("... a human compared to a shapeshifter.", message = "This statement is incorrect: the *p*-value of 0.99 is larger than 0.05 implying a non-significant difference between mean vocalizations."),
         answer("... a mannequin compared to a shapeshifter.", message = "This statement is incorrect: the *p*-value of 0.99 is larger than 0.05 implying a non-significant difference between mean vocalizations."),
         answer("... a human compared to a mannequin.", message = "This statement is incorrect: the *p*-value of 0.96 is larger than 0.05 implying a non-significant difference between mean vocalizations."),
         correct = "Correct - well done!",
         allow_retry = T,
         random_answer_order = T
)
)
```

### `r user_astronaut()` Robust models [(C)]{.lbl} 

You can fit a robust model using the `rlmer()` function from the `robustlmm` package [@R-robustlmm], which essentially takes the same syntax as `glmmTMB()`, so we could a robust version of our previous model using

```{r, eval = F}
sniff_rob <- robustlmm::rlmer(vocalizations ~ entity + (1|dog_name), data = sniff_tib)
# view the parameters
model_parameters(sniff_rob) |> 
  display()
```

There are other arguments that can be used to tweak the settings but the defaults will give robust estimates. For such a small dataset the above will almost certainly fail, and these robust models can take a lot of time to converge (if they converge at all) so this section is more for information - we will not attempt to fit this model.


## `r user_astronaut()` Factorial repeated measures designs [(C)]{.lbl}

The aliens, excited by humans' apparent inability to train sniffer dogs to detect them, decided to move their invasion plan forward. Aliens are far too wedded to *p*-values in small samples. They decided that they could make themselves even harder to detect by fooling the sniffer dogs by masking their alien smell. After extensive research they agreed that the two most effective masking scents would be human pheromones (which they hoped would make them smell human-like) and fox-pheromones (because they are a powerful, distracting smell for dogs). The aliens started smearing themselves with humans and foxes and prepared to invade.

Meanwhile, the top-secret government agency for Training Extra-terrestrial Reptile Detection (TERD) had got wind of their plan and set about testing how effective it would be. They trained 50 sniffer dogs. During training, these dogs were rewarded for making vocalizations while sniffing alien space lizards. On the test trials, the 50 dogs were allowed to sniff 9 different entities for 1-minute each: 3 alien space lizards, 3 shapeshifting alien space lizard who had taken on humanoid form, and 3 humans. Within each type of entity, 1 had no masking scent, 1 was smothered in human pheromones and 1 wore fox pheromones. The number of vocalizations made during each 1-minute sniffing session was recorded.

The model we're fitting is described by the following equation (which is simplified in that I have represented the predictor variables `entity` and `scent_mask` as single variables). (In fact, because both variables contain three categories they would each be entered into the model as two dummy variables, and the interaction would be 4 dummy variables. Under the hood the model, therefore, has 8 predictors.)

The design of the study is such that each dog sniffed 9 things. We denote these sniffs with the letter $i$. Also, let's denote the dogs with the letter $j$. Using these symbols, we can say that each of $i = 1, 2, \ldots , 9$ sniffs is nested within each of $j = 1, 2, \ldots , 50$ dogs. If we fit a random intercept model; that is, we estimate the variability in vocalizations due to individual differences between dogs, we would write the model as:

$$
\begin{aligned}
\text{vocalizations}_{ij} & = \left[\beta_{0} + \beta_{1}\text{entity}_{ij} + \beta_{2}\text{scent}_{ij} + \beta_{3}(\text{entity}_{ij}\times\text{scent}_{ij})\right] + \\
&\quad \ \left[u_{0j} + \varepsilon_{ij}\right]\\
\end{aligned}
$$

This model will capture the 'repeated measures' aspect of the design in that $u_{0j}$ represents the difference in vocalizations for a particular dog from the overall mean number of vocalizations, and the model will now include a parameter that estimates the variance in vocalizations across dogs ($\hat{\sigma}^2_{\mu_0}$).


<div class="infobox">
  `r info()` **The maximal model**

You obtain the maximal model by adding random effects for all predictors (including the interaction). If we again simplify by ignoring the fact that the model actually contains 8 dummy variables (2 for each main effect and 4 for the interaction), we'd be looking at this model:

$$
\begin{aligned}
\text{vocalizations}_{ij} &= \left[\beta_{0} + \beta_{1}\text{entity}_{ij} + \beta_{2}\text{scent}_{ij} + \beta_{3}(\text{entity}_{ij}\times\text{scent}_{ij})\right] + \\
&\quad \left[u_{0j} + u_{1j}\text{entity}_{ij} + u_{2j}\text{scent}_{ij} + u_{3j}(\text{entity}_{ij}\times\text{scent}_{ij}) + \varepsilon_{ij}\right]\\
\end{aligned}
$$

Even this simplified representation model includes 4 variance parameters and 6 covariances to be estimated from the data. (The actual model with dummy variables includes 8 variance parameters and 28 covariances.) This maximal model introduces a lot of complexity into the model and is likely to have too many parameters for the data to support. So, we will include only a random intercept overall.

</div>


With multiple predictors, assuming we care about testing the individual contributions of predictors, we need to build up the model in steps. One way to do this for the current example is to fit the following models

- Model 1 (intercept only): $\text{vocalizations}_{ij} = \left[\beta_{0} \right] + \left[u_{0j} + \varepsilon_{ij}\right]$
- Model 2 (entity only): $\text{vocalizations}_{ij} = \left[\beta_{0} + \beta_{1}\text{entity}_{ij} \right] + \left[u_{0j} + \varepsilon_{ij}\right]$
- Model 3 (main effects only): $\text{vocalizations}_{ij} = \left[\beta_{0} + \beta_{1}\text{entity}_{ij} + \beta_{2}\text{scent}_{ij}\right] + \left[u_{0j} + \varepsilon_{ij}\right]$
- Model 4 (full model): $\text{vocalizations}_{ij} = \left[\beta_{0} + \beta_{1}\text{entity}_{ij} + \beta_{2}\text{scent}_{ij} + \beta_{3}(\text{entity}_{ij}\times\text{scent}_{ij})\right] + \left[u_{0j} + \varepsilon_{ij}\right]$

The order of models 2 and 3 is determined by the fact we assume entity would have a significant effect in a larger sample based on the previous study. However, in many cases we will care mainly about the interaction so the order in which we enter main effects might be relatively arbitrary.  


### `r user_visor()` Planned contrasts [(B)]{.lbl}

There are no specific hypotheses this time around, but we still need to think about what to do with out categorical predictors (`scent_mask` and `entity`). We have a natural control group for the `entity`, which is when dogs sniffed a human (both other categories involve aliens). A sensible coding scheme would therefore be the default of dummy coding.

> Entity:
> 
> - **Contrast 1**: {alien} vs. {human}
> - **Contrast 2**: {shapeshifter} vs. {human}

We also have a natural control group for the scent masks, which is when no scent is applied. Again then, it would make sense to use dummy coding. 

> Scent mask:
> 
> - **Contrast 1**: {human} vs. {none}
> - **Contrast 2**: {fox} vs. {none}

Having thought about all this stuff, we can proceed with our 5-steps of model-fitting!

## `r bmu()` Step 1: summarize [(A)]{.lbl}

<div class="stepbox">
  `r step()` **Step 1**

Get your data into `r rproj()` and pre-process using [tidyverse]{.pkg} packages or the [datawizard]{.pkg} package from [easystats]{.pkg}.
</div>

#### `r alien()` Alien coding challenge

View the data in [scent_tib]{.alt}.

```{r scent_tib, exercise = TRUE, exercise.lines = 2}

```

```{r scent_tib-solution}
scent_tib
```

Note that there are four variables: the participant's `dog_id`, which is a character variable (note the `<chr>` under the name), the `entity` that the dog sniffed (alien, shapeshifter or human) and the `scent_mask` used to cover the entity's natural smell (none, human scent, fox scent), both of which are factors (note the `<fct>` under the names). Finally, `vocalizations` is a numeric variable and has the data type 'double' (note the `<dbl>` under the name).

In terms of the contrasts we want, we decided that for `entity` we'd compare each category to [human]{.alt} and for `scent` we'd compare each category to [none]{.alt}. Ideally we want to order the factors so that these reference categories are first; that way, we don't have to set contrasts explicitly and can use `r rproj()`'s default behaviour of comparing each category to the first. Let's check whether it is the case that for `entity` the first category is 'human' and for `scent_mask` the first category is 'none' (i.e. no scent was worn).

#### `r alien()` Alien coding challenge

Check the order of the levels of the variables `entity` and `scent_mask`.

```{r chk_levels2, exercise = TRUE, exercise.lines = 4}

```

```{r chk_levels2-hint-1}
# use this function:
levels()
```

```{r chk_levels2-hint-2}
# Remember that to access a variable you use:
name_of_tibble$name_of_variable
```

```{r chk_levels2-hint-3}
# solution:
levels(scent_tib$entity)
levels(scent_tib$scent_mask)
```

Because I have set up the data within this tutorial you should see that the levels are listed in the order that we want them when you execute the code. The information at the beginning of the tutorial about the data files gives you code to order the factor levels manually.

#### `r alien()` Alien coding challenge

Adapt the code from the previous example to create and display an object called [scent_sum]{.alt} that is a table of descriptive statistics for vocalizations grouped by the entity sniffed and the type of scent.


```{r scent_sum, exercise = TRUE, exercise.lines = 7}

```

```{r scent_sum-solution}
# Solution
scent_sum <- scent_tib |> 
  group_by(entity, scent_mask) |> 
  describe_distribution(select = "vocalizations") 

scent_sum |> 
  data_remove("Variable") |> # optional to remove redundant column
  display()
```

The number of vocalizations tended to be lower (on average) when sniffing humans compared to either type of alien. When sniffing aliens the number of vocalizations was largest when they wore no scent.


## `r bmu()` Step 2: Visualize [(A)]{.lbl}

<div class="stepbox">
  `r step()` **Step 2**

Visualise the data using the [ggplot2]{.pkg} package from [tidyverse]{.pkg}.
</div>

We'll come back to this step because we'll plot our contrasts and to do that we need to fit the model.

## `r user_astronaut()` Step 3: Fit the model [(C)]{.lbl}


```{r, echo = F}
scent_base <- glmmTMB::glmmTMB(vocalizations ~ 1 + (1|dog_id), data = scent_tib)
scent_ent <- update(scent_base, .~. + entity)
scent_scent <- update(scent_ent, .~. + scent_mask)
scent_int <- update(scent_scent, .~. + entity:scent_mask)
scent_lrt <- test_lrt(scent_base, scent_ent, scent_scent, scent_int)
scent_fit <- model_performance(scent_int)
scent_par <- model_parameters(scent_int)
```


The first thing to do is set the contrasts we want.

### `r user_visor()` Set contrasts [(B)]{.lbl}

Earlier we decided that for `entity` we'd compare each category to [human]{.alt} and for `scent` we'd compare each category to [none]{.alt}. We also checked that the factor levels for these variables were ordered such that for `entity` the first category is 'human' and for `scent_mask` the first category is 'none' (i.e. no scent was worn). Therefore, we don't have to set contrasts explicitly; instead we use `r rproj()`'s default behaviour of comparing each category to the first to get the contrasts we want.

<div class="tip">
  `r cat_space()` **Tip**

If the categorical variables had not had their levels in the order we want, we can use `fct_relevel()` to re-order them, but we can also use `cont.treatment(base = x)` to set the reference category to level `x`. For example, if we want to set the reference category to be the first category we'd use `base = 1`, but we could set the second category to be the reference with `base = 2` and so on). Although we don't need to do this for the current data, if we were to use this function we'd set contrasts as follows:

```{r, eval = F, echo = T, class.source = '.panel_alt'}
contrasts(scent_tib$entity) <- contr.treatment(3, base = 1)
contrasts(scent_tib$scent_mask) <- contr.treatment(3, base = 1)
```

</div>

### `r user_astronaut()` Fit the models [(C)]{.lbl}

Having set up contrasts (or at least set up our categorical predictors to have the correct reference category) we can use the `glmmTMB()` function in much the same way as in the previous example. We saw earlier that we'll build our model sequentially.

#### `r robot()` Code example

To recap, the `glmmTMB()` function can be stripped back to this general format:

```{r, eval = F}
my_model <- glmmTMB(outcome ~ predictor(s) + (random_effects), 
                 data = tibble)
```

We can, therefore, specify the models we defined earlier as:

```{r, eval = F}
scent_base <- glmmTMB::glmmTMB(vocalizations ~ 1 + (1|dog_id), data = scent_tib)
scent_ent <- glmmTMB::glmmTMB(vocalizations ~ entity + (1|dog_id), data = scent_tib)
scent_scent <- glmmTMB::glmmTMB(vocalizations ~ entity + scent_mask + (1|dog_id),data = scent_tib)
scent_int <- glmmTMB::glmmTMB(vocalizations ~ entity + scent_mask + entity:scent_mask + (1|dog_id), data = scent_tib)
```

Note that in each model we add something to the formula.

- Model 1 (`scent_base`) is created using `vocalizations ~ 1` to predict vocalisations from only the intercept (`1`), which is allowed to vary across dogs (`+ (1|dog_id)`)
- Model 2 (`scent_ent`) changes only the formula to add entity as a predictor (`vocalizations ~ entity`)
- Model 3 (`scent_scent`) adds scent mask as a predictor (`vocalizations ~ entity + scent_mask`)
- Model 4 (`scent_int`) adds the interaction (`entity:scent_mask`) as a predictor (`vocalizations ~ entity + scent_mask + entity:scent_mask`)

Alternatively, I prefer to create sequential models using the `update()` function:

```{r, eval = F}
scent_base <- glmmTMB::glmmTMB(vocalizations ~ 1 + (1|dog_id), data = scent_tib)
scent_ent <- update(scent_base, .~. + entity)
scent_scent <- update(scent_ent, .~. + scent_mask)
scent_int <- update(scent_scent, .~. + entity:scent_mask)
```


<div class="infobox">
  `r info()` **Refresher of the `update()` function**

The `update()` function takes the general form:

```{r, eval = F, class.source = '.panel_alt'}
new_model <- update(old_model, update_instructions)
```

Where [new_model]{.alt} is the name assigned to the updated model, [old_model]{.alt} is the name of the model you're updating, and [update_instructions]{.alt} specify how to update the old model. The update instructions use a tilde (`~`) to represent the tilde in the original formula, a dot on the left to indicate 'the existing outcome', and a dot on the right to indicate 'the existing predictors'. From this we get the following possibilities:

-	`.~.` means keep the original outcome and any existing predictors.
-	`.~` means keep the original outcome but replace all existing predictors.
-	`~.` means replace the original outcome but keep all existing predictors.

We can add or subtract predictors using the `+` and `–` symbols.

</div>


#### `r alien()` Alien coding challenge

Use the `glmmTMB()` function to fit the four models and compare them using `test_lrt()`.

```{r scent_glmmTMB, exercise = TRUE, exercise.lines = 10}

```

```{r scent_glmmTMB-hint-1}
# fit the model (replace the xs):
scent_base <- glmmTMB::glmmTMB(xxxxx ~ 1 + (1|xxxxx), data = xxxxx)
# now use update() to create the next model
```

```{r scent_glmmTMB-hint-2}
scent_base <- glmmTMB::glmmTMB(vocalizations ~ 1 + (1|dog_id), data = scent_tib)
# use update (replace the xs)
scent_ent <- update(xxxxx, .~. + xxxxx)
```

```{r scent_glmmTMB-hint-3}
scent_base <- glmmTMB::glmmTMB(vocalizations ~ 1 + (1|dog_id), data = scent_tib)
scent_ent <- update(scent_base, .~. + entity)
# now use update() to create the next model (replace the xs)
scent_scent <- update(xxxxx, .~. + xxxxx)
```

```{r scent_glmmTMB-hint-4}
scent_base <- glmmTMB::glmmTMB(vocalizations ~ 1 + (1|dog_id), data = scent_tib)
scent_ent <- update(scent_base, .~. + entity)
scent_scent <- update(scent_ent, .~. + scent_mask)
# now use update() to create the next model (replace the xs)
scent_int <- update(xxxxx, .~. + xxxxx:xxxxx)
```

```{r scent_glmmTMB-hint-5}
scent_base <- glmmTMB::glmmTMB(vocalizations ~ 1 + (1|dog_id), data = scent_tib)
scent_ent <- update(scent_base, .~. + entity)
scent_scent <- update(scent_ent, .~. + scent_mask)
scent_int <- update(scent_scent, .~. + entity:scent_mask)
```

```{r scent_glmmTMB-solution}
scent_base <- glmmTMB::glmmTMB(vocalizations ~ 1 + (1|dog_id), data = scent_tib)
scent_ent <- update(scent_base, .~. + entity)
scent_scent <- update(scent_ent, .~. + scent_mask)
scent_int <- update(scent_scent, .~. + entity:scent_mask)
```

<div class="bug">
  `r bug()` **De-bug**

This code creates the models but doesn't display them so it will appear as though nothing has happened, but it has.

</div>

Having fit the models we can compare them using a likelihood ratio test as we did in the previous example:

```{r, eval = F}
test_lrt(scent_base, scent_ent, scent_scent, scent_int) |> 
  display()
```

#### `r alien()` Alien coding challenge

Use the code box below to compare the models we have just created.

```{r scent_models}
scent_base <- glmmTMB::glmmTMB(vocalizations ~ 1 + (1|dog_id), data = scent_tib)
scent_ent <- update(scent_base, .~. + entity)
scent_scent <- update(scent_ent, .~. + scent_mask)
scent_int <- update(scent_scent, .~. + entity:scent_mask)
scent_emm <- modelbased::estimate_means(scent_int, by = c("scent_mask", "entity"))
```



```{r compare_mods, exercise = TRUE, exercise.lines = 4, exercise.setup = "scent_models"}
          
```


```{r compare_mods-solution}
test_lrt(scent_base, scent_ent, scent_scent, scent_int) |> 
  display()
```

<div class="reportbox">
  `r pencil()` **Report**`r rproj()`

There were significant main effects of the type of entity `r report_lrt(scent_lrt, row = 2)`, and the scent mask used, `r report_lrt(scent_lrt, row = 3)`, on sniffer dog's vocalizations. Most important, the type of scent used significantly moderated the effect of entity on sniffer dog's vocalizations,  `r report_lrt(scent_lrt, row = 4)`.

</div>

## `r user_visor()` Step 4: Evaluate the model [(B)]{.lbl}

<div class="stepbox">
  `r step()` **Step 4**

Evaluate the model using a small number of functions from the [performance]{.pkg} package from [easystats]{.pkg}. Specially, we evaluate

- How well the model fits the data
- Whether the underlying assumptions of the model are met. If not, we refit the model using robust methods (see Figure 3).
 
</div>

As with previous models we can get fit statistics for our final model [scent_int]{.alt} by placing it within `model_performance()`.

#### `r alien()` Alien coding challenge

Use the code box below to get fit statistics for the final model.

```{r mod_fit, exercise = TRUE, exercise.lines = 4, exercise.setup = "scent_models"}
          
```


```{r mod_fit-solution}
model_performance(scent_int) |> 
  display()
```


The intraclass correlation, ICC, indicates that `r value_from_ez(scent_fit, value = "ICC")` (`r percent_from_ez(scent_fit, value = "ICC")`) of the variance in vocalizations is attributable to the dog. In short, there were individual differences in dog vocalizations. The conditional $R^2$ tells us the proportion of variance attributable to both the fixed and random effects (`r value_from_ez(scent_fit, value = "R2_conditional")` or `r percent_from_ez(scent_fit, value = "R2_conditional")`) whereas the marginal $R^2$ tells us the proportion of variance attributable to only the fixed effects (`r value_from_ez(scent_fit, value = "R2_marginal")` or `r percent_from_ez(scent_fit, value = "R2_marginal")`).



#### `r alien()` Alien coding challenge

As with any linear model, we can use the `check_model()` function to produce diagnostic plots. Use the code box to do this for the final model ([scent_int]{.alt}).

```{r mod_chk, exercise = TRUE, exercise.lines = 4, exercise.setup = "scent_models", fig.height=10, fig.width=8}
          
```


```{r mod_chk-solution}
check_model(scent_int)
```

The top right and middle left plots can be used for testing homoscedasticity. Both plots imply heteroscedasticity because the trend lines are not flat. The bottom left Q-Q plot suggests some non-normality of residuals because they deviate from the line at the extremes. The bottom right plot suggests relative normality of the random effects (an additional assumption for our model). The middle right plot shows indicates collinearity in the interaction term (which we would expect), and it appears highly collinear with both terms that contribute to it (both VIFs are close to 10).

## `r bmu()` Step 5: Interpret the model [(A)]{.lbl}

<div class="stepbox">
  `r step()` **Step 5**

Interpret the model

- Interpret the values that define the model (the [parameter estimates]{.alt}) using the [parameters]{.pkg} package from [easystats]{.pkg}.
- Make specific predictions from our model using the [modelbased]{.pkg} package from [easystats]{.pkg}.
 
</div>

The interaction effect supersedes the main effects, so let's concentrate on interpreting that with the parameter estimates form the model.

<div class="tip">
  `r cat_space()` **Tip**

With multilevel models, `model_parameters()` extracts both the fixed and random effects. However, we can extract *only* the fixed effects by including [effects = "fixed"]{.alt}. For example:

```{r, eval = F, class.source = '.panel_alt'}
model_parameters(scent_int, effects = "fixed") |> 
  display()
```

Similarly, we can extract *only* the random effects by including [effects = "random"]{.alt}. For example:

```{r, eval = F, class.source = '.panel_alt'}
model_parameters(scent_int, effects = "random") |> 
  display()
```

</div>

#### `r alien()` Alien coding challenge

Display the parameter estimates for only the fixed effects of the final model ([scent_int]{.alt}) in a nice table using `model_parameters()` (use `display()` to print them).


```{r show_scent_pars, exercise = TRUE, exercise.lines = 4, exercise.setup = "scent_models"}

```

```{r show_scent_pars-solution}
model_parameters(scent_int, effects = "fixed") |> 
  display()
```



<div class="tip">
  `r cat_space()` **Tip**

For each contrast it's useful to get a quick plot of what it represents. Rather than spending ages getting a  publication-ready plot, we can get a quick plot using `estimate_means()` and `plot()` (see `discovr_14`):

1. Get the estimated marginal means for all combinations of `entity` and `scent_mask` using `estimate_means()` from the [modelbased]{.pkg} package [@R-modelbased]:

```{r, eval = F, echo = T, class.source = '.panel_alt'}
scent_emm <- estimate_means(scent_int, by = c("scent_mask", "entity"))
```

2. Use `filter()` to remove the categories you don't want to plot. For example, contrast 1 does not involve the alien entity, of the fox scent. Remembering that `!=` means not equal to, we can filter out both categories using

```{r, eval = F, echo = T, class.source = '.panel_alt'}
filter(entity != "Alien" & scent_mask != "Fox")
```

which translates as *entity is not equal to the word "Alien" and scent_mask is not equal to the word "Fox"*. This will leave us with the means for humans and shapeshifters and no scent and human scent.

3. Use the `plot()` function from the `see` package to plot these means on top of the data. The resulting plot is a [ggplot2]{.pkg} object so we can add a theme, labels and so on in the usual way. So, to plot the means in contrast 1 we'd use:

```{r, eval = F, echo = T, class.source = '.panel_alt'}
scent_emm |> 
  filter(entity != "Alien" & scent_mask != "Fox") |> 
  plot() + theme_minimal()
```

</div>

We want to focus on the last four effects, which relate to the interaction. The first contrast for the interaction looks at level 2 of `entity` (shapeshifter) compared to level 1 (human), when a human scent (level 2) is used compared to no scent (level 1). 

#### `r alien()` Alien coding challenge

Let's get a quick plot of these means. We can use the code in the tip.

```{r c1_plot, exercise = TRUE, exercise.lines = 6, exercise.setup = "scent_models"}
scent_emm <- modelbased::estimate_means(scent_int, by = c("scent_mask", "entity"))
scent_emm |> 
  filter(entity != "Alien" & scent_mask != "Fox") |> 
  plot() + theme_minimal()
```


<div class="reportbox">
  `r pencil()` **Report**`r rproj()`

The first contrast was significant, `r report_pe(scent_par, row = 6)`. The plot shows that the significance means that the distance between the lines when no scent is used is greater than the distance between the lines when a human scent is used. In other words, the difference in the number of vocalisations (which reflect a difference when dogs sniff shapeshifters compared to humans) is more pronounced when no scent is used compared to human scent. Basically, dogs' abilities to differentiate shapeshifters from humans is reduced by applying a human scent.

</div>


#### `r alien()` Alien coding challenge

Contrast 2 looks at level 3 of `entity` (alien) compared to level 1 (human), when a human scent (level 2) is used compared to no scent (level 1). Adapt the code you used above to get a quick plot of the means in this contrast.

```{r c2_plot, exercise = TRUE, exercise.lines = 6, exercise.setup = "scent_models"}
xxxxx |> 
  filter(entity != xxxxx & scent_mask != xxxxx) |> 
  plot() + theme_minimal()
```

```{r c2_plot-solution}
scent_emm |> 
  filter(entity != "Shapeshifter" & scent_mask != "Fox") |> 
  plot() + theme_minimal()
```



<div class="reportbox">
  `r pencil()` **Report**`r rproj()`

The second contrast was significant, `r report_pe(scent_par, row = 7)`. The plot shows that the significance means that the distance between the lines when no scent is used is greater than the distance between the lines when a human scent is used. In other words, the difference in the number of vocalisations (which reflect a difference when dogs sniff an alien that looks like a lizard compared to humans) is more pronounced when no scent is used compared to human scent. Basically, dogs' abilities to differentiate aliens from humans is reduced by applying a human scent.

</div>


#### `r alien()` Alien coding challenge

Contrast 3 looks at level 2 of `entity` (shapeshifter) compared to level 1 (human), when a fox scent (level 3) is used compared to no scent (level 1). Adapt the code you used above to get a quick plot of the means in this contrast.

```{r c3_plot, exercise = TRUE, exercise.lines = 6, exercise.setup = "scent_models"}
xxxxx |> 
  filter(entity != xxxxx & scent_mask != xxxxx) |> 
  plot() + theme_minimal()
```

```{r c3_plot-solution}
scent_emm |> 
  filter(entity != "Alien" & scent_mask != "Human") |> 
  plot() + theme_minimal()
```



<div class="reportbox">
  `r pencil()` **Report**`r rproj()`

The third contrast was significant, `r report_pe(scent_par, row = 8)`. The plot shows that the significance means the significance means that the distance between the lines when no scent is used is greater than the distance between the lines when fox scent is used. In other words, the difference in the number of vocalisations (which reflect a difference when dogs sniff shapeshifters compared to humans) is more pronounced when no scent is used compared to fox scent. Dogs' abilities to differentiate shapeshifters from humans is reduced by applying a fox scent.

</div>

#### `r alien()` Alien coding challenge

Contrast 4 looks at level 3 of `entity` (alien) compared to level 1 (human), when a fox scent (level 3) is used compared to no scent (level 1). Adapt the code you used above to get a quick plot of the means in this contrast.

```{r c4_plot, exercise = TRUE, exercise.lines = 6, exercise.setup = "scent_models"}
xxxxx |> 
  filter(entity != xxxxx & scent_mask != xxxxx) |> 
  plot() + theme_minimal()
```

```{r c4_plot-solution}
scent_emm |> 
  filter(entity != "Shapeshifter" & scent_mask != "Human") |> 
  plot() + theme_minimal()
```



<div class="reportbox">
  `r pencil()` **Report**`r rproj()`

The final contrast was significant, `r report_pe(scent_par, row = 9)`. The plot shows that the significance means the significance means that the distance between the lines when no scent is used is greater than the distance between the lines when fox scent is used. In other words, the difference in the number of vocalisations (which reflect a difference when dogs sniff aliens compared to humans) is more pronounced when no scent is used compared to fox scent. Dogs' abilities to differentiate aliens from humans is reduced by applying a fox scent.

</div>

### `r user_astronaut()` Random effects [(C)]{.lbl}

#### `r alien()` Alien coding challenge

Display the random effects of the final model ([scent_int]{.alt}).

```{r show_scent_ranpars, exercise = TRUE, exercise.lines = 4, exercise.setup = "scent_models"}

```

```{r show_scent_ranpars-solution}
model_parameters(scent_int, effects = "random") |> 
  display()
```

The standard deviation of intercepts was $\hat{\sigma}_{u_{0}}$ = `r value_from_ez(scent_par, row = 10)` suggesting there was variance in overall vocalisations across dogs.


### `r user_astronaut()` Robust models [(C)]{.lbl} 

Again we can (but we won't) fit a robust model using the `rlmer()` function from the `robustlmm` package [@R-robustlmm], which essentially takes the same syntax as `glmmTMB()`. We could fit a robust version of our final model using

```{r, eval = F}
scent_rob <- robustlmm::rlmer(vocalizations ~ entity*scent_mask (1|dog_name), data = scent_tib)
```


<div class="infobox">
  <img src="./images/discovr_hex.png" alt="discovr package hex sticker, female space pirate with gun. Gunsmoke forms the letter R." style="width:100px;height:116px;" class = "img_left">
  
  **A message from Mae Jemstone:**
  
  Repeated measures experimental designs are a very efficient way to test hypotheses. Back in my trainee days, we were about to do battle with the Crimson Riders, a band of space outlaws who spread statistical misinformation. If you think that a confidence interval relates to your confidence in a value then your brain has probably been infiltrated by the Crimson Riders' propaganda. The thing about this band of reprobates is that they're very hard to track down. To work out how to defeat them we needed to test their reaction to different doses of a truth serum we had developed, but we could only capture a dozen or so of them. Enter the repeated measures design and victory. Although, their foul work survived that particular battle. Thanks to your hard work, you can now analyse repeated measures designs. Happy days cadets - well done!
</div>

## Resources/References {data-progressive=FALSE}

```{r, child = "./docs/resources.Rmd"}

```


### References
