Much of the hard work behind building disease simulation models is deciding which compartments to include and how best to model the transition of individuals between them. Once we have generated our list of model parameters, we then need to estimate numeric values for them. Sometimes we can do this using existing data from published studies. Sometimes we have to design new field studies to collect our own data to generate good estimates. When all else fails, we can take our best guess based on expert opinion and then conduct a sensitivity analysis to see how much the model results might change if we used a different value instead. We may also choose to make our model stochastic rather than deterministic to incorporate an additional element of uncertainty in the outcomes. Lastly, we want to check our final model against field data from infected herds to make sure that its producing reasonable representations of the real world. It is important that our basic model of disease transmission dynamics is operating well before we try to impose disease control measures on the system.
By the end of this module, participants should be able to:
Statistician George Box famously said in 1978 that “All models are wrong, but some are useful”. As much as we would like, we can never truly capture all of the complexities that occur in biological systems through just a few simple boxes and equations. The question becomes “Is our model good enough to support the inferences or decisions we wish to make from the outputs?”. In an ideal world, we would always have detailed field data on the infection dynamics in real herds by collecting serial diagnostic samples, monitoring how individuals progress through the different disease and immunological states, and generating good epidemic curves. We could then compare our model predictions against the real data to make sure the dynamics have been adequately described. Unfortunately, this kind of data is not always available and so we resort to other methods to get a sense of how well our model is behaving.
The easiest parameters to estimate for a model are the ones describing what happens to an individual after they have become exposed to the virus. The data for this is generally collected from experimental studies where they take susceptible animals, give them an infectious dose of the pathogen, and then collect serial diagnostic samples to check for the presence of antigen (indicating active shedding) or antibody (indicative of an immune response). From this we can get data on the average length of the latent period, incubation period, infectious period, and recovery period. We then simply take the inverse of these values to estimate a rate for use in compartmental models. If we follow the animals for long enough over time, we can identify dynamics the presence of carrier states or the rate of return to susceptibility. We can also use the results from observational studies or even expert opinion to help develop estimates for parameters. For example, we might use estimates of incidence risk or incidence risk ratios from observational studies to help determine which animals are more likely to be susceptible to getting infected or we can track animals over time to monitor the effects of disease on animal health and production.
For food animal production systems, the demographic structure and demographic parameters such as births, transitions between production or physiological states, and deaths are most often obtained through expert opinion on what is normal behaviour for a farm. The nice thing about programming the simulation model code yourself is that you can then change these values to see what the outcomes would be for a herd with different management practices. Wildlife populations are much more difficult, but you can do mark recapture studies or longitudinal tracking studies to estimate population size and turnover.
The most challenging parameter to estimate is the transmission coefficient β, which represents the rate at which susceptible animals become infected as the result of the infection pressure exerted by infectious animals. Remember that this a function of both the average contact rate between individuals in population, the virulence of the pathogen, and probability of transmission occurring through the contact. If you have data on the number of new infections occurring at each time step plus the total number of S and the total number of I, you can set this up as a regression equation with no intercept (# of new infections = β * S * I) and fit a regression line to it to estimate the slope. This data is generally obtained from intensive studies in newly infected herds where repeat sampling is conducted in the whole herd at frequent intervals to determine when seroconversion occurred.
The Approximate Bayesian Computation (ABC) approach is another useful method for estimating parameters values and/or comparing different model structures. It is particular useful in situations where there multiple parameters with uncertainty and/or you have many different model structures to compare. The basic idea is that you select one or more model outcome measures that would indicate that your simulation model is accurately matching the disease dynamics observed in the field. These will end being whatever you have actually measured in the field, which could be things like the seroprevalence of animals at a particular time of year or in a particular stock class, the rate of disease spread in the herd during an epidemic (slope of the epidemic curve), annual mortality due to disease, and so on.
Let’s use BVD as a simple example and assume that we have uncertainty around both the β describing within-herd transmission from persistently infected animals (βPI) and the β describing within-herd transmission from transiently infected animals (βTI). We have field data on the seroprevalence in heifers at mating in an endemically infected herd.
We would first come up with a range of possible values these βs could take on usually either from a best guess based on other models in the literature or from manually playing with the model to get a general sense of what is reasonable. At the start of each simulation, we would then randomly select a value for βPI from the βPI distribution and a value for βTI from the βTI distribution. The starting conditions for the simulation would match the herd(s) we got the field data from as closely as possible. We would then run the simulation model and “measure” the seroprevalence of BVD in heifers at mating after the disease has reached equilibrium. We repeat this process several thousand times to generate a dataframe that looks something like this:
| Simulation | βPI | βTI | Seroprevalence |
|---|---|---|---|
| 1 | 0.003 | 0.0001 | 76% |
| 2 | 0.02 | 0.0002 | 80% |
| 3 | 0.0056 | 0.001 | 56% |
We then manually set a threshold for what values of seroprevalence generated by the model are close enough to the observed value (i.e. if the observed seroprevalence in the field was 66%, we might be willing to accept values from our model ranging from 64% to 68%). If we tracked other model outcome measures as well, we can also set threshold criteria around those to even further narrow the list of acceptable parameter combinations. We can then plot out the frequency distributions of βPI and βTI and hope it produces something like the following plot:
We can generate the most likely estimate plus confidence interval from this, which will become the value used in the final model. If the distributions don’t look like that, then either (a) there is something wrong with your model structure and/or parameters or (b) the thing you were using as a benchmark was not a good enough indicator. Either way, back to the drawing board! Note – there are much more efficient computer algorithms out there for “exploring the parameter space”. Basically, if your model accuracy gets progressively worse as you move in a certain direction away from the starting value, the computer won’t go any further in that direction and instead explore another range of parameter values.
Along a similar vein, model validation simply means comparing the outputs from a model whether that’s an epidemic curve or another measure like seroprevalence or longevity and then, usually through manual visual inspection, identifying whether the model produces a similar enough value to data observed in the real world. You’ll often hear the term calibration used as well, which describes the process of tweaking the model parameters and structure until the outputs match what was observed in the real world.
As a researcher, it’s your job to convince end-users that your model is credible. This includes providing evidence that:
The following is a good open-access review article that discusses model validation in greater detail:
Sometimes despite our best efforts to get real data to back up our choice for model structure or values for the parameters, we still end up having to make a best guesstimate, which can decrease confidence in our ability to make inferences or decisions from the results. In these instances, we will often do what is called a sensitivity analysis. This involves coming up with a range of possible values that our uncertain parameter(s) could take on in the real world. We then re-run the model across these different parameter values to see how much impact it has on the model outcomes or if it would change the decision we would make based on the model outcomes.
Ideally, what we would like to see is that even if we change the parameter value quite substantially, it makes no difference to the final model results. This increases our confidence that our model is robust and it actually doesn’t if there is some uncertainty in that parameter. The worst-case scenario is that the model results fluctuate quite substantially. This tells us that we either need to get better research data to generate an accurate estimate for the parameter or alternatively, that we should target whatever biological process that parameter represents to have a big impact on disease dynamics.
The Discussion section of your manuscript should at the very least contain a paragraph for each key finding that you presented in your Results section. No matter how exciting the results may be, it is always important to interpret the findings with some degree of caution since every model is forced to make simplifying assumptions about the real world and we as modellers rarely do any follow-up studies post-publication to determine how well the models perform when put to use. Some common points to discuss around limitations include:
5. Evaluating Disease Interventions
Advancing Animal Care through Shared Learning