[Basta-users] BaSTA model and covariate selection: DIC, K-L, and/of confidence intervals

[maren]

Hi Diederik,

Overall your approach is correct. You choose the model form using DIC 
and then you check whether your covariates are relevant looking either 
at the Kullback-Leibler distances or the 95% confidence intervals. 
However, I would start right away with a model including all three 
covariates. For this model you test the different functional forms using 
DIC. In the model fitting your data best according to the DIC, you then 
look at the results for each covariate in detail.

Best wishes,
Maren


Am 02.10.2015 um 14:53 schrieb Diederik Strubbe:
> Dear BaSTA people,
>
> Thanks to your help we are making progress concerning our analysis of
> post-fledging survival of a tropical passerine. We have now identified a
> ‘best model’, using MultiBaSTA.R and based on the DIC values. The best
> model is a Weibull-Makeham, characterized by a DIC value of 351.6 (our
> 'base-model').
>
> We are now exploring the influence of covariates, but stumble upon
> something we cannot immediately understand. When we perform univariate
> tests (i.e. base model + one covariate), we get the follow results:
>
> Covariate 1: developmental age:
> DIC: 336.7
> Estimate: -0.117
> StdErr:    0.0918
> Lower95%CI: -0.295
> Upper95%CI: 0.0628
>
> Covariate 2: body condition:
> DIC: 337.2
> Estimate: 0.0076
> StdErr:    0.128
> Lower95%CI: -0.246
> Upper95%CI: 0.250
>
> Covariate 3: group size:
> DIC: 362.8
> Estimate: -0.211
> StdErr:    0.084
> Lower95%CI: -0.394
> Upper95%CI: -0.063
>
>
> According to the DIC criterion, Covariate1 and Covariate2 results in a
> better model fit (DIC values about 15 points lower than the
> Weibull-Makeham model without covariates). Covariate3 results in a worse
> fit (DIC values about 10 points higher).
>
> However, when looking at the estimates and standard errors (and
> confidence intervals), Covariate3 seems to be best supported: estimate >
> standard error, 0 not included in confidence interval. In contrast,
> estimate2 has a low (ie good) DIC value, but its estimate is much
> smaller than the standard deviation, and the confidence intervals surely
> includes 0. This is counterintuitive.
>
> After reading some recent papers using BaSTA, I think I come to the
> following conclusion about model and variable selection
>
> -    DIC is used to select between different survival models (e.g.
> through MultiBaSTA). Once the ‘best’ survival model is selected,
> covariate selection proceeds through:
> -    Categorical variables: Kullback-Leibler distances (using 0.65 as
> approximat threshold)
> -    Continuous variables: assess mean and standard deviation, and
> whether 0 is included in the 95%confidence interval.
>
> Following this logical, in the example above, I’d conclude that survival
> for our tropical birds is influenced by group size, that there is a weak
> trend for developmental age and no relationship with body condition.
>
> I’d appreciate any thoughts or comments about this interpretation of
> BaSTA results!
> Many thanks in advance,
>
> Diederik
>
>
> PS: I should note we found that DIC values can differ somewhat between
> model run (ie each time an identical model was run, DIC values are not
> identical). I notice that such differences are to be expected
> (Spiegelhalter et al 2002, p607). Therefore, all values reported above
> are averaged obtained from 100 model runs.
>