[Biomod-commits] Model Discrepancies

[Wilfried Thuiller]

Dear Chris,
This is difficult to answer that questions without knowing what you've 
really done. Model parameterizations have a strong influence on model's 
outputs.
You talk about discrepancies. I do not see any difference for Species A 
but they are indeed differences for species B, especially for GLM and 
GBM. Do not beleive that differences of about 0.1 or 0.2 is enormous, 
this is nothing. AUC is an index, it is not a statistic. It is also very 
sensitive to the prevalence of the species.

Few things to remember:

1- Models are sensitive to the initial conditions and parameterisation.
=> did you use the same parameterisation than in BIOMOD? for instance, 
did you use the library GAM or MGCV? number of degree of freedom? Which 
optimisation strategy did you sue for GBM? Cross-validation or 
out-of-bag selection?
=> Did you run the models on 100% of your initial data or did you first 
calibrate on a random sample and evaluate on the remaining part? did you 
keep the prevalence fixed?
=> For GLM for instance: did you use polynomial, quadratic or linear 
terms? which level? stepwise regression, backward or forward?
=> Randomforest, GBM, neural networks have some buid-in stochasticity. 
They could give different outputs when run several times.

There are so many ways to parameterise those models that this is very 
difficult without having the data or the code you used. In any way, I am 
not surprised.

In anyway, this is an interesting email. It shows that although we have 
been mostly focusing on inter-model variability, there is also a 
substantial intra-model variability due to sample size, optimisation 
algorithm, and stochasticity.
I know Jane Elith (Australia) started to work on bit on this issue but 
this is relatively well know actually. Take SAS, SPSS and R (gam, mgcv) 
and run a GAM or neural-network on the same data, you'll get different 
results.

Hope it helps,
Wilfried


Christopher A Walter a écrit :
>
> Wilfried
>
> We have been testing the BIOMOD models against the stand-alone 
> versions in R (functions glm, gam, gbm, and randomForest) and while we 
> understand that there are many different options for running them, we 
> cannot seem to make them match even when trying to equalize the defaults.
>
>  Below are tables of AUC and Somer's rank correlation for two species, 
> ran with BIOMOD and the stand-alone models. We are trying to justify 
> using BIOMOD, but we cannot explain why it creates such a different 
> output. Any thoughts on why this is happening?
>
>
>
>
>
>
>
>
> All the best
>
> Chris
>
> Christopher A. Walter
> Biological Science Technician
> USGS Leetown Science Center
> 11649 Leetown Road
> Kearneysville, WV 25430
> Phone: 304 724-4479
>
> "Its easy not to think when you're not told that much."
>    -Justin Pierre
> ------------------------------------------------------------------------
>
> _______________________________________________
> Biomod-commits mailing list
> Biomod-commits at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits
>   

-- 
--
Wilfried Thuiller
Laboratoire d'Ecologie Alpine, UMR-CNRS 5553
Université J. Fourier
BP 53, 38041 Grenoble Cedex 9, France
Tel: +33 (0)4 76 63 54 53
Fax: +33 (0)4 76 51 42 79

Email: wilfried.thuiller at ujf-grenoble.fr
Home page: http://www.will.chez-alice.fr
Website: http://www-leca.ujf-grenoble.fr/equipes/tde.htm

FP6 European MACIS project: http://www.macis-project.net
FP6 European EcoChange project: http://www.ecochange-project.eu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.r-forge.r-project.org/pipermail/biomod-commits/attachments/20090731/f2ebf1a2/attachment-0001.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 27762 bytes
Desc: not available
Url : http://lists.r-forge.r-project.org/pipermail/biomod-commits/attachments/20090731/f2ebf1a2/attachment-0001.gif