<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Dear Popko,</div><div><br></div><div>The predictive accuracy estimated during the Calibration phase is NOT done onto the calibration data (80% of the data in your case) but on the remaining part (20% in your case). The Cross-validation is thus the mean of the evaluations onto the 20% (in your case a mean on 3 repetitions x 2 pseudo-absence runs). This is thus not really surprising to see that the predictive accuracy estimated onto a completely independant dataset (indepdt.data) is higher than the one estimated onto 20% of the initial datasets). </div><div>Therefore, you have showed what you called "lower" or "higher". TSS/Kappa or AUC are indices not statistics. They should be taken as a statistical tests. </div><div><br></div><div>Finally, if I understood well, you calibrated your models using 70 different variables? What is the purpose of using so many variables? I suppose that many are correlated. GLM and especially stepwise regressions are not very good to deal with large number of correlated variables. GBM or randomForest are better techniques, robust to multi-colinearity and data hungry methods. Just an opinion... </div><div><br></div><div>Hope it helps,</div><div>Wilfried</div><div><br></div><div><br></div><br><div><div>Le 26 janv. 2010 à 14:51, Popko a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Dear colleagues,<br><br>Looking at my Evaluation.results I noticed that predictive accuracy of <br>Calibration (called Cross.validation in output table) is lower than the <br>predictive accuracy using Evaluation (called "indepdt.data" in output <br>table). This is true for all 63 species I've analyzed, and independent of <br>method (ROC, kappa, TSS).<br>Q: How is this possible?<br><br>This is the model I ran:<br>Models(GLM=T, TypeGLM="quad", Test="AIC", GBM=F, No.trees=2000, GAM=F, <br>Spline=3, CTA=F, CV.tree=50, ANN=F, CV.ann=2, SRE=F, Perc025=T, Perc05=F, <br>MDA=F, MARS=F, RF=F, NbRunEval=3, DataSplit=80, Yweights=NULL, Roc=T, <br>Optimized.Threshold.Roc=T, Kappa=T, TSS=T, KeepPredIndependent=T, <br>VarImport=5, NbRepPA=2, strategy="circles", coor=CoorXY, distance=2, <br>nb.absences=1000)<br><br>Data consisted of presence/absence data with ca. 2500 cases per species. <br>Some 70 environmental variables were entered in the model.<br><br>Eagerly awaiting your responses,<br><br>Popko Wiersma<br>SOVON Dutch Centre for Field Ornithology<br><br><br>_______________________________________________<br>Biomod-commits mailing list<br><a href="mailto:Biomod-commits@lists.r-forge.r-project.org">Biomod-commits@lists.r-forge.r-project.org</a><br>https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits<br></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><span class="Apple-style-span" style="font-size: 12px; "><div>--------------------------</div><div>Dr. Wilfried Thuiller</div><div>Laboratoire d'Ecologie Alpine, UMR CNRS 5553</div><div>Université Joseph Fourier</div><div>BP53, 38041 Grenoble cedex 9, France</div><div>tel: +33 (0)4 76 63 54 53</div><div>fax: +33 (0)4 76 51 42 79</div><div><br></div><div>Email: <a href="mailto:wilfried.thuiller@ujf-grenoble.fr">wilfried.thuiller@ujf-grenoble.fr</a><br>Home page: <a href="http://www.will.chez-alice.fr/">http://www.will.chez-alice.fr</a><br>Website: <a href="http://www-leca.ujf-grenoble.fr/equipes/tde.htm">http://www-leca.ujf-grenoble.fr/equipes/tde.htm</a><br><br>FP6 European MACIS project: <a href="http://www.macis-project.net/">http://www.macis-project.net</a><br>FP6 European EcoChange project: <a href="http://www.ecochange-project.eu/">http://www.ecochange-project.eu</a></div><div><br></div></span></div></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br></body></html>