<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>First of all you should read the BIOMOD manual. It is not entirely up-to-date but it should give you the first principles.</div><div><br></div><div>Just type the following in the R console</div><div><br></div><div>> Biomod.Manuals()</div><div><br></div><div>Secondly, BIOMOD is going to do the splitting procedure for you. You do not have to do it yourself. </div><div>When you have "NbRunEval=3", this means you repeat the splitting procedure three times. Every-time the data are split, the models are calibrated on 80% and evaluated on 20%. Then, when it is done three times, only the average evaluation is recorded (in the Cross-Validation column). </div><div><br></div><div>It is only when you have TRULY independent data that you should use the IndependantResponse call.</div><div><br></div><div>SO, what you should do: </div><div><br></div><div><blockquote type="cite"><div>Initial.State(Response = data_1spec[2], Explanatory = data_1spec[,5:lastcol2], sp.name=i)<br><br>Models(GLM=T, TypeGLM="quad", Test="AIC", GBM=F, No.trees=2000, GAM=F,<br>Spline=3, CTA=F, CV.tree=50, ANN=F, CV.ann=2, SRE=F, Perc025=T, Perc05=F,<br>MDA=F, MARS=F, RF=F, NbRunEval=3, DataSplit=80, Yweights=NULL, Roc=T,<br>Optimized.Threshold.Roc=T, Kappa=T, TSS=T, KeepPredIndependent=T,<br>VarImport=5, NbRepPA=2, strategy="circles", coor=CoorXY, distance=2,<br>nb.absences=1000)</div></blockquote></div><div><br></div><div>Best</div><div>Wilfried</div><div><br></div><div><br></div><div><br></div><div><br></div><br><div><div>Le 26 janv. 2010 à 15:57, Popko a écrit :</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>In reply to your (swift) response: I guess I'm still missing something, but <br>I did not have a 'completely independent dataset' as you assumed, but split <br>my dataset (80/20). I assumed that 20% of the data was used as if it were <br>independent data. This should then have resulted in on average equal results <br>in the first and second column, cross.validation and indepdt.data, resp. In <br>fact the validation results of the independent data are always higher (by 7% <br>on average).<br><br>FYI:<br>Initial.State(Response = data_1spec[2], Explanatory = <br>data_1spec[,5:lastcol2], IndependentResponse = data_1spec[2], <br>IndependentExplanatory = data_1spec[,5:lastcol2], sp.name=i)<br><br>Models(GLM=T, TypeGLM="quad", Test="AIC", GBM=F, No.trees=2000, GAM=F,<br>Spline=3, CTA=F, CV.tree=50, ANN=F, CV.ann=2, SRE=F, Perc025=T, Perc05=F,<br>MDA=F, MARS=F, RF=F, NbRunEval=3, DataSplit=80, Yweights=NULL, Roc=T,<br>Optimized.Threshold.Roc=T, Kappa=T, TSS=T, KeepPredIndependent=T,<br>VarImport=5, NbRepPA=2, strategy="circles", coor=CoorXY, distance=2,<br>nb.absences=1000)<br><br>Popko Wiersma<br>SOVON Dutch Centre for Field Ornithology<br><br>--------------------------------------------------<br>From: "Wilfried Thuiller" <<a href="mailto:wilfried.thuiller@ujf-grenoble.fr">wilfried.thuiller@ujf-grenoble.fr</a>><br>Sent: Tuesday, January 26, 2010 3:08 PM<br>To: "Popko" <<a href="mailto:popkowiersma@hotmail.com">popkowiersma@hotmail.com</a>><br>Cc: <<a href="mailto:biomod-commits@r-forge.wu-wien.ac.at">biomod-commits@r-forge.wu-wien.ac.at</a>><br>Subject: Re: [Biomod-commits] predictive accuracy issue<br><br><blockquote type="cite">Dear Popko,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The predictive accuracy estimated during the Calibration phase is NOT done <br></blockquote><blockquote type="cite">onto the calibration data (80% of the data in your case) but on the <br></blockquote><blockquote type="cite">remaining part (20% in your case). The Cross-validation is thus the mean <br></blockquote><blockquote type="cite">of the evaluations onto the 20% (in your case a mean on 3 repetitions x 2 <br></blockquote><blockquote type="cite">pseudo-absence runs). This is thus not really surprising to see that the <br></blockquote><blockquote type="cite">predictive accuracy estimated onto a completely independant dataset <br></blockquote><blockquote type="cite">(indepdt.data) is higher than the one estimated onto 20% of the initial <br></blockquote><blockquote type="cite">datasets).<br></blockquote><blockquote type="cite">Therefore, you have showed what you called "lower" or "higher". TSS/Kappa <br></blockquote><blockquote type="cite">or AUC are indices not statistics. They should be taken as a statistical <br></blockquote><blockquote type="cite">tests.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Finally, if I understood well, you calibrated your models using 70 <br></blockquote><blockquote type="cite">different variables? What is the purpose of using so many variables? I <br></blockquote><blockquote type="cite">suppose that many are correlated. GLM and especially stepwise regressions <br></blockquote><blockquote type="cite">are not very good to deal with large number of correlated variables. GBM <br></blockquote><blockquote type="cite">or randomForest are better techniques, robust to multi-colinearity and <br></blockquote><blockquote type="cite">data hungry methods. Just an opinion...<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Hope it helps,<br></blockquote><blockquote type="cite">Wilfried<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Le 26 janv. 2010 à 14:51, Popko a écrit :<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">Dear colleagues,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Looking at my Evaluation.results I noticed that predictive accuracy of<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Calibration (called Cross.validation in output table) is lower than the<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">predictive accuracy using Evaluation (called "indepdt.data" in output<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">table). This is true for all 63 species I've analyzed, and independent of<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">method (ROC, kappa, TSS).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Q: How is this possible?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">This is the model I ran:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Models(GLM=T, TypeGLM="quad", Test="AIC", GBM=F, No.trees=2000, GAM=F,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Spline=3, CTA=F, CV.tree=50, ANN=F, CV.ann=2, SRE=F, Perc025=T, Perc05=F,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">MDA=F, MARS=F, RF=F, NbRunEval=3, DataSplit=80, Yweights=NULL, Roc=T,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Optimized.Threshold.Roc=T, Kappa=T, TSS=T, KeepPredIndependent=T,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">VarImport=5, NbRepPA=2, strategy="circles", coor=CoorXY, distance=2,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">nb.absences=1000)<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Data consisted of presence/absence data with ca. 2500 cases per species.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Some 70 environmental variables were entered in the model.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Eagerly awaiting your responses,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Popko Wiersma<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">SOVON Dutch Centre for Field Ornithology<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Biomod-commits mailing list<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="mailto:Biomod-commits@lists.r-forge.r-project.org">Biomod-commits@lists.r-forge.r-project.org</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits</a><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--------------------------<br></blockquote><blockquote type="cite">Dr. Wilfried Thuiller<br></blockquote><blockquote type="cite">Laboratoire d'Ecologie Alpine, UMR CNRS 5553<br></blockquote><blockquote type="cite">Université Joseph Fourier<br></blockquote><blockquote type="cite">BP53, 38041 Grenoble cedex 9, France<br></blockquote><blockquote type="cite">tel: +33 (0)4 76 63 54 53<br></blockquote><blockquote type="cite">fax: +33 (0)4 76 51 42 79<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Email: <a href="mailto:wilfried.thuiller@ujf-grenoble.fr">wilfried.thuiller@ujf-grenoble.fr</a><br></blockquote><blockquote type="cite">Home page: <a href="http://www.will.chez-alice.fr">http://www.will.chez-alice.fr</a><br></blockquote><blockquote type="cite">Website: <a href="http://www-leca.ujf-grenoble.fr/equipes/tde.htm">http://www-leca.ujf-grenoble.fr/equipes/tde.htm</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">FP6 European MACIS project: <a href="http://www.macis-project.net">http://www.macis-project.net</a><br></blockquote><blockquote type="cite">FP6 European EcoChange project: <a href="http://www.ecochange-project.eu">http://www.ecochange-project.eu</a><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote>_______________________________________________<br>Biomod-commits mailing list<br><a href="mailto:Biomod-commits@lists.r-forge.r-project.org">Biomod-commits@lists.r-forge.r-project.org</a><br>https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits<br></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><span class="Apple-style-span" style="font-size: 12px; "><div>--------------------------</div><div>Dr. Wilfried Thuiller</div><div>Laboratoire d'Ecologie Alpine, UMR CNRS 5553</div><div>Université Joseph Fourier</div><div>BP53, 38041 Grenoble cedex 9, France</div><div>tel: +33 (0)4 76 63 54 53</div><div>fax: +33 (0)4 76 51 42 79</div><div><br></div><div>Email: <a href="mailto:wilfried.thuiller@ujf-grenoble.fr">wilfried.thuiller@ujf-grenoble.fr</a><br>Home page: <a href="http://www.will.chez-alice.fr/">http://www.will.chez-alice.fr</a><br>Website: <a href="http://www-leca.ujf-grenoble.fr/equipes/tde.htm">http://www-leca.ujf-grenoble.fr/equipes/tde.htm</a><br><br>FP6 European MACIS project: <a href="http://www.macis-project.net/">http://www.macis-project.net</a><br>FP6 European EcoChange project: <a href="http://www.ecochange-project.eu/">http://www.ecochange-project.eu</a></div><div><br></div></span></div></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br></body></html>