[Biomod-commits] Predictive accuracy in Ensemble.Forecasting and Models functions

[Raquel A. Garcia]

Dear BIOMODers 

I wonder if you can help with a few questions about the predictive accuracy of models. 

1.For the consensus predictions from Ensemble.Forecasting(), the Manual says that the test is performed on the calibration data. So, to confirm, the first column of $test.results reports the ROC scores using all the data, and the following columns the ROC scores using the calibration data for each repetition?  (there was a post a few months ago about the possibility of these tests being done on testing data instead, but I am assuming that the current version still uses calibration data) 

2. The help for the Ensemble.Forecasting() function says that the test scores “ are obtained by applying the same ensemble computation on the current predictions as on the future forecasts, and compared with the data input for that species (using the Roc evaluation method)”. So, again to confirm, are they calculated by running new ROC evaluations on the consensus projections?  

3. For the Models() function, the evaluation.results output for each species, as far as I understand, gives in the last columns of $Spp_full the sensitivity and specificity calculated using all the data (the full model). Is it fair then to expect that calculating the Sens-Spec from the observed data and the binary predictions using the total data should lead to similar results as the Sens-Spec from the Models? (for presence-only data species I would use the pseudo-absence data instead of the observed data).  When I try these calculations for the individual model projections, I get similar values of specificity but lower values of sensitivity;  and when I try it for the mean, weighted mean and median consensus projections I get unrealistically low values of sensitivity. 

And two last ones still about the Ensemble.Forecasting() function: 

4. The Total_consensus_proj.name output generates a “general ensemble forecast across all the runs” for each consensus method. Whether this ensemble takes into account the final models as well as the repetitions depends on the final.model.out argument, I think – what is the default for this argument? 

5. In the consensus_spp_proj.name outputs, the second dimension includes the full model and the repetitions. Does this full model refer to ensembles built with the total data only or with total data plus the repetitions?  If the latter, is there no output with the consensus predictions based on the total data only? 

Hope this is not too long or unclear - many thanks  

Raquel  

Raquel A. Garcia
Biodiversity and Global Change Lab
www.biochange-lab.eu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/biomod-commits/attachments/20101215/0efa1b00/attachment.htm>