[Biomod-commits] Predictive accuracy in Ensemble.Forecasting and Models functions

[Wilfried Thuiller]

Dear Raquel, 

Sorry for the very late answer.

I really need to rewrite completely the Ensemble Forecasting function which is not that clear. 

> As for question 5, as far as I could gather, the consensus_spp_proj.name
> gives predictions on the total data (as well as on each repetition), whereas
> Total_consensus_proj.name gives predictions based on all repetitions (if
> final.model.out=F) or on all the repetitions and total data (if
> final.model.out=T).  

Right. 

> However, the consensus_spp_proj.name_Bin output doesn't
> seem to give binary predictions for the ensemble median (the manual says
> that a threshold cannot be calculated), but the
> Total_consensus_proj.name_Bin output does - on which thresholds are they
> based? 

Indeed, this is not clear. For this Total_consensus_proj.name_Bin, the average threshold from the selected models is used. I just changed by the median which is more logical. 
In order words, the same threshold was used for "mean" and "median" until now. 

I'll also modify the function to make sure consensus_spp_proj.name_Bin could also give a binary projection for the median by taking the median threshold. 

Cheers,
Wilfried




> 
> Thanks once again,
> Raquel 
> 
> 
> -----Original Message-----
> From: biomod-commits-bounces at lists.r-forge.r-project.org
> [mailto:biomod-commits-bounces at lists.r-forge.r-project.org] On Behalf Of
> biomod-commits-request at lists.r-forge.r-project.org
> Sent: 16 December 2010 12:01 PM
> To: biomod-commits at lists.r-forge.r-project.org
> Subject: Biomod-commits Digest, Vol 20, Issue 4
> 
> Send Biomod-commits mailing list submissions to
> 	biomod-commits at lists.r-forge.r-project.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits
> 
> or, via email, send a message with subject or body 'help' to
> 	biomod-commits-request at lists.r-forge.r-project.org
> 
> You can reach the person managing the list at
> 	biomod-commits-owner at lists.r-forge.r-project.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Biomod-commits digest..."
> 
> 
> Today's Topics:
> 
>   1. Predictive accuracy in Ensemble.Forecasting and	Models
>      functions (Raquel A. Garcia)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Wed, 15 Dec 2010 15:54:19 +0100
> From: "Raquel A. Garcia" <raquel.garcia at mncn.csic.es>
> Subject: [Biomod-commits] Predictive accuracy in Ensemble.Forecasting
> 	and	Models functions
> To: biomod-commits at lists.r-forge.r-project.org
> Message-ID: <20101215155419.42243vxobjhtsk9n at webmail.csic.es>
> Content-Type: text/plain; charset="utf-8"
> 
> 
> 
> Dear BIOMODers 
> 
> I wonder if you can help with a few questions about the predictive accuracy
> of models. 
> 
> 1.For the consensus predictions from Ensemble.Forecasting(), the Manual says
> that the test is performed on the calibration data. So, to confirm, the
> first column of $test.results reports the ROC scores using all the data, and
> the following columns the ROC scores using the calibration data for each
> repetition?  (there was a post a few months ago about the possibility of
> these tests being done on testing data instead, but I am assuming that the
> current version still uses calibration data) 
> 
> 2. The help for the Ensemble.Forecasting() function says that the test
> scores ? are obtained by applying the same ensemble computation on the
> current predictions as on the future forecasts, and compared with the data
> input for that species (using the Roc evaluation method)?. So, again to
> confirm, are they calculated by running new ROC evaluations on the consensus
> projections?  
> 
> 3. For the Models() function, the evaluation.results output for each
> species, as far as I understand, gives in the last columns of $Spp_full the
> sensitivity and specificity calculated using all the data (the full model).
> Is it fair then to expect that calculating the Sens-Spec from the observed
> data and the binary predictions using the total data should lead to similar
> results as the Sens-Spec from the Models? (for presence-only data species I
> would use the pseudo-absence data instead of the observed data).  When I try
> these calculations for the individual model projections, I get similar
> values of specificity but lower values of sensitivity;  and when I try it
> for the mean, weighted mean and median consensus projections I get
> unrealistically low values of sensitivity. 
> 
> And two last ones still about the Ensemble.Forecasting() function: 
> 
> 4. The Total_consensus_proj.name output generates a ?general ensemble
> forecast across all the runs? for each consensus method. Whether this
> ensemble takes into account the final models as well as the repetitions
> depends on the final.model.out argument, I think ? what is the default for
> this argument? 
> 
> 5. In the consensus_spp_proj.name outputs, the second dimension includes the
> full model and the repetitions. Does this full model refer to ensembles
> built with the total data only or with total data plus the repetitions?  If
> the latter, is there no output with the consensus predictions based on the
> total data only? 
> 
> Hope this is not too long or unclear - many thanks  
> 
> Raquel  
> 
> Raquel A. Garcia
> Biodiversity and Global Change Lab
> www.biochange-lab.eu/
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> <http://lists.r-forge.r-project.org/pipermail/biomod-commits/attachments/201
> 01215/0efa1b00/attachment.html>
> 
> ------------------------------
> 
> _______________________________________________
> Biomod-commits mailing list
> Biomod-commits at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits
> 
> 
> End of Biomod-commits Digest, Vol 20, Issue 4
> *********************************************
> 
> _______________________________________________
> Biomod-commits mailing list
> Biomod-commits at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits

--------------------------
Dr. Wilfried Thuiller
Laboratoire d'Ecologie Alpine, UMR CNRS 5553
Université Joseph Fourier
BP53, 38041 Grenoble cedex 9, France
tel: +33 (0)4 76 51 44 97
fax: +33 (0)4 76 51 42 79

Email: wilfried.thuiller at ujf-grenoble.fr
Home page: http://www.will.chez-alice.fr
Website: http://www-leca.ujf-grenoble.fr/equipes/tde.htm

FP6 European MACIS project: http://www.macis-project.net
FP6 European EcoChange project: http://www.ecochange-project.eu





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.r-forge.r-project.org/pipermail/biomod-commits/attachments/20110103/f532af61/attachment.htm>