[Biomod-commits] Predictive accuracy in Ensemble.Forecasting and Models functions

[Raquel A. Garcia]

Hi again,

Following my previous post - question 3 regarding sens and spec has been
solved! Please ignore that one.

As for question 5, as far as I could gather, the consensus_spp_proj.name
gives predictions on the total data (as well as on each repetition), whereas
Total_consensus_proj.name gives predictions based on all repetitions (if
final.model.out=F) or on all the repetitions and total data (if
final.model.out=T).  However, the consensus_spp_proj.name_Bin output doesn't
seem to give binary predictions for the ensemble median (the manual says
that a threshold cannot be calculated), but the
Total_consensus_proj.name_Bin output does - on which thresholds are they
based? 

Thanks once again,
Raquel 


-----Original Message-----
From: biomod-commits-bounces at lists.r-forge.r-project.org
[mailto:biomod-commits-bounces at lists.r-forge.r-project.org] On Behalf Of
biomod-commits-request at lists.r-forge.r-project.org
Sent: 16 December 2010 12:01 PM
To: biomod-commits at lists.r-forge.r-project.org
Subject: Biomod-commits Digest, Vol 20, Issue 4

Send Biomod-commits mailing list submissions to
	biomod-commits at lists.r-forge.r-project.org

To subscribe or unsubscribe via the World Wide Web, visit
	
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits

or, via email, send a message with subject or body 'help' to
	biomod-commits-request at lists.r-forge.r-project.org

You can reach the person managing the list at
	biomod-commits-owner at lists.r-forge.r-project.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Biomod-commits digest..."


Today's Topics:

   1. Predictive accuracy in Ensemble.Forecasting and	Models
      functions (Raquel A. Garcia)


----------------------------------------------------------------------

Message: 1
Date: Wed, 15 Dec 2010 15:54:19 +0100
From: "Raquel A. Garcia" <raquel.garcia at mncn.csic.es>
Subject: [Biomod-commits] Predictive accuracy in Ensemble.Forecasting
	and	Models functions
To: biomod-commits at lists.r-forge.r-project.org
Message-ID: <20101215155419.42243vxobjhtsk9n at webmail.csic.es>
Content-Type: text/plain; charset="utf-8"



Dear BIOMODers 

I wonder if you can help with a few questions about the predictive accuracy
of models. 

1.For the consensus predictions from Ensemble.Forecasting(), the Manual says
that the test is performed on the calibration data. So, to confirm, the
first column of $test.results reports the ROC scores using all the data, and
the following columns the ROC scores using the calibration data for each
repetition?  (there was a post a few months ago about the possibility of
these tests being done on testing data instead, but I am assuming that the
current version still uses calibration data) 

2. The help for the Ensemble.Forecasting() function says that the test
scores ? are obtained by applying the same ensemble computation on the
current predictions as on the future forecasts, and compared with the data
input for that species (using the Roc evaluation method)?. So, again to
confirm, are they calculated by running new ROC evaluations on the consensus
projections?  

3. For the Models() function, the evaluation.results output for each
species, as far as I understand, gives in the last columns of $Spp_full the
sensitivity and specificity calculated using all the data (the full model).
Is it fair then to expect that calculating the Sens-Spec from the observed
data and the binary predictions using the total data should lead to similar
results as the Sens-Spec from the Models? (for presence-only data species I
would use the pseudo-absence data instead of the observed data).  When I try
these calculations for the individual model projections, I get similar
values of specificity but lower values of sensitivity;  and when I try it
for the mean, weighted mean and median consensus projections I get
unrealistically low values of sensitivity. 

And two last ones still about the Ensemble.Forecasting() function: 

4. The Total_consensus_proj.name output generates a ?general ensemble
forecast across all the runs? for each consensus method. Whether this
ensemble takes into account the final models as well as the repetitions
depends on the final.model.out argument, I think ? what is the default for
this argument? 

5. In the consensus_spp_proj.name outputs, the second dimension includes the
full model and the repetitions. Does this full model refer to ensembles
built with the total data only or with total data plus the repetitions?  If
the latter, is there no output with the consensus predictions based on the
total data only? 

Hope this is not too long or unclear - many thanks  

Raquel  

Raquel A. Garcia
Biodiversity and Global Change Lab
www.biochange-lab.eu/
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.r-forge.r-project.org/pipermail/biomod-commits/attachments/201
01215/0efa1b00/attachment.html>

------------------------------

_______________________________________________
Biomod-commits mailing list
Biomod-commits at lists.r-forge.r-project.org
https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits


End of Biomod-commits Digest, Vol 20, Issue 4
*********************************************