[Biomod-commits] Error in ensemble by algorithm
Damien Georges
damien.georges2 at gmail.com
Wed Mar 13 10:29:02 CET 2013
Dear Josep,
Did you check your model projections? It's seems that one models predict
presences (or absences) everywhere.. That's probably why you have the
error in auc calculation reported below (question 2 ).
Please compute the projections over formal models with initial data and
look at binaries produced.. something like :
myBiomodProjection <- BIOMOD_Projection(modeling.output = myBiomodModelOut,
new.env =
getModelsInputData(myBiomodModelOut,'expl.var'),
proj.name = 'test',
selected.models = 'all',
binary.meth = 'ROC',
build.clamping.mask = FALSE)
binProjFile <- list.files(path= file.path(myBiomodProjection at sp.name,
paste("proj_",myBiomodProjection at proj.names,sep="")),pattern="_ROCbin.RData",full.names=T,recursive=T,include.dirs=F)
myBinProj <- get(load(binProjFile))
apply(myBinProj, c(2,3,4),summary)
If you encounter any probleme, you can send me your data and your
scripts (in private mail) and I will try to see what's going wrong.
Best,
Damien G.
On 12/03/2013 20:16, Josep M Serra diaz wrote:
> Wilfried and colleagues,
>
>
> This error in ensembling by algorithm comes up again, even though I am not
> selecting for a high quality threshold.
> I am using last version of R and biomod2
>
> question 1: Why does this warning appear appear?
> *
> "Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions "*
>
> question 2: the real error that breaks the ensembling is this one. Why is
> that?
> *Error in roc.default(Obs, Fit, percent = T) : No control observation.*
>
>
> Thanks a lot for your time and effort,
>
> Pep
>
>
> Find history hereunder:
>
>
> myBiomodEM.algo <- BIOMOD_EnsembleModeling (
> + em.by="algo" ,
> + modeling.output =
> myBiomodModelOut,
> + chosen.models = 'all',
> + eval.metric = 'TSS',
> + eval.metric.quality.threshold =
> c(0.0), # we want them all
> + prob.mean=T,
> + prob.cv = T,
> + prob.ci = T,
> + prob.ci.alpha = 0.05,
> + prob.median = T,
> + committee.averaging = T,
> + prob.mean.weight = F,
> + prob.mean.weight.decay =
> 'proportional'
> + )
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Build
> Ensemble Models
> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>
> ! all models available will be included in ensemble.modeling
> > Evaluation & Weighting methods summary :
> TSS over 0
>
>
> > GLM_AllRun ensemble modeling
> > TSS
> > models kept : Quercushumilis_AllData_RUN1_GLM,
> Quercushumilis_AllData_RUN2_GLM, Quercushumilis_AllData_RUN3_GLM
> ! Models projections for whole zonation required...
> > Projecting Quercushumilis_AllData_RUN1_GLM ...
> > Projecting Quercushumilis_AllData_RUN2_GLM ...
> > Projecting Quercushumilis_AllData_RUN3_GLM ...
>
> > Mean of probabilities...
> > Coef of variation of probabilities...
> > Median of ptobabilities...
> > Confidence Interval...
> > 2.5 %
> > 97.5 %
> > Comittee averaging...
>
> > GBM_AllRun ensemble modeling
> > TSS
> > models kept : Quercushumilis_AllData_RUN1_GBM,
> Quercushumilis_AllData_RUN2_GBM, Quercushumilis_AllData_RUN3_GBM
> ! Models projections for whole zonation required...
> > Projecting Quercushumilis_AllData_RUN1_GBM ...
> > Projecting Quercushumilis_AllData_RUN2_GBM ...
> > Projecting Quercushumilis_AllData_RUN3_GBM ...
>
> > Mean of probabilities...
> > Coef of variation of probabilities...
> > Median of ptobabilities...
> > Confidence Interval...
> > 2.5 %
> > 97.5 %
> > Comittee averaging...
>
> > GAM_AllRun ensemble modeling
> > TSS
> > models kept : Quercushumilis_AllData_RUN1_GAM,
> Quercushumilis_AllData_RUN2_GAM, Quercushumilis_AllData_RUN3_GAM
> ! Models projections for whole zonation required...
> > Projecting Quercushumilis_AllData_RUN1_GAM ...
> > Projecting Quercushumilis_AllData_RUN2_GAM ...
> > Projecting Quercushumilis_AllData_RUN3_GAM ...
>
> > Mean of probabilities...
> > Coef of variation of probabilities...
> > Median of ptobabilities...
> > Confidence Interval...
> > 2.5 %
> > 97.5 %
> > Comittee averaging...Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR",
> "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
>
>
> > CTA_AllRun ensemble modeling
> > TSS
> > models kept : Quercushumilis_AllData_RUN1_CTA,
> Quercushumilis_AllData_RUN2_CTA, Quercushumilis_AllData_RUN3_CTA
> ! Models projections for whole zonation required...
> > Projecting Quercushumilis_AllData_RUN1_CTA ...
> > Projecting Quercushumilis_AllData_RUN2_CTA ...
> > Projecting Quercushumilis_AllData_RUN3_CTA ...
>
> > Mean of probabilities...
> > Coef of variation of probabilities...
> > Median of ptobabilities...
> > Confidence Interval...
> > 2.5 %
> > 97.5 %
> > Comittee averaging...
>
> > ANN_AllRun ensemble modeling
> > TSS
> > models kept : Quercushumilis_AllData_RUN3_ANN
> ! Models projections for whole zonation required...
> > Projecting Quercushumilis_AllData_RUN3_ANN ...
>
> > Mean of probabilities...
> > Coef of variation of probabilities...
> > Median of ptobabilities...
> > Confidence Interval...
> > 2.5 %
> > 97.5 %
> > Comittee averaging...Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR",
> "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS", :
>
> Observed or fited data contains a unique value.. Be carefull with this
> models predictions
>
> Error in roc.default(Obs, Fit, percent = T) : No control observation.
> In addition: There were 50 or more warnings (use warnings() to see the
> first 50)
>
>
>
>
>
>
>> 2013/3/11 Wilfried Thuiller <wilfried.thuiller at ujf-grenoble.fr>
>>
>>> Dear Josep,
>>>
>>> Please make sure to use the latest version (2.1.13). From the what is
>>> pasted below, it seems that the minimum threshold to select the models for
>>> the ensemble forecast is too high and no models are selected.
>>> Try to put 0.4 for instance and run the script again.
>>>
>>> Best regards,
>>> Wilfried
>>>
>>>
>>>
>>> Le 10 mars 2013 à 20:20, Josep M Serra diaz a écrit :
>>>
>>> Dear BIOMODers,
>>>
>>> I found an error while trying to perform modeling ensemble by algorithm in
>>> order to produce an output for each statistical technique
>>>
>>>
>>> Any clue of what does this mean???
>>>
>>> The strane
>>>
>>>
>>> ########################
>>>
>>> #ensemble through algorithm
>>> myBiomodEM.algo <- BIOMOD_EnsembleModeling (
>>> em.by="algo" ,
>>> modeling.output = myBiomodModelOut,
>>> chosen.models = 'all',
>>> eval.metric = 'TSS',
>>> eval.metric.quality.threshold =
>>> c(0.6),
>>> prob.mean=T,
>>> prob.cv = T,
>>> prob.ci = T,
>>> prob.ci.alpha = 0.05,
>>> prob.median = T,
>>> committee.averaging = T,
>>> prob.mean.weight = F,
>>> prob.mean.weight.decay =
>>> 'proportional'
>>> )
>>>
>>>
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>>> Build Ensemble Models
>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>>>
>>> ! all models available will be included in ensemble.modeling
>>>
>>> Evaluation & Weighting methods summary :
>>>
>>> TSS over 0.6
>>>
>>>
>>> GLM_AllRun ensemble modeling
>>>
>>> TSS
>>>
>>> ! No models kept due to treshold filtering... Ensemble Modeling was
>>> skip!
>>>
>>> GBM_AllRun ensemble modeling
>>>
>>> TSS
>>>
>>> ! No models kept due to treshold filtering... Ensemble Modeling was
>>> skip!
>>>
>>> GAM_AllRun ensemble modeling
>>>
>>> TSS
>>>
>>> ! No models kept due to treshold filtering... Ensemble Modeling was
>>> skip!
>>>
>>> CTA_AllRun ensemble modeling
>>>
>>> TSS
>>>
>>> ! No models kept due to treshold filtering... Ensemble Modeling was
>>> skip!
>>>
>>> ANN_AllRun ensemble modeling
>>>
>>> TSS
>>>
>>> ! No models kept due to treshold filtering... Ensemble Modeling was
>>> skip!
>>>
>>> FDA_AllRun ensemble modeling
>>>
>>> TSS
>>>
>>> ! No models kept due to treshold filtering... Ensemble Modeling was
>>> skip!
>>>
>>> MARS_AllRun ensemble modeling
>>>
>>> TSS
>>>
>>> ! No models kept due to treshold filtering... Ensemble Modeling was
>>> skip!
>>>
>>> RF_AllRun ensemble modeling
>>>
>>> TSS
>>>
>>> models kept : Quercusilex_AllData_RUN2_RF
>>>
>>> ! Models projections for whole zonation required...
>>>
>>> Projecting Quercusilex_AllData_RUN2_RF ...
>>>
>>>
>>> Mean of probabilities...
>>>
>>> Coef of variation of probabilities...
>>>
>>> Median of ptobabilities...
>>>
>>> Confidence Interval...
>>>
>>> 2.5 %
>>>
>>> 97.5 %
>>>
>>> Comittee averaging...Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR",
>>>
>>> "SR", "ACCURACY", "BIAS", :
>>>
>>> Observed or fited data contains a unique value.. Be carefull with this
>>> models predictions
>>>
>>> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS",
>>> :
>>>
>>> Observed or fited data contains a unique value.. Be carefull with this
>>> models predictions
>>>
>>> Warning in FUN(c("KAPPA", "TSS", "ROC", "FAR", "SR", "ACCURACY", "BIAS",
>>> :
>>>
>>> Observed or fited data contains a unique value.. Be carefull with this
>>> models predictions
>>>
>>> *Error in roc.default(Obs, Fit, percent = T) : No control observation.*
>>>
>>> In addition: There were 50 or more warnings (use warnings() to see the
>>> first 50)
>>>
>>> warnings()
>>>
>>> Warning messages:
>>> 1: In (forecast_1 * observed_1) + (forecast_0 * observed_0) :
>>> NAs produced by integer overflow
>>> 2: In (forecast_1 * observed_1) + (forecast_0 * observed_0) :
>>> NAs produced by integer overflow
>>> 3: In (forecast_1 * observed_1) + (forecast_0 * observed_0) :
>>> NAs produced by integer overflow
>>> 4: In (forecast_1 * observed_1) + (forecast_0 * observed_0) :
>>> NAs produced by integer overflow
>>> 5: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 6: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 7: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 8: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 9: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 10: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 11: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 12: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 13: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 14: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 15: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 16: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 17: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 18: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 19: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 20: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 21: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 22: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 23: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 24: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 25: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 26: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 27: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 28: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 29: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 30: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 31: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 32: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 33: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 34: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 35: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 36: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 37: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 38: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 39: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 40: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 41: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 42: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 43: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 44: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 45: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 46: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 47: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 48: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 49: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> 50: In forecast_0 * observed_0 : NAs produced by integer overflow
>>> _______________________________________________
>>> Biomod-commits mailing list
>>> Biomod-commits at lists.r-forge.r-project.org
>>>
>>> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits
>>>
>>>
>>> --------------------------
>>> Dr. Wilfried Thuiller
>>> Laboratoire d'Ecologie Alpine, UMR CNRS 5553
>>> Université Joseph Fourier
>>> BP53, 38041 Grenoble cedex 9, France
>>> tel: +33 (0)4 76 51 44 97
>>> fax: +33 (0)4 76 51 42 79
>>>
>>> Email: wilfried.thuiller at ujf-grenoble.fr
>>> Personal website: http://www.will.chez-alice.fr
>>> Team website: http://www-leca.ujf-grenoble.fr/equipes/emabio.htm
>>>
>>> ERC Starting Grant TEEMBIO project:
>>> http://www.will.chez-alice.fr/Research.html
>>> FP6 European EcoChange project: http://www.ecochange-project.eu
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
> _______________________________________________
> Biomod-commits mailing list
> Biomod-commits at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/biomod-commits
More information about the Biomod-commits
mailing list