<div dir="ltr"><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Caitlin Collins</b> <span dir="ltr"><<a href="mailto:caitiecollins@gmail.com">caitiecollins@gmail.com</a>></span><br>Date: Thu, Oct 16, 2014 at 1:27 PM<br>Subject: Re: Question about how to interpret Cross validation in my analysis. Thanks!<br>To: Angela Merino <<a href="mailto:Angela.Merino@cawthron.org.nz">Angela.Merino@cawthron.org.nz</a>><br>Cc: "Collins, Caitlin" <<a href="mailto:caitlin.collins12@imperial.ac.uk">caitlin.collins12@imperial.ac.uk</a>>, "Jombart, Thibaut" <<a href="mailto:t.jombart@imperial.ac.uk">t.jombart@imperial.ac.uk</a>><br><br><br><div dir="ltr"><p class="MsoNormal">Hi Angela, <br>
<br>
Well, I have two pieces of good news for you, and one piece of mediocre news.<br>
<br>
First, there’s nothing to worry about with respect to the “NULL” that you are
seeing. It just gets printed when xval.plot=TRUE as an artefact of one of the lines
of the printing function. It has no meaning, and certainly does not imply that
your model is not valid. (Given the stress that I now realise this glaring “NULL”
may cause, I’ve changed the way the plots print now, so in the next release of
adegenet this won’t happen.)<br>
<br>
Second, you are absolutely correct in your interpretation of the results of xvalDapc
(which are stored in whatever object you assigned the results to, in your case,
“xval”). </p><p class="MsoNormal"><br></p>
<p class="MsoNormal">This brings me to the mediocre news: given that your
interpretation is correct, it seems that the best model you can achieve with
DAPC, where n.pca=25, is only able to predict the group membership of validation
set individuals in 63% of the cases, with a 32% root mean squared error.
Arguably, this is not great. Your final comment on the matter, though, is quite
insightful. The fact that you can achieve the same modest level of success with
20-80 PCs indicates that the optimisation procedure has not been particularly
successful. Ideally, one would like to see an arch, with a maximum success
point somewhere in the middle. In your case, there is a bit of an arch, but it
isn’t particularly striking. </p><p class="MsoNormal"><br></p>
<p class="MsoNormal">The only thing I might add to your interpretation of this
result is that it’s not so much that the model is poor because a similar level
of success can be achieved with variable numbers of PCs. If mean success was
virtually constant, but varying around 90%, the interpretation would not be
that the model is poor, but rather that most levels of PC retention can compose
a model that effectively discriminates between groups. <br>
<br>
I hope this has helped answer some of your questions. If you have any more,
please feel free to ask. <br>
<br>
Best, <br>
Caitlin. </p>
<p class="MsoNormal"> </p></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 13, 2014 at 11:48 PM, Angela Merino <span dir="ltr"><<a href="mailto:Angela.Merino@cawthron.org.nz" target="_blank">Angela.Merino@cawthron.org.nz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-NZ" link="blue" vlink="purple">
<div>
<p class="MsoNormal">Hi Caitlin Collins and Thibaut Jombart,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">My name is Angela Parody-Merino and I am a PhD student at Massey University (New Zealand). I am studying the population genetic structure in a migratory bird (the New Zealand Godwit) with 23 microsatellites. Anyway, maybe this is a very
simple question but I really want to understand and be sure about the meaning and interpretation of the output when doing cross-validation. I have been some days looking in the internet and reading explanations etc…without being able to really understand what’s
going on with my analysis. Could you help me please? <span style="font-family:Wingdings">
J</span><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">This is the script of the analysis:<u></u><u></u></p>
<p class="MsoNormal">> x <- ELpop<u></u><u></u></p>
<p class="MsoNormal">> mat <- as.matrix(na.replace(x, method="mean"))<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Replaced 371 missing values <u></u><u></u></p>
<p class="MsoNormal">> grp <- pop(x)<u></u><u></u></p>
<p class="MsoNormal">> xval <- xvalDapc(mat, grp, n.pca.max = 40, training.set = 0.9,<u></u><u></u></p>
<p class="MsoNormal">+ result = "groupMean", center = TRUE, scale = FALSE,<u></u><u></u></p>
<p class="MsoNormal">+ n.pca = NULL, n.rep = 500, xval.plot = TRUE)<u></u><u></u></p>
<p class="MsoNormal"><span style="color:red">NULL </span><b>>>> What does it mean this NULL? Does it mean that the model is not valid?</b><u></u><u></u></p>
<p class="MsoNormal"><u></u><img width="821" height="647" src="cid:image002.jpg@01CFE7A4.CCC02130" align="left" hspace="12"><u></u><span style="color:red"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="color:#558ed5">$`Median and Confidence Interval for Random Chance`<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5"> 2.5% 50% 97.5%
<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5">0.4294840 0.4928747 0.5962807
<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5">$`Mean Successful Assignment by Number of PCs of PCA`<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5"> 5 10 15 20 25 30 35 40
<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5">0.5871429 0.6000000 0.5819048 0.6014286 0.6952381 0.6747619 0.6333333 0.6109524
<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5">$`Number of PCs Achieving Highest Mean Success`<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5">[1] "25"<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5">$`Root Mean Squared Error by Number of PCs of PCA`<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5"> 5 10 15 20 25 30 35 40
<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5">0.4301795 0.4141872 0.4389381 0.4131429 0.3241735 0.3531491 0.3885084 0.4145894
<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5">$`Number of PCs Achieving Lowest MSE`<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:#558ed5">[1] "25"<u></u><u></u></span></b></p>
<p class="MsoNormal"><b><span style="color:red"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b>From the screenshot and the output results of the cross validation (in blue), I would say that my model (retaining 25PCs) can predict with a mean of 63% but it is not such a good model because most of the models that can be obtained
by retaining 20, 40, 60, 80 PCs are quite the same successful. Is it my interpretation correct?<u></u><u></u></b></p>
<p class="MsoNormal"><b><u></u> <u></u></b></p>
<p class="MsoNormal"><b><u></u> <u></u></b></p>
<p class="MsoNormal"><b><u></u> <u></u></b></p>
<p class="MsoNormal">Thanks in advance,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Kind regards,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">‘Angela Parody-Merino<u></u><u></u></p>
</div>
<hr>
<strong><font color="#ff0000">Attention: </font></strong><br>
This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies
of it and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Cawthron reserves the right to monitor all e-mail communications through its networks.
Any opinions expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorised to make that statement.
<p></p>
<p>This e-mail message has been scanned and cleared by <font color="#400080"><strong>MailMarshal
</strong></font></p>
<hr>
</div>
</blockquote></div><br></div>
</div></div></div><br></div>