<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>Hi,</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>I'll chime in despite being fairly ignorant of a lot of the use cases for phylobase.</div><div><br></div><div>My experience dealing with files from users is that you have to be prepared for anything occurring in a taxon name. So I'd say that you have to have a system in which the name can be any character string, not a system in which the taxon name can be a variable name in R. So I would recommend getting the charcter string from NCL, and then using whatever R-specific character escaping conventions are needed to generate the equivalent string literal in R (you'll have to use \ before any quote or \ characters).</div><div><br></div><div><br></div><div><br></div><div>Phylobase *could* strip problematic symbols out of the names easily enough but:</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>1. it is confusing to user's because the name of the taxon changes (usually only slightly, but it still changes), and</div><div><span class="Apple-tab-span" style="white-space:pre"> </span>2. then you have to deal with name clashes (if a file has 'a b' and 'ab' as taxa names, then stripping spaces will cause a clash).</div><div><br></div><div>NEXUS has syntactic rules for tokenizing. They usually don't change the internal representation much, but NCL uses the syntactic rules to build a character string in memory. In the following table I'll use double-quotes to delimit the character string that NCL would store in memory (the quotes are not part of the name).</div><div><br></div><div>Syntax Internal representation</div><div>t1 "t1"</div><div>'t 2' "t 2"</div><div>t_2 "t 2"</div><div>t.3 "t.3"</div><div>'t.3' "t.3"</div><div>t-4 not a valid name this is 3 tokens "t", "-" and "4"</div><div>'t-4' "t-4"</div><div>'t''5 x' "t'5 x"</div><div>'t_6' "t_6" </div><div><br></div><div>Basically, the syntactic rules just let you group tokens with token-breaking symbols (white space and punctuation). The exceptions are the case t_2 and 't''5 x' above. Both of those case involve substitution (in the second case the internal pair of single-quotes is collapsed to single quote in the internal representation).</div><div><br></div><div><br></div><div>What really confuses users is the fact that in some formats a_b might refer to the name "a_b", but in NEXUS it is "a b" So the same syntax has different interpretation in different formatting rules. NCL uses the NEXUS tokenizing for NEXUS files and newick tree files (NCL supports fasta, phylip, and relaxed phylip formats, but does not impose NEXUS quoting rules on those formats). So if you use NCL to read a FASTA and NEXUS file both of which use the name a_b (without any quotes) then in the taxa gleaned from the FASTA file, the internal name will be "a_b", but the same name will have the internal representation of "a b" when read from NEXUS. That invariably trips people up, but I don't think that there is anyway to avoid it because that behavior is mandated by both standards.</div><div><br></div><div>I think that if phylobase just include a brief discussion of this issue in a README or other documentation, then that is about as good as you can do.</div></div><div><br></div><div><br></div><div>all the best,</div><div>Mark</div><div><br></div><div><div><div>On Apr 29, 2010, at 10:51 AM, Orme, David wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Sorry - I take it back about the 'not problematic in a character string' - the quotes and the backslash are of course fairly problematic.<div><br></div><div>Cheers</div><div>David</div><div><br><div><div>On 29 Apr 2010, at 16:43, Orme, David wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi,<div><br></div><div>I know this isn't the bible - but the PAUP manual specifies the following:</div><div><br></div><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Palatino; ">"Identifiers" are simply names given to taxa, characters, and other PAUP input elements such as character-sets, taxon-sets, and exclusion-sets. They may include any combination of upper- and lower-case alphabetic characters, digits, and punctuation. If the identifier contains any of the following characters:</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Palatino; ">( ) [ ] { } / \ , ; : = * ' "` + - < ></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Palatino; ">or a blank, the entire identifier must be enclosed in single quotes.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Palatino; "><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; font: normal normal normal 12px/normal Palatino; "><span class="Apple-style-span" style="font-family: Helvetica; "><div>They're going to be rare but they will happen. Any of those are problematic in a valid R name - although not in a character string. The taxon identifiers could come into R as character vectors just as they appear in the Nexus file (possibly stripping the enclosing single quotes). The next question is then whether we need them to be valid R names rather than character strings - in which case make.names() could be invoked.</div><div><br></div><div>Cheers,</div><div>David</div><div><br></div><div><br></div></span></div><div><div>On 28 Apr 2010, at 22:30, François Michonneau wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><br>Hi all,<br><br> Sorry if this is a dumb question, but why do we need to remove spaces<br>and underscore from the species names when building the data frame? The<br>only character that I can think of that could be an issue is ", and I<br>don't think that it's allowed by software using NEXUS/used.<br><br> In other words, do we really need to use RemoveUnderscoresAndSpaces in<br>NCLInterface.cpp?<br><br> Thanks,<br> -- François <br><br>On Mon, 2010-04-26 at 12:11 +0100, Orme, David wrote:<br><blockquote type="cite">I'd guess we want the names to be syntactically valid R names - and ideally that would be through running make.names() across them. The problem is then that the NCLInterface can easily pass the raw PAUP identifiers for the data (which we can then make.names()) but that the tree input is currently via a text string. Again, probably easy enough to have the raw PAUP names in the string but these would be horrible to extract with regex. Is there any way that NCLInterface can pass the tree using numeric symbols and then pass a translate block as a vector? Then make.names() could be run easily on both the data names and the tree names...<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Cheers,<br></blockquote><blockquote type="cite">David<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On 23 Apr 2010, at 14:30, François Michonneau wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Hi,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Ouch! We need to fix this.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> There might be some hope if we use Rcpp to build the data frame<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">instead of building and parsing a string.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Let me talk to Dirk about it and see what we can do.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> Cheers,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"> -- François<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">On Fri, 2010-04-23 at 13:59 +0100, Orme, David wrote:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Hi all,<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">From an e-mail on 03/03/10:<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Mark then Peter<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">The main potential problems that I see with the ways that phylobase is using NCL now are:<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><span class="Apple-tab-span" style="white-space:pre"> </span>1. in NCLInterface.cpp there are lots of call to RemoveUnderscoresAndSpaces to get rid of spaces and _ in names. That makes names easier to deal with, but at some point will bite you (somebody will have dataset with a taxon labelled "AB" and another with "A B", after transformation there will be a name clash).<br></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">I agree that this is something to address. Not only might there be clashes but changing names, will be annoying to users. Brian or Derrick could answer better, but I assume this is because some of the code used to parse the tree string can't handle the underscores and spaces.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Has just bitten me! There is a deeper problem here in that readNexus uses the NCLInterface code to get the data frame as parsable R code - with stripped spaces and underscores - but the tree block is passed over as a block of raw text from the file. These names _aren't_ then stripped of underscores and spaces by read.nexustreestring() and so the name checking throws an error. <br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Obviously there is an ongoing deeper discussion about how to handle passing the tree from NCL and how to handle the dismayingly wide range of official valid PAUP identifiers using regex but currently we've got a simpler problem of different handling. Underscores in names are very commonly used to avoid the quoting problem with spaces so I think this current problem will come up a lot. <br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Cheers,<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">David<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Phylobase-devl mailing list<br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="mailto:Phylobase-devl@lists.r-forge.r-project.org">Phylobase-devl@lists.r-forge.r-project.org</a><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/phylobase-devl">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/phylobase-devl</a><br></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">Phylobase-devl mailing list<br></blockquote><blockquote type="cite"><a href="mailto:Phylobase-devl@lists.r-forge.r-project.org">Phylobase-devl@lists.r-forge.r-project.org</a><br></blockquote><blockquote type="cite"><a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/phylobase-devl">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/phylobase-devl</a><br></blockquote></div></blockquote></div><br></div></div><span><ATT00002..txt></span></blockquote></div><br></div></div>_______________________________________________<br>Phylobase-devl mailing list<br><a href="mailto:Phylobase-devl@lists.r-forge.r-project.org">Phylobase-devl@lists.r-forge.r-project.org</a><br><a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/phylobase-devl">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/phylobase-devl</a><br></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Monaco; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Monaco; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Arial; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Courier; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Andale Mono'; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Andale Mono'; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: 'Andale Mono'; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: 'Andale Mono'; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: 'Andale Mono'; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><span class="Apple-style-span" style="border-collapse: separate; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; color: rgb(0, 0, 0); font-family: 'Andale Mono'; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; -webkit-text-decorations-in-effect: none; text-indent: 0px; -webkit-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><div><div><span class="Apple-style-span" style="font-family: Arial; "><div class="AppleMailSignature" id="BD214290-7324-48EC-A16C-4BCD48002BDD"><span class="Apple-style-span" style="font-family: Courier; "><span class="Apple-style-span" style="font-family: 'Andale Mono'; "><div><div><span class="Apple-style-span" style="font-family: Courier; "><div class="AppleMailSignature" id="BD214290-7324-48EC-A16C-4BCD48002BDD"><span class="Apple-style-span" style="font-family: 'Andale Mono'; "><div><div><font class="Apple-style-span" face="Arial">Mark Holder</font></div><div><font class="Apple-style-span" face="Arial"><br class="webkit-block-placeholder"></font></div><div><font class="Apple-style-span" face="Arial"><a href="mailto:mtholder@ku.edu">mtholder@ku.edu</a></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="Arial"><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font style="font: normal normal normal 12px/normal Helvetica; "><font class="Apple-style-span" face="Arial" size="4"><span class="Apple-style-span" style="font-size: 14px; "><a href="http://phylo.bio.ku.edu/mark-holder">http://phylo.bio.ku.edu/mark-holder</a></span></font></font></div></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="Arial"><br class="webkit-block-placeholder"></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="Arial">==============================================</font></div><div><font class="Apple-style-span" face="Arial">Department of Ecology and Evolutionary Biology</font></div><div><font class="Apple-style-span" face="Arial">University of Kansas</font></div><div><font class="Apple-style-span" face="Arial">6031 Haworth Hall</font></div><div><font class="Apple-style-span" face="Arial">1200 Sunnyside Avenue</font></div><div><font class="Apple-style-span" face="Arial">Lawrence, Kansas 66045</font></div><div><font class="Apple-style-span" face="Arial"><br class="khtml-block-placeholder"></font></div><div><font class="Apple-style-span" face="Arial">lab phone: 785.864.5789</font></div><div><br></div><div><font class="Apple-style-span" face="Arial">fax (shared): 785.864.5860</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="Arial">==============================================</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" face="Arial"><br></font></div><font class="Apple-style-span" face="Arial"><br></font></div></span></div></span><font class="Apple-style-span" face="Arial"><br></font></div></div></span></span></div><font class="Apple-style-span" face="Monaco"></font></span><br></div></div></span></span></span></span></span></span></span></span></span><br class="Apple-interchange-newline"></div></span></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</div>
<br></div></body></html>