<div>
<blockquote type="cite"><p dir="ltr">Now the issue you ran into is that you didn't realize that you were using non-standard naming (or even wanted to, but we can't guess what you want :)). </p></blockquote><div><p dir="ltr">I've to disagree here. This is one of the things `data.table` is *extremely* good at. The error/warning messages are precise and under most circumstances provides the solution (as to what you ought to do) by spotting the mistake exactly.</p><p dir="ltr"></p><blockquote type="cite"><p dir="ltr">Once you understand that there is nothing wrong with duplicate names, it should be clear that the appropriate warning spot is when you use them potentially incorrectly, and not when you set them.</p></blockquote><p></p><p dir="ltr">This, I believe is also not entirely true, at least in this scenario. For example, an error happens when assigning a duplicate column using `:=`, for example.</p></div>
</div>
<div><div>DT <- data.table(x=1:5, y=6:10)</div><div><div>> DT[, c("y", "y") := 1L]</div><div>Error in `[.data.table`(DT, , `:=`(c("y", "y"), 1L)) : </div><div> Can't assign to the same column twice in the same query (duplicates detected).</div></div><div><br></div><div>So, it's only natural to expect a warning/error in other cases as well. In general, prevention is better - it's nicer to catch it earlier, spit a warning/error rather than letting it on to only catch later.</div><div><br></div><div>Overall, I agree keeping duplicate names may help some users. But then, the potential side-effects should be marked with warnings/errors distinctly, in all cases (and preferably documented). Ex: grouping/aggregating is once such scenario (Ricardo's bug report) where we can not possibly know which column to use.. </div><div><br></div><div>Arun</div><div><br></div></div>
<p style="color: #A0A0A8;">On Saturday, November 2, 2013 at 4:30 PM, Eduard Antonyan wrote:</p>
<blockquote type="cite" style="border-left-style:solid;border-width:1px;margin-left:0px;padding-left:10px;">
<span><div><div><p dir="ltr">Thanks Alexandre. I added (a non-committal) FR about this - <a href="https://r-forge.r-project.org/tracker/index.php?func=detail&aid=5037&group_id=240&atid=978">https://r-forge.r-project.org/tracker/index.php?func=detail&aid=5037&group_id=240&atid=978</a>, which will likely go in the direction this thread goes.</p>
<p dir="ltr">To address your points: </p>
<p dir="ltr"> 1. If user decides to have column with duplicate names, yes, their job will become harder, but that's a user decision and everyone else who doesn't use duplicate names does not lose flexibility and doesn't need to use column numbers or whatnot. <br>
2. I agree that this should be documented better and appropriate warnings should be added.</p>
<p dir="ltr">One of the cool things about data.table that's very different from data.frame is that you can have arbitrary column names. Whether they include spaces, crazy symbols or are duplicate - it'll all be valid. This is very useful for reading and writing/presenting arbitrary data.</p>
<p dir="ltr">This does mean though that if (and *only* if) you choose to use non standard names you'll need to do more work.</p>
<p dir="ltr">Now the issue you ran into is that you didn't realize that you were using non-standard naming (or even wanted to, but we can't guess what you want :)). And a warning in the right place can help you out and also let non-standard users proceed. </p>
<p dir="ltr">Once you understand that there is nothing wrong with duplicate names, it should be clear that the appropriate warning spot is when you use them potentially incorrectly, and not when you set them.</p>
<p dir="ltr">For reference there are a *lot* of different ways to get duplicate names, to name a few besides setnames and creating one straight up - cbinding similarly named data.tables, merging, having default named columns and grouping (e.g. dt[, sum(smth), by = V1]), freading, etc.<br>
</p>
<div style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<div style="margin:0px">My 2 cents here.</div><div style="margin:0px"><br></div><div style="margin:0px">There are several reasons why I don’t think, IMHO, allowing multiple columns with the same name is a good idea:</div>
<div style="margin:0px"> </div><div style="margin:0px">- It will force the code to use column numbers to access all the data in a predictable fashion (since depending on your code you might now know which of the two columns with the same name will be the first), so we’ll lose all the delicious syntactic sugar painstakingly added to data.table.</div>
<div style="margin:0px"><br></div><div style="margin:0px">- For people learning data.table and having data.frame or even the concept of a relational table as a reference, this is a definite WTF and will cause confusion and complicate troubleshooting. I speak from experience on this matter. :)</div>
<div style="margin:0px"><br></div><div style="margin:0px">Even though there might be some situations where this might be a plus, I imagine they are few and far between and could be worked around. I could be wrong, it’s been know to happen :) - but I have never seen and can’t even imagine a situation where multiple columns with the same name would be essential. So in the balance I consider keeping this behavior as a bad trade-off for most users.</div>
<div style="margin:0px"><br></div><div style="margin:0px">Having said that, this is a design decision and it's up to the data.table demigods to decide. :)</div><div style="margin:0px"><br></div><div style="margin:0px">
BTW, is there any part of the data.table documentation that covers this? If you choose to maintain this property, I strongly suggest it be documented somewhere that most beginners would read.</div><div style="margin:0px">
<br></div><div style="margin:0px">In my personal example, I ran into this problem after a rather long troubleshooting of a very esoteric problem that was happening in my code. I was renaming a column to a name that already existed, and this broke things in a completely different part of my code. If ‘setnames()’ had at least warned me that a duplicate column name was created, I would have been able to detect the source cause much faster.</div>
<div style="margin:0px"><br></div></div> <div><span style="font-family:helvetica,arial;font-size:13px"></span><div style="font-family:Helvetica;line-height:normal">-- </div><div style="font-family:Helvetica;line-height:normal">
<span style="font-family:arial;font-size:small">Alexandre Sieira</span><br style="font-family:arial;font-size:small"><span style="font-family:arial;font-size:small">CISA, CISSP, ISO 27001 Lead Auditor</span><br style="font-family:arial;font-size:small">
<br style="font-family:arial;font-size:small"><span style="font-family:arial;font-size:small">"The truth is rarely pure and never simple."</span><br style="font-family:arial;font-size:small"><span style="font-family:arial;font-size:small">Oscar Wilde, The Importance of Being Earnest, 1895, Act I</span></div>
</div> <br><p style="color:#a0a0a8">On 1 de novembro de 2013 at 21:10:45, Arunkumar Srinivasan (<a href="mailto://aragorn168b@gmail.com" target="_blank">aragorn168b@gmail.com</a>) wrote:</p><blockquote type="cite"><div><span><div>
<div>
<div>Hm, I've not encountered that use myself, can't comment there.
Probably then it should be allowed everywhere except where deciding
which column could be an issue? Ex:
subsetting/aggregating/grouping/by-without-by etc.. should result
in error (if one has the time, one could do this by checking if the
duplicate column is in use actually or not and then issue an
error/warning). </div>
<div><br></div>
<div>At the moment, I'm not convinced that it's worth that much
trouble to help data presentation.</div>
<div>
<div><br></div>
<div>Arun</div>
<div><br></div>
</div>
<p style="color:#a0a0a8">On Saturday, November 2, 2013 at 12:05
AM, Eduard Antonyan wrote:</p><blockquote type="cite"><div>
<div>
<div>
<div dir="ltr"><span>Because it's very useful for e.g. data
presentation purposes.</span></div>
<div><span><br>
<br></span>
<div><span>On Fri, Nov 1, 2013 at 6:02 PM, Arunkumar Srinivasan
<span dir="ltr"><<a href="mailto:aragorn168b@gmail.com" target="_blank">aragorn168b@gmail.com</a>></span> wrote:<br></span><blockquote type="cite"><div>
<div>
<div>Yes, it chooses the first. But we won't be able to perform any
operation as intended. So why allow duplicate names (ex: in
`setnames` as Alexandre asks)?</div>
<div>
<div><br></div>
<div>Arun</div>
<div><br></div>
</div>
<div>
<div>
<p style="color:#a0a0a8">On Friday, November 1, 2013 at 11:57 PM,
Eduard Antonyan wrote:</p><blockquote type="cite"><div>
<div>
<div>
<div>
<div dir="ltr"><span>I think currently it chooses the first "x",
but it's definitely a good idea to add a warning
there.</span></div>
<div><span><br>
<br></span>
<div><span>On Fri, Nov 1, 2013 at 5:51 PM, Arunkumar Srinivasan
<span dir="ltr"><<a href="mailto:aragorn168b@gmail.com" target="_blank">aragorn168b@gmail.com</a>></span> wrote:<br></span><blockquote type="cite"><div>
<div>
<div>Ricardo added a bug report here on this topic: <a href="https://r-forge.r-project.org/tracker/index.php?func=detail&aid=5008&group_id=240&atid=975" target="_blank">https://r-forge.r-project.org/tracker/index.php?func=detail&aid=5008&group_id=240&atid=975</a></div>
<div>
<div>But I don't think having duplicate names is an
easy-to-implement concept. For ex:</div>
<div><br></div>
<div>dt <- data.table(x=1:3, x=4:6, y=c(1,1,2))</div>
<div>dt[, print(.SD), by=y]</div>
<div>
<div> x</div>
<div>1: 1</div>
<div>2: 2</div>
<div> x</div>
<div>1: 3</div>
</div>
<div><br></div>
<div>.SD loses the second "x". Also, some other questions become
difficult to handle. Ex: </div>
<div><br></div>
<div>
<div>dt <- data.table(x=c(1,1,2,2), y=c(1,2,3,4),
x=c(2,2,1,1))</div>
<div>dt[, list(x=x/x[1], y=y), by=x]</div>
</div>
<div><br></div>
<div>Which "x" should be choose for which operation?</div>
<div><br></div>
<div>Arun</div>
<div><br></div>
</div>
<div>
<div>
<p style="color:#a0a0a8">On Friday, November 1, 2013 at 10:59 PM,
Eduard Antonyan wrote:</p><blockquote type="cite"><div>
<div>
<div>
<div>
<div dir="ltr"><span>Having duplicate names is allowed and not that
unusual in data.table framework, so there is no need to signal
anything here.</span>
<div><span><br></span></div>
<div><span>A different question is whether there should be a
warning here:</span></div>
<div><span><br></span></div>
<div><span> dt = data.table(a = 1, a = 2)</span></div>
<div><span> dt[, a]</span></div>
<div><span><br></span></div>
<div><span>and I think that'd be a pretty good FR to
have.</span></div>
</div>
<div><span><br>
<br></span>
<div><span>On Fri, Nov 1, 2013 at 4:49 PM, Alexandre Sieira
<span dir="ltr"><<a href="mailto:alexandre.sieira@gmail.com" target="_blank">alexandre.sieira@gmail.com</a>></span>
wrote:<br></span><blockquote type="cite"><div>
<div>
<div style="word-wrap:break-word">
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
I found this behavior during a debugging session: </div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<div style="margin:0px">> d = data.table(a=1, b=2, c=3)</div>
<div style="margin:0px">> setnames(d, "a", "b")</div>
<div style="margin:0px">> d</div>
<div style="margin:0px"> b b c</div>
<div style="margin:0px">1: 1 2 3</div>
<div style="margin:0px"><br></div>
<div style="margin:0px">Shouldn’t setnames() check if the new
column names already exist before renaming, and signal an error or
at least a warning if they do?</div>
</div>
<span><font color="#888888"><br></font></span>
<div>
<div style="font-family:Helvetica;line-height:normal">
<span>-- </span></div>
<div style="font-family:Helvetica;line-height:normal">
<span><span style="font-family:arial;font-size:small">Alexandre
Sieira</span><br style="font-family:arial;font-size:small">
<span style="font-family:arial;font-size:small">CISA, CISSP, ISO
27001 Lead Auditor</span><br style="font-family:arial;font-size:small">
<br style="font-family:arial;font-size:small">
<span style="font-family:arial;font-size:small">"The truth is
rarely pure and never simple."</span><br style="font-family:arial;font-size:small">
<span style="font-family:arial;font-size:small">Oscar Wilde, The
Importance of Being Earnest, 1895, Act I</span></span></div>
</div>
</div>
<br>
_______________________________________________<br>
datatable-help mailing list<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org" target="_blank">datatable-help@lists.r-forge.r-project.org</a><br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br>
</div>
</div></blockquote></div>
<br></div>
</div>
<div>
<div>_______________________________________________</div>
<div>datatable-help mailing list</div>
<div><a href="mailto:datatable-help@lists.r-forge.r-project.org" target="_blank">datatable-help@lists.r-forge.r-project.org</a></div>
<div><a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a></div>
</div>
</div>
</div>
</div></blockquote><div><br></div>
</div>
</div>
</div>
</div></blockquote></div>
<br></div>
</div>
</div>
</div>
</div></blockquote><div><br></div>
</div>
</div>
</div>
</div></blockquote></div>
<br></div>
</div>
</div>
</div></blockquote><div><br></div>
</div></div></span></div></blockquote></div></div>
</div></div></span>
</blockquote>
<div>
<br>
</div>