<div dir="ltr">Sorry for the off topic question, but what is "merge.levels"? That looks potentially useful.<div><br></div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 14, 2015 at 1:45 PM, Gerald Jean <span dir="ltr"><<a href="mailto:gerald.jean@dgag.ca" target="_blank">gerald.jean@dgag.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="FR-CA" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hello,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">thanks Frank, you were right. I am converting roughly 2000 lines of code using data.frames to the data.table way, this one skipped me!!! By the
way, on this data set, 4750880 observations, the processing time went from 1hr.45 to 12.5 minutes. If we could parallelize this it would run under a minute, I have 24 processors on the server where that runs.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks again,<u></u><u></u></span></p><span class="">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Gérald<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<table border="0" cellpadding="0" width="640" style="width:480.0pt">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><img width="136" height="54" src="cid:image001.gif@01D08E54.6EF8F1B0"></span><span style="font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
</tr>
<tr>
<td width="300" valign="top" style="width:225.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black">Gerald Jean, M. Sc. en statistiques</span></b><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black"><br>
Conseiller senior en statistiques<br>
<br>
Actuariat corporatif,<br>
Modélisation et Recherche<br>
Assurance de dommages<br>
Mouvement Desjardins</span><span style="font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
</td>
<td width="170" valign="top" style="width:127.5pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black"><br>
Lévis (siège social)<br>
<br>
<a href="tel:418%20835-4900" value="+14188354900" target="_blank">418 835-4900</a>,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black">poste 5527639<br>
<a href="tel:1%20877%20835-4900" value="+18778354900" target="_blank">1 877 835-4900</a>, <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black">poste 5527639<br>
Télécopieur : <a href="tel:418%20835-6657" value="+14188356657" target="_blank">418 835-6657</a></span><span style="font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
</td>
<td width="170" valign="top" style="width:127.5pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black"><br>
<br>
<br>
<br>
</span><span style="font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<table border="0" cellpadding="0" width="640" style="width:480.0pt">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><span style="font-size:7.0pt;font-family:"Verdana","sans-serif";color:black">Faites bonne impression et imprimez seulement au besoin!<br>
<br>
</span><span style="font-size:7.0pt;font-family:"Verdana","sans-serif";color:dimgray">Ce courriel est confidentiel, peut être protégé par le secret professionnel et est adressé exclusivement au destinataire. Il est strictement interdit à toute autre personne
de diffuser, distribuer ou reproduire ce message. Si vous l'avez reçu par erreur, veuillez immédiatement le détruire et aviser l'expéditeur. Merci.</span><span style="font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
</span><p class="MsoNormal"><b><span lang="FR" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">De :</span></b><span lang="FR" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:by.hook.or@gmail.com" target="_blank">by.hook.or@gmail.com</a> [mailto:<a href="mailto:by.hook.or@gmail.com" target="_blank">by.hook.or@gmail.com</a>]
<b>De la part de</b> Frank Erickson<br>
<b>Envoyé :</b> 14 mai 2015 13:19<br>
<b>À :</b> Gerald Jean<br>
<b>Cc :</b> <a href="mailto:datatable-help@lists.r-forge.r-project.org" target="_blank">datatable-help@lists.r-forge.r-project.org</a><br>
<b>Objet :</b> Re: [datatable-help] Can you explain what is going on???<u></u><u></u></span></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi Gérald,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Your question is not really data.table specific, I think. Your <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New"">ttt[ttt == "0"] <- "O"</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">does not affect the result because you overwrite with<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier New"">ttt <- ifelse(...</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">immediately afterwards. Maybe you meant to have ttt on the right-hand side of the latter command, instead of membre.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">--Frank<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, May 14, 2015 at 10:04 AM, Gerald Jean <<a href="mailto:gerald.jean@dgag.ca" target="_blank">gerald.jean@dgag.ca</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New"">Hello,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">the following code is extracted from a function where roughly 150 variables of a large data set are transformed using data.table.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">The variable “membre” was coming out with one missing value, in trying to understand why, I extracted the code from the function,
added a few “cat” statements and ran it directly in the terminal.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">ttt.test.sima[, ":=" (membre = {##</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ cat(" Processing: membre", sep = "\n")</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ ttt <- membre</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ cat(paste(" Class ttt = ", class(ttt), sep = ""), sep = "\n")</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ cat(paste(" Length ttt = ", length(ttt), sep = ""), sep = "\n")</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ cat(paste(" sum(ttt == 0) = ", sum(ttt == "0"), sep = ""), sep = "\n")</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ ttt[ttt == "0"] <- "O" ## A few capital “O” are coded as zero “0”.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ cat(paste(" sum(ttt == 0) = ", sum(ttt == "0"), sep = ""), sep = "\n")</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ ttt <- ifelse(PROV != " QC", " OAO",</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ ifelse(membre == "", " Ma ", membre))</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ cat(paste(" sum(ttt == 0) = ", sum(ttt == "0"), sep = ""), sep = "\n")</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">+ merge.levels(factor(ttt, levels = c("O", "N", " Ma ", " OAO"),</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">+ labels = c(" Oui", " Non", " Ma ", " OAO")),</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">+ k = list(" Oui" = c(" Oui", " OAO")))})]</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Processing: membre</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Class ttt = character</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Length ttt = 4750880</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">sum(ttt == 0) = 2</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">sum(ttt == 0) = 0</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">sum(ttt == 0) = 1</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">I don’t understand why after the « ifelse» statement the temporary variable « ttt» is back with a single « 0 (zero)» in it,
resulting of course in the missing value of the factor created from it.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Thanks for your support,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Gérald</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<table border="0" cellpadding="0" width="640" style="width:480.0pt">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><img border="0" width="136" height="54" src="cid:image001.gif@01D08E54.6EF8F1B0"><u></u><u></u></p>
</td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
<td style="padding:.75pt .75pt .75pt .75pt"></td>
</tr>
<tr>
<td width="300" valign="top" style="width:225.0pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><b><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black">Gerald Jean, M. Sc. en statistiques</span></b><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black"><br>
Conseiller senior en statistiques<br>
<br>
Actuariat corporatif,<br>
Modélisation et Recherche<br>
Assurance de dommages<br>
Mouvement Desjardins</span><u></u><u></u></p>
</td>
<td width="170" valign="top" style="width:127.5pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black"><br>
Lévis (siège social)<br>
<br>
<a href="tel:418%20835-4900" target="_blank">418 835-4900</a>,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black">poste 5527639<br>
<a href="tel:1%20877%20835-4900" target="_blank">1 877 835-4900</a>, </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black">poste 5527639<br>
Télécopieur : <a href="tel:418%20835-6657" target="_blank">418 835-6657</a></span><u></u><u></u></p>
</td>
<td width="170" valign="top" style="width:127.5pt;padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:8.0pt;font-family:"Verdana","sans-serif";color:black"><br>
<br>
<br>
</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"> <u></u><u></u></p>
<table border="0" cellpadding="0" width="640" style="width:480.0pt">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal"><span style="font-size:7.0pt;font-family:"Verdana","sans-serif";color:black">Faites bonne impression et imprimez seulement au besoin!<br>
<br>
</span><span style="font-size:7.0pt;font-family:"Verdana","sans-serif";color:dimgray">Ce courriel est confidentiel, peut être protégé par le secret professionnel et est adressé exclusivement au destinataire. Il est strictement interdit à toute autre personne
de diffuser, distribuer ou reproduire ce message. Si vous l'avez reçu par erreur, veuillez immédiatement le détruire et aviser l'expéditeur. Merci.</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><br>
_______________________________________________<br>
datatable-help mailing list<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org" target="_blank">datatable-help@lists.r-forge.r-project.org</a><br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
<br>_______________________________________________<br>
datatable-help mailing list<br>
<a href="mailto:datatable-help@lists.r-forge.r-project.org">datatable-help@lists.r-forge.r-project.org</a><br>
<a href="https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help" target="_blank">https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help</a><br></blockquote></div><br></div></div></div>