<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 70.85pt 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US">I’ve discovered a 32bit limit on cases (even on a 64bit system). This is due to the vector size limit in R (3.2.3, 64bit, Windows x64), which is 2^31-1.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">> .Machine$integer.max<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[1] 2147483647<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">> 2^31-1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[1] 2147483647<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><br>
> sqrt(2^31-1)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[1] 46340.95<br>
<br>
Regardless of full.matrix=true/false (because vector size doesn’t change), seqdist() stops abruptly whenever there are more than 46341 cases. 46341 works fine, but 46342 does not. You can try this yourself (but if you change the size to anything less, you need
a lot of RAM. 46341 eats about 30 gbs of RAM):<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">----------<br>
library(TraMineR)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">id <- seq(from=1, to=46342, by=1)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">set.seed(234324)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">time1 <- sample(seq(from=1, to=3, by=1), size=46342, replace=TRUE)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">time2 <- sample(seq(from=1, to=3, by=1), size=46342, replace=TRUE)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">time3 <- sample(seq(from=1, to=3, by=1), size=46342, replace=TRUE)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">testdata <- data.frame(id, time1, time2, time3)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">testseq <- seqdef(testdata, 2:4)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">testdist <- seqdist(testseq, method="OM", indel=1, sm="TRATE", full.matrix=FALSE)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">---------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><br>
This is important, because adding more RAM won’t help, and neither won’t renting a super computer.<br>
<br>
One might ask if a smaller sample would work, but I want to use all the cases I have (a birth cohort of 60,000) to get more reliable results later on (narrower confidence intervals). I can at least create clusters from two smaller samples and combine visually
similar clusters from the two datas. <br>
<br>
Do you think we could get around the 2^31-1 limit? There has been a int64 package, which doesn’t seem to be maintained anymore. Any other ideas? Input from the developers? I’m not a developer myself, so I can’t do much.<br>
<br>
I haven’t found many similar issues, but some have been solved with wcAggregateCases, which has happened to lower the case amount to less than 2^31-1: http://stackoverflow.com/questions/15929936/problem-with-big-data-during-computation-of-sequence-distances-using-tramine<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><br>
Pasi Haapakorva<o:p></o:p></span></p>
</div>
</body>
</html>