[Phylobase-devl] Rcpp and OS X compiliation

Mon Jan 18 01:54:56 CET 2010

Hi Mark, 

On 17 January 2010 at 08:50, Mark Holder wrote:
| Hi all,
| 	Things are clearing up for me, but I'm afraid that I have not been  
| terribly disciplined as I play around with potential fixes.  So I  
| don't have terminal output from each step.

Well then you will have to put up with my nagging. :)

| Based on using "otool -L" (to see which libraries a dylib or so is  
| referring to), I now think that this is what is happening:
| 
| I. The Rcpp package that you can install via install.packages contains  
| a reference to
| 	/Builds/Rdev-web/QA/Simon/packages/tiger-universal/Rlib/2.10/Rcpp/lib/ 
| i386/libRcpp.dylib
|   This looks like the source of the problem, as no user's machine will  
| have that path. It may be OK for the library just lists itself by name  
| rather than full path and that this may solve the problems (but I'm  
| not sure about that).

That may be something that we need to discuss with Simon.

| II. Compiling Rcpp  (at least from svn) puts the correct install  
| location for the library in its list of libraries ("/Users/mholder/ 
| Library/R/2.10/library/Rcpp/lib/i386/libRcpp.dylib" in my case).  I  
| think that it is INSTALL that does this (it is interesting really  
| because in my build dir ~/builds/rin/rcpp/pkg/inst/lib/i386 the  
| library has a reference to its install rather than build location).

Good -- that is what my last email referred to as /usr/bin/install_name_tool
knows about the 'target' directory based on R's knowledge of where its
packages go.  

So provided you build Rcpp think seem fine.

| III. A package that compiles against libRcpp seems to be picking up  
| the path to libRcpp.dylib from the path listed in the library itself  
| (rather than the path where the library is found during compilation).   

Can you explain this again? You confuse me here. Aren't those the same?

Packages compiling against Rcpp are typically using facilities provided by
Rcpp -- in particular Rcpp:::LdFlags() -- which will point to the installed
library (as in the previous point).

| I don't know if this is a standard aspect of dynamic linking/loading  
| or something that is a glitch in the R package installation.

Can you explain again what the glitch is?

| IV. When dynamically loading a package, the loader tries:
| 	A. "Special" locations (things listed in DYLD_LIBRARY_PATH or ldpaths)
| 	B. the full path listed in the calling library if the path is listed  
| as a full path.
| 	C. the full library search path which R augments with all of the  
| package directories.  So library(x) will load any dylibs in that  
| packages own lib directory because R tells the dynamic loader to look  
| there.

Ok. 

| So here is the behavior that I see:
| 1. the function symbols found in libRCpp are fine whether the library  
| is installed from source or not -- so if the loader finds the library  
| all will be well.
| 
| 2. If you installed Rcpp FROM SOURCE. and then build your Rcpp- 
| dependent packages all is well. (libRcpp.dylib will list the correct  
| path to itself, and so the calling lib will list the correct path to  
| libRcpp.dylib)
| 
| 3. If you installed Rcpp using install.packages and THEN build your  
| Rcpp-dependent packages:
| 	1. You get the infamous "unable to load shared library... Library not  
| loaded: /Builds/Rdev-web/QA/Simon/packages/tiger-universal/Rlib/2.10/ 
| Rcpp/lib/i386/libRcpp.dylib" error

Not good.

| 	2. If you add the correct path to libRcpp.dylib in some special way,  
| then point IV(A) above preempts step IV(B).  Everything is fine.

But that would require every user to dance around R.  We don't need this on
Linux and Windows, and I'd rather have something that 'just works' on OS X
too. 

(Sure. What matter, AFAICT, is how phylobase encodes references to Rcpp. As
we started with the right information we are good.)

| Solutions seem to be:
| 	1. (best) figure out how to get install.packages to modify  
| libRcpp.dylib so that it list its own location correctly,

Agreed. Now how do we do this?

| 	2. distribute the needed parts of libRcpp as a tool that installs to / 
| usr/local/lib rather than an R package,

We can't (easily) as R does not let a package install outside of its package
directories. 

| 	2. disable compiling against libRcpp package if it was downloaded  
| from the web (I don't know how to do that, mind you), or

Hard to impossible. 

| 	3. alter the default dyld behavior (through DYLD_LIBRARY_PATH or  
| copying libRcpp.dylib to /usr/lib) so that it does not Matter that the  
| path the libRcpp.dylib is incorrectly listed.

Maybe add this option:

      4. Stop using dynamic linking on OS X and link statically to libRcpp.a.
This one is already provided, and I can make it easier to get the proper link
instruction.  That would be more failsafe, but less elegant (as dynamic
linking is).

Cheers, Dirk

| all the best,
| Mark
| 
| 
| 	
| 	
| 
| 
| 
| On Jan 16, 2010, at 10:17 PM, Dirk Eddelbuettel wrote:
| 
| >
| > Hi Mark,
| >
| > On 16 January 2010 at 20:57, Mark Holder wrote:
| > | Hi again,
| > |
| > | On Jan 15, 2010, at 11:16 AM, Ben Bolker wrote:
| > |
| > | > Mark Holder wrote:
| > | >>
| > | >> I'm not sure how to fix it other than altering the user's env  
| > (which
| > | >> is obviously not ideal).
| > | >
| > | >  No, but it might be an interim hack: R has Sys.getenv() and
| > | > Sys.setenv() which might be usable ?  (although they might have  
| > to be
| > | > called too early in the package-loading process)
| > |
| > | I am not having any success getting the library to load by modifying
| > | DYLD_LIBRARY_PATH from within R.
| >
| > Seconded. I played with that a looong time ago and concluded that  
| > these
| > environment variables are 'sealed' once you are running your  
| > process. You
| > cannot modify your own process. Which is probably why the (shell  
| > script)
| > /usr/bin/R sets them up before calling the underlying binary.
| >
| > | I can augment the list of paths successfully (with
| > | Sys.setenv(DYLD_LIBRARY_PATH="/Us..."),
| > | but that doesn't seem to help the loader.  I suspect that you are
| > | correct that using setenv is simply too late.
| >
| > Agreed.
| >
| > | According to "Writing R extensions
| > | ${R_HOME}/etc/${R_ARCH}/ldpaths is sourced before the R process is
| > | launched.
| >
| > Exactly.
| >
| > | On my machine this is:
| > | 	/Library/Frameworks/R.framework/Resources/etc/i386/ldpaths
| > |
| > | This file adds ${R_HOME}/lib/${R_ARCH} to the dynamic load library
| > | path list (one an intel-mac this is /Library/Frameworks/R.framework/
| > | Resources/lib/i386). This directory contains libR.dylib,
| > | libRblas.dylib, and  libRlapack.dylib on my machine.
| > |
| > | I can get phylobase to run if I do any one of the  following four
| > | things:
| > | 	1. Add ~/Library/R/2.10/library/Rcpp/lib/i386 to the user's env
| > | before calling R
| > | 	2. Add ~/Library/R/2.10/library/Rcpp/lib/i386 to  
| > DYLD_LIBRARY_PATH in
| > | the /Library/Frameworks/R.framework/Resources/etc/i386/ldpaths file
| > | 	3. Copy ~/Library/R/2.10/library/Rcpp/lib/i386/libRcpp.dylib to  /
| > | Library/Frameworks/R.framework/Resources/lib/i386
| > | 	4. Copy ~/Library/R/2.10/library/Rcpp/lib/i386/libRcpp.dylib to / 
| > usr/
| > | local/lib
| >
| > Hm. None of 1 to 4 should be necessary if the 'rpath' option works  
| > correctly.
| > Does that exist / work on OS X?  Otherwise I have a problem with  
| > Rcpp that is
| > much larger than phylobase :-/ --- and I am surprised that has not  
| > come up
| > before.  As AFAIK the build bot on R-Forge doesn't use this either.
| >
| > I would still like to disentangle things, if possible.
| >
| > Could you have a look at the baby package 'earthmovdist' from R- 
| > Forge?  I
| > attach a tarball for your persusal.  It simple uses Rcpp to get to a  
| > simple
| > C++ class.  No external library, but of course a dependency on Rcpp's
| > library.   On my system this builds as
| >
| > edd at ron:~/svn/earthmovdist$ R CMD INSTALL earthmovdist_0.1.0.tar.gz
| > * installing to library ‘/usr/local/lib/R/site-library’
| > * installing *source* package ‘earthmovdist’ ...
| > ** libs
| > g++ -I/usr/share/R/include -I/usr/local/lib/R/site-library/Rcpp/ 
| > lib     -fpic  -g -O3 -Wall -pipe -c emdL1.cpp -o emdL1.o
| > g++ -I/usr/share/R/include -I/usr/local/lib/R/site-library/Rcpp/ 
| > lib     -fpic  -g -O3 -Wall -pipe -c emdL1_R.cpp -o emdL1_R.o
| > g++ -shared -o earthmovdist.so emdL1.o emdL1_R.o -L/usr/local/lib/R/ 
| > site-library/Rcpp/lib -lRcpp -Wl,-rpath,/usr/local/lib/R/site- 
| > library/Rcpp/lib -L/usr/lib/R/lib -lR
| > ** R
| > ** inst
| > ** preparing package for lazy loading
| > ** help
| > *** installing help indices
| > ** building package indices ...
| > * DONE (earthmovdist)
| > edd at ron:~/svn/earthmovdist$
| >
| > and I can then say
| >
| > edd at ron:~/svn/earthmovdist$ Rscript -e 'library(earthmovdist);  
| > example(emdL1)'
| > Loading required package: Rcpp
| > Loading required package: methods
| >
| > emdL1R>   set.seed(42)
| >
| > emdL1R>   x <- rnorm(100)
| >
| > emdL1R>   y <- x + rnorm(100)/10
| >
| > emdL1R>   emdL1(x, y, verbose=TRUE)
| > 1D - EmdL1(h1,h2)=68.420658
| > [1] 68.42
| > edd at ron:~/svn/earthmovdist$
| >
| > as
| >
| > edd at ron:~/svn/earthmovdist$ ldd /usr/local/lib/R/site-library/ 
| > earthmovdist/libs/earthmovdist.so
| >        linux-gate.so.1 =>  (0xb7f67000)
| >        libRcpp.so => /usr/local/lib/R/site-library/Rcpp/lib/ 
| > libRcpp.so (0xb7f2c000)
| >        libR.so => /usr/lib/R/lib/libR.so (0xb7bfb000)
| >        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7b09000)
| >        libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7ae3000)
| >        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7ac6000)
| >        libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb797f000)
| >        libblas.so.3gf => /usr/lib/atlas/libblas.so.3gf (0xb7604000)
| >        libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb753e000)
| >        libreadline.so.6 => /lib/libreadline.so.6 (0xb7509000)
| >        libpcre.so.3 => /lib/libpcre.so.3 (0xb74d9000)
| >        libbz2.so.1.0 => /lib/libbz2.so.1.0 (0xb74c9000)
| >        libz.so.1 => /usr/lib/libz.so.1 (0xb74b5000)
| >        libdl.so.2 => /lib/i686/cmov/libdl.so.2 (0xb74b1000)
| >        /lib/ld-linux.so.2 (0xb7f68000)
| >        libncurses.so.5 => /lib/libncurses.so.5 (0xb7478000)
| > edd at ron:~/svn/earthmovdist$
| >
| > That would be the same for phylobase (in the sense that it would  
| > know where
| > to find libRcpp.so), plus we still would have to deal with NCL.
| >
| > Now, I just checked on R-Forge on earthmovdist did NOT build cleanly  
| > the last
| > time which has me puzzled.  This should have worked.
| >
| > (Small) tarball for earthmovdist attached.
| >
| > <earthmovdist_0.1.0.tar.gz>
| > This corresponds to the most recent revision r24.
| >
| > | On Jan 15, 2010, at 11:56 AM, Dirk Eddelbuettel wrote:
| > | >  [snip successful attempt]
| > | > | >
| > | > | > So it appears to be an issue of the dynamic linker not  
| > finding the
| > | > | > right version of libRcpp.dylib
| > | > | >
| > | > | >
| > | > | > I'm not sure how to fix it other than altering the user's env
| > | > (which
| > | > | > is obviously not ideal).
| > | >
| > | > The idea (originally suggested by Simon Urbanek who also supplied
| > | > quite some
| > | > help in getting the dylib business right) was that from an R  
| > prompt
| > | > (or shell
| > | > prompt via Rscript / littler) you could call Rcpp itself by  
| > using the
| > | > functions
| > | >
| > | > 	  Rcpp:::CxxFlags()
| > | > 	  Rcpp:::LdFlags()
| > |
| > | Sorry for the confusion -- I meant to say the "loader not finding"
| > | rather than "linker not finding".  It links, but the loading  
| > fails. So
| > | I think that your Rcpp:::LdFlags() is working.
| >
| > Then I may well need some OS X help on learning you guys instrument  
| > the
| > loader.  I thought Simon and I had that squared away.
| >
| > | [snip]
| > | >
| > | > ( Building RQuantLib munges a lot CPU because QuantLib so huge.   
| > And
| > | > you'd
| > | > need Quantlib (Debian/ Ubuntu do. So this may be easier to look at
| > | > than to
| > | > try to run at your end. )
| > |
| > | You were not kidding -- I've never seen a library as big as  
| > QuantLib!
| > | I did get it installed.
| > |
| > | I see the same errors with running RQuantLib from an unaltered env
| > | that I do for phylobase:
| > |
| > | ###################################
| > | $ R CMD check  pkg
| > | * checking for working pdflatex ... OK
| > | * using log directory '/Users/mholder/builds/rin/rquantlib/ 
| > pkg.Rcheck'
| > | [snip]
| > | * checking R files for syntax errors ... OK
| > | * checking whether the package can be loaded ... ERROR
| > |
| > | Loading required package: Rcpp
| > | Error in dyn.load(file, DLLpath = DLLpath, ...) :
| > |    unable to load shared library '/Users/mholder/builds/rin/ 
| > rquantlib/
| > | pkg.Rcheck/RQuantLib/libs/i386/RQuantLib.so':
| > |    dlopen(/Users/mholder/builds/rin/rquantlib/pkg.Rcheck/RQuantLib/
| > | libs/i386/RQuantLib.so, 6): Library not loaded: /Builds/Rdev-web/QA/
| > | Simon/packages/tiger-universal/Rlib/2.10/Rcpp/lib/i386/libRcpp.dylib
| > |    Referenced from: /Users/mholder/builds/rin/rquantlib/pkg.Rcheck/
| > | RQuantLib/libs/i386/RQuantLib.so
| > |    Reason: image not found
| > | Error in library(RQuantLib) : .First.lib failed for 'RQuantLib'
| > | Execution halted
| > |
| > | ###################################
| > |
| > | If I can install and then I add the parent of libRcpp.dylib to my
| > | DYLD_LIBRARY_PATH, then library(RQuantLib) succeeds.
| > |
| > |
| > | I looked at https://r-forge.r-project.org/R/?group_id=111&log=check_x86_64_mac&pkg=RQuantLib&flavor=patched
| > | which indicates no problems.
| >
| > Right.  I do not know what R-Forge does but they don't cut any  
| > specials for
| > Rcpp as far as I know.
| >
| > | Could it be that it is the
| > | "Additional arguments to R CMD check: --install=fake"
| > | line that explains why this build-bot is succeeding while  
| > phylobase's
| > | is failing?
| >
| > AFAIK --fake does less rather than more and I would expect it to  
| > fail earlier
| > / more often.  R CMD INSTALL --help says:
| >
| > --fake            do minimal install for testing purposes
| >
| >
| > Dirk
| >
| > | I tried adding that --install-args="--install=fake" but that never
| > | seemed to result in a "Additional arguments to R CMD check: ..."
| > | statement in the output.
| > |
| > |
| > | all the best,
| > | Mark
| > |
| > |
| > |
| > |
| > |
| > |
| > | Mark Holder
| > |
| > | mtholder at ku.edu
| > | http://www.people.ku.edu/~mtholder/
| > |
| > | ==============================================
| > | Department of Ecology and Evolutionary Biology
| > | University of Kansas
| > | 6031 Haworth Hall
| > | 1200 Sunnyside Avenue
| > | Lawrence, Kansas 66045
| > |
| > | lab phone:  785.864.5789
| > |
| > | fax (shared): 785.864.5860
| > | ==============================================
| > |
| > |
| > |
| > |
| > |
| > |
| > |
| >
| > -- 
| > Three out of two people have difficulties with fractions.
| > _______________________________________________
| > Phylobase-devl mailing list
| > Phylobase-devl at lists.r-forge.r-project.org
| > https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/phylobase-devl
| 
| Mark Holder
| 
| mtholder at ku.edu
| http://www.people.ku.edu/~mtholder/
| 
| ==============================================
| Department of Ecology and Evolutionary Biology
| University of Kansas
| 6031 Haworth Hall
| 1200 Sunnyside Avenue
| Lawrence, Kansas 66045
| 
| lab phone:  785.864.5789
| 
| fax (shared): 785.864.5860
| ==============================================
| 
| 
| 
| 
| 
| 
| 

-- 
Three out of two people have difficulties with fractions.