[Phylobase-devl] Rcpp and OS X compiliation

Mark Holder mtholder at ku.edu
Sun Jan 17 15:50:29 CET 2010


Hi all,
	Things are clearing up for me, but I'm afraid that I have not been  
terribly disciplined as I play around with potential fixes.  So I  
don't have terminal output from each step.

Based on using "otool -L" (to see which libraries a dylib or so is  
referring to), I now think that this is what is happening:

I. The Rcpp package that you can install via install.packages contains  
a reference to
	/Builds/Rdev-web/QA/Simon/packages/tiger-universal/Rlib/2.10/Rcpp/lib/ 
i386/libRcpp.dylib
  This looks like the source of the problem, as no user's machine will  
have that path. It may be OK for the library just lists itself by name  
rather than full path and that this may solve the problems (but I'm  
not sure about that).

II. Compiling Rcpp  (at least from svn) puts the correct install  
location for the library in its list of libraries ("/Users/mholder/ 
Library/R/2.10/library/Rcpp/lib/i386/libRcpp.dylib" in my case).  I  
think that it is INSTALL that does this (it is interesting really  
because in my build dir ~/builds/rin/rcpp/pkg/inst/lib/i386 the  
library has a reference to its install rather than build location).

III. A package that compiles against libRcpp seems to be picking up  
the path to libRcpp.dylib from the path listed in the library itself  
(rather than the path where the library is found during compilation).   
I don't know if this is a standard aspect of dynamic linking/loading  
or something that is a glitch in the R package installation.

IV. When dynamically loading a package, the loader tries:
	A. "Special" locations (things listed in DYLD_LIBRARY_PATH or ldpaths)
	B. the full path listed in the calling library if the path is listed  
as a full path.
	C. the full library search path which R augments with all of the  
package directories.  So library(x) will load any dylibs in that  
packages own lib directory because R tells the dynamic loader to look  
there.



So here is the behavior that I see:
1. the function symbols found in libRCpp are fine whether the library  
is installed from source or not -- so if the loader finds the library  
all will be well.

2. If you installed Rcpp FROM SOURCE. and then build your Rcpp- 
dependent packages all is well. (libRcpp.dylib will list the correct  
path to itself, and so the calling lib will list the correct path to  
libRcpp.dylib)

3. If you installed Rcpp using install.packages and THEN build your  
Rcpp-dependent packages:
	1. You get the infamous "unable to load shared library... Library not  
loaded: /Builds/Rdev-web/QA/Simon/packages/tiger-universal/Rlib/2.10/ 
Rcpp/lib/i386/libRcpp.dylib" error
	2. If you add the correct path to libRcpp.dylib in some special way,  
then point IV(A) above preempts step IV(B).  Everything is fine.


Confusingly, if you:
	- build rcpp from source
	- build phylobase
	- then install rcpp from install.packages()
everything will work because phylobase will have the correct location  
for libRcpp.dylib (even though the version of libRcpp.dylib that it  
finds will list the wrong path to itself).




Solutions seem to be:
	1. (best) figure out how to get install.packages to modify  
libRcpp.dylib so that it list its own location correctly,
	2. distribute the needed parts of libRcpp as a tool that installs to / 
usr/local/lib rather than an R package,
	2. disable compiling against libRcpp package if it was downloaded  
from the web (I don't know how to do that, mind you), or
	3. alter the default dyld behavior (through DYLD_LIBRARY_PATH or  
copying libRcpp.dylib to /usr/lib) so that it does not Matter that the  
path the libRcpp.dylib is incorrectly listed.




all the best,
Mark


	
	



On Jan 16, 2010, at 10:17 PM, Dirk Eddelbuettel wrote:

>
> Hi Mark,
>
> On 16 January 2010 at 20:57, Mark Holder wrote:
> | Hi again,
> |
> | On Jan 15, 2010, at 11:16 AM, Ben Bolker wrote:
> |
> | > Mark Holder wrote:
> | >>
> | >> I'm not sure how to fix it other than altering the user's env  
> (which
> | >> is obviously not ideal).
> | >
> | >  No, but it might be an interim hack: R has Sys.getenv() and
> | > Sys.setenv() which might be usable ?  (although they might have  
> to be
> | > called too early in the package-loading process)
> |
> | I am not having any success getting the library to load by modifying
> | DYLD_LIBRARY_PATH from within R.
>
> Seconded. I played with that a looong time ago and concluded that  
> these
> environment variables are 'sealed' once you are running your  
> process. You
> cannot modify your own process. Which is probably why the (shell  
> script)
> /usr/bin/R sets them up before calling the underlying binary.
>
> | I can augment the list of paths successfully (with
> | Sys.setenv(DYLD_LIBRARY_PATH="/Us..."),
> | but that doesn't seem to help the loader.  I suspect that you are
> | correct that using setenv is simply too late.
>
> Agreed.
>
> | According to "Writing R extensions
> | ${R_HOME}/etc/${R_ARCH}/ldpaths is sourced before the R process is
> | launched.
>
> Exactly.
>
> | On my machine this is:
> | 	/Library/Frameworks/R.framework/Resources/etc/i386/ldpaths
> |
> | This file adds ${R_HOME}/lib/${R_ARCH} to the dynamic load library
> | path list (one an intel-mac this is /Library/Frameworks/R.framework/
> | Resources/lib/i386). This directory contains libR.dylib,
> | libRblas.dylib, and  libRlapack.dylib on my machine.
> |
> | I can get phylobase to run if I do any one of the  following four
> | things:
> | 	1. Add ~/Library/R/2.10/library/Rcpp/lib/i386 to the user's env
> | before calling R
> | 	2. Add ~/Library/R/2.10/library/Rcpp/lib/i386 to  
> DYLD_LIBRARY_PATH in
> | the /Library/Frameworks/R.framework/Resources/etc/i386/ldpaths file
> | 	3. Copy ~/Library/R/2.10/library/Rcpp/lib/i386/libRcpp.dylib to  /
> | Library/Frameworks/R.framework/Resources/lib/i386
> | 	4. Copy ~/Library/R/2.10/library/Rcpp/lib/i386/libRcpp.dylib to / 
> usr/
> | local/lib
>
> Hm. None of 1 to 4 should be necessary if the 'rpath' option works  
> correctly.
> Does that exist / work on OS X?  Otherwise I have a problem with  
> Rcpp that is
> much larger than phylobase :-/ --- and I am surprised that has not  
> come up
> before.  As AFAIK the build bot on R-Forge doesn't use this either.
>
> I would still like to disentangle things, if possible.
>
> Could you have a look at the baby package 'earthmovdist' from R- 
> Forge?  I
> attach a tarball for your persusal.  It simple uses Rcpp to get to a  
> simple
> C++ class.  No external library, but of course a dependency on Rcpp's
> library.   On my system this builds as
>
> edd at ron:~/svn/earthmovdist$ R CMD INSTALL earthmovdist_0.1.0.tar.gz
> * installing to library ‘/usr/local/lib/R/site-library’
> * installing *source* package ‘earthmovdist’ ...
> ** libs
> g++ -I/usr/share/R/include -I/usr/local/lib/R/site-library/Rcpp/ 
> lib     -fpic  -g -O3 -Wall -pipe -c emdL1.cpp -o emdL1.o
> g++ -I/usr/share/R/include -I/usr/local/lib/R/site-library/Rcpp/ 
> lib     -fpic  -g -O3 -Wall -pipe -c emdL1_R.cpp -o emdL1_R.o
> g++ -shared -o earthmovdist.so emdL1.o emdL1_R.o -L/usr/local/lib/R/ 
> site-library/Rcpp/lib -lRcpp -Wl,-rpath,/usr/local/lib/R/site- 
> library/Rcpp/lib -L/usr/lib/R/lib -lR
> ** R
> ** inst
> ** preparing package for lazy loading
> ** help
> *** installing help indices
> ** building package indices ...
> * DONE (earthmovdist)
> edd at ron:~/svn/earthmovdist$
>
> and I can then say
>
> edd at ron:~/svn/earthmovdist$ Rscript -e 'library(earthmovdist);  
> example(emdL1)'
> Loading required package: Rcpp
> Loading required package: methods
>
> emdL1R>   set.seed(42)
>
> emdL1R>   x <- rnorm(100)
>
> emdL1R>   y <- x + rnorm(100)/10
>
> emdL1R>   emdL1(x, y, verbose=TRUE)
> 1D - EmdL1(h1,h2)=68.420658
> [1] 68.42
> edd at ron:~/svn/earthmovdist$
>
> as
>
> edd at ron:~/svn/earthmovdist$ ldd /usr/local/lib/R/site-library/ 
> earthmovdist/libs/earthmovdist.so
>        linux-gate.so.1 =>  (0xb7f67000)
>        libRcpp.so => /usr/local/lib/R/site-library/Rcpp/lib/ 
> libRcpp.so (0xb7f2c000)
>        libR.so => /usr/lib/R/lib/libR.so (0xb7bfb000)
>        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7b09000)
>        libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7ae3000)
>        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7ac6000)
>        libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb797f000)
>        libblas.so.3gf => /usr/lib/atlas/libblas.so.3gf (0xb7604000)
>        libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb753e000)
>        libreadline.so.6 => /lib/libreadline.so.6 (0xb7509000)
>        libpcre.so.3 => /lib/libpcre.so.3 (0xb74d9000)
>        libbz2.so.1.0 => /lib/libbz2.so.1.0 (0xb74c9000)
>        libz.so.1 => /usr/lib/libz.so.1 (0xb74b5000)
>        libdl.so.2 => /lib/i686/cmov/libdl.so.2 (0xb74b1000)
>        /lib/ld-linux.so.2 (0xb7f68000)
>        libncurses.so.5 => /lib/libncurses.so.5 (0xb7478000)
> edd at ron:~/svn/earthmovdist$
>
> That would be the same for phylobase (in the sense that it would  
> know where
> to find libRcpp.so), plus we still would have to deal with NCL.
>
> Now, I just checked on R-Forge on earthmovdist did NOT build cleanly  
> the last
> time which has me puzzled.  This should have worked.
>
> (Small) tarball for earthmovdist attached.
>
> <earthmovdist_0.1.0.tar.gz>
> This corresponds to the most recent revision r24.
>
> | On Jan 15, 2010, at 11:56 AM, Dirk Eddelbuettel wrote:
> | >  [snip successful attempt]
> | > | >
> | > | > So it appears to be an issue of the dynamic linker not  
> finding the
> | > | > right version of libRcpp.dylib
> | > | >
> | > | >
> | > | > I'm not sure how to fix it other than altering the user's env
> | > (which
> | > | > is obviously not ideal).
> | >
> | > The idea (originally suggested by Simon Urbanek who also supplied
> | > quite some
> | > help in getting the dylib business right) was that from an R  
> prompt
> | > (or shell
> | > prompt via Rscript / littler) you could call Rcpp itself by  
> using the
> | > functions
> | >
> | > 	  Rcpp:::CxxFlags()
> | > 	  Rcpp:::LdFlags()
> |
> | Sorry for the confusion -- I meant to say the "loader not finding"
> | rather than "linker not finding".  It links, but the loading  
> fails. So
> | I think that your Rcpp:::LdFlags() is working.
>
> Then I may well need some OS X help on learning you guys instrument  
> the
> loader.  I thought Simon and I had that squared away.
>
> | [snip]
> | >
> | > ( Building RQuantLib munges a lot CPU because QuantLib so huge.   
> And
> | > you'd
> | > need Quantlib (Debian/ Ubuntu do. So this may be easier to look at
> | > than to
> | > try to run at your end. )
> |
> | You were not kidding -- I've never seen a library as big as  
> QuantLib!
> | I did get it installed.
> |
> | I see the same errors with running RQuantLib from an unaltered env
> | that I do for phylobase:
> |
> | ###################################
> | $ R CMD check  pkg
> | * checking for working pdflatex ... OK
> | * using log directory '/Users/mholder/builds/rin/rquantlib/ 
> pkg.Rcheck'
> | [snip]
> | * checking R files for syntax errors ... OK
> | * checking whether the package can be loaded ... ERROR
> |
> | Loading required package: Rcpp
> | Error in dyn.load(file, DLLpath = DLLpath, ...) :
> |    unable to load shared library '/Users/mholder/builds/rin/ 
> rquantlib/
> | pkg.Rcheck/RQuantLib/libs/i386/RQuantLib.so':
> |    dlopen(/Users/mholder/builds/rin/rquantlib/pkg.Rcheck/RQuantLib/
> | libs/i386/RQuantLib.so, 6): Library not loaded: /Builds/Rdev-web/QA/
> | Simon/packages/tiger-universal/Rlib/2.10/Rcpp/lib/i386/libRcpp.dylib
> |    Referenced from: /Users/mholder/builds/rin/rquantlib/pkg.Rcheck/
> | RQuantLib/libs/i386/RQuantLib.so
> |    Reason: image not found
> | Error in library(RQuantLib) : .First.lib failed for 'RQuantLib'
> | Execution halted
> |
> | ###################################
> |
> | If I can install and then I add the parent of libRcpp.dylib to my
> | DYLD_LIBRARY_PATH, then library(RQuantLib) succeeds.
> |
> |
> | I looked at https://r-forge.r-project.org/R/?group_id=111&log=check_x86_64_mac&pkg=RQuantLib&flavor=patched
> | which indicates no problems.
>
> Right.  I do not know what R-Forge does but they don't cut any  
> specials for
> Rcpp as far as I know.
>
> | Could it be that it is the
> | "Additional arguments to R CMD check: --install=fake"
> | line that explains why this build-bot is succeeding while  
> phylobase's
> | is failing?
>
> AFAIK --fake does less rather than more and I would expect it to  
> fail earlier
> / more often.  R CMD INSTALL --help says:
>
> --fake            do minimal install for testing purposes
>
>
> Dirk
>
> | I tried adding that --install-args="--install=fake" but that never
> | seemed to result in a "Additional arguments to R CMD check: ..."
> | statement in the output.
> |
> |
> | all the best,
> | Mark
> |
> |
> |
> |
> |
> |
> | Mark Holder
> |
> | mtholder at ku.edu
> | http://www.people.ku.edu/~mtholder/
> |
> | ==============================================
> | Department of Ecology and Evolutionary Biology
> | University of Kansas
> | 6031 Haworth Hall
> | 1200 Sunnyside Avenue
> | Lawrence, Kansas 66045
> |
> | lab phone:  785.864.5789
> |
> | fax (shared): 785.864.5860
> | ==============================================
> |
> |
> |
> |
> |
> |
> |
>
> -- 
> Three out of two people have difficulties with fractions.
> _______________________________________________
> Phylobase-devl mailing list
> Phylobase-devl at lists.r-forge.r-project.org
> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/phylobase-devl

Mark Holder

mtholder at ku.edu
http://www.people.ku.edu/~mtholder/

==============================================
Department of Ecology and Evolutionary Biology
University of Kansas
6031 Haworth Hall
1200 Sunnyside Avenue
Lawrence, Kansas 66045

lab phone:  785.864.5789

fax (shared): 785.864.5860
==============================================









More information about the Phylobase-devl mailing list