[Phylobase-devl] Rcpp and OS X compiliation
Mark Holder
mtholder at ku.edu
Mon Jan 18 03:07:03 CET 2010
Hi,
>
> | III. A package that compiles against libRcpp seems to be picking up
> | the path to libRcpp.dylib from the path listed in the library itself
> | (rather than the path where the library is found during
> compilation).
>
> Can you explain this again? You confuse me here. Aren't those the
> same?
> Packages compiling against Rcpp are typically using facilities
> provided by
> Rcpp -- in particular Rcpp:::LdFlags() -- which will point to the
> installed
> library (as in the previous point).
They are the same if you build Rcpp yourself. If you downloaded the
lib using install.packages
then the libraries linking to Rcpp (such as phylobase.so) will include
a link to some path
on Simon's machine (instead of the location of Rcpp on my hard drive).
This seems to happen regardless of the fact that at the stage of
linking phylobase, the LdFlags refer
to the correct location of libRcpp. phylobase.so will have the path
that libRcpp.dylib contains.
> | I don't know if this is a standard aspect of dynamic linking/loading
> | or something that is a glitch in the R package installation.
>
> Can you explain again what the glitch is?
I was referring to the fact that it trusts where the library is rather
than where it found the library (effectively making it impossible to
move to the library and then dynamically link to it).
> | Confusingly, if you:
> | - build rcpp from source
> | - build phylobase
> | - then install rcpp from install.packages()
> | everything will work because phylobase will have the correct
> location
> | for libRcpp.dylib (even though the version of libRcpp.dylib that it
> | finds will list the wrong path to itself).
>
> (Sure. What matter, AFAICT, is how phylobase encodes references to
> Rcpp. As
> we started with the right information we are good.)
>
> | Solutions seem to be:
> | 1. (best) figure out how to get install.packages to modify
> | libRcpp.dylib so that it list its own location correctly,
>
> Agreed. Now how do we do this?
It looks like your:
/usr/bin/install_name_tool -id $(R_PACKAGE_DIR)/lib$(R_ARCH)/$
(USERLIB) $(USERLIB);
works like a charm.
If we could get that executed as via some hook during install.packages
(for Mac or perhaps any BSD system), then I think that would make it
possible to install Rcpp using install.packages.
I'm still not sure if any Rcpp-dependent libs would also have to call
/usr/bin/install_name -change libRcpp.dylib $(DIR_WITH_RCPP_LIB)/
libRcpp.dylib
where DIR_WITH_RCPP_LIB would have to be detected at install time.
> Maybe add this option:
>
> 4. Stop using dynamic linking on OS X and link statically to
> libRcpp.a.
> This one is already provided, and I can make it easier to get the
> proper link
> instruction. That would be more failsafe, but less elegant (as
> dynamic
> linking is).
I agree. Static linking is probably the most reliable way to fix all
of this.
all the best,
Mark
> Cheers, Dirk
>
>
>
>
> | all the best,
> | Mark
> |
> |
> |
> |
> |
> |
> |
> | On Jan 16, 2010, at 10:17 PM, Dirk Eddelbuettel wrote:
> |
> | >
> | > Hi Mark,
> | >
> | > On 16 January 2010 at 20:57, Mark Holder wrote:
> | > | Hi again,
> | > |
> | > | On Jan 15, 2010, at 11:16 AM, Ben Bolker wrote:
> | > |
> | > | > Mark Holder wrote:
> | > | >>
> | > | >> I'm not sure how to fix it other than altering the user's env
> | > (which
> | > | >> is obviously not ideal).
> | > | >
> | > | > No, but it might be an interim hack: R has Sys.getenv() and
> | > | > Sys.setenv() which might be usable ? (although they might
> have
> | > to be
> | > | > called too early in the package-loading process)
> | > |
> | > | I am not having any success getting the library to load by
> modifying
> | > | DYLD_LIBRARY_PATH from within R.
> | >
> | > Seconded. I played with that a looong time ago and concluded that
> | > these
> | > environment variables are 'sealed' once you are running your
> | > process. You
> | > cannot modify your own process. Which is probably why the (shell
> | > script)
> | > /usr/bin/R sets them up before calling the underlying binary.
> | >
> | > | I can augment the list of paths successfully (with
> | > | Sys.setenv(DYLD_LIBRARY_PATH="/Us..."),
> | > | but that doesn't seem to help the loader. I suspect that you
> are
> | > | correct that using setenv is simply too late.
> | >
> | > Agreed.
> | >
> | > | According to "Writing R extensions
> | > | ${R_HOME}/etc/${R_ARCH}/ldpaths is sourced before the R
> process is
> | > | launched.
> | >
> | > Exactly.
> | >
> | > | On my machine this is:
> | > | /Library/Frameworks/R.framework/Resources/etc/i386/ldpaths
> | > |
> | > | This file adds ${R_HOME}/lib/${R_ARCH} to the dynamic load
> library
> | > | path list (one an intel-mac this is /Library/Frameworks/
> R.framework/
> | > | Resources/lib/i386). This directory contains libR.dylib,
> | > | libRblas.dylib, and libRlapack.dylib on my machine.
> | > |
> | > | I can get phylobase to run if I do any one of the following
> four
> | > | things:
> | > | 1. Add ~/Library/R/2.10/library/Rcpp/lib/i386 to the user's env
> | > | before calling R
> | > | 2. Add ~/Library/R/2.10/library/Rcpp/lib/i386 to
> | > DYLD_LIBRARY_PATH in
> | > | the /Library/Frameworks/R.framework/Resources/etc/i386/ldpaths
> file
> | > | 3. Copy ~/Library/R/2.10/library/Rcpp/lib/i386/libRcpp.dylib
> to /
> | > | Library/Frameworks/R.framework/Resources/lib/i386
> | > | 4. Copy ~/Library/R/2.10/library/Rcpp/lib/i386/libRcpp.dylib
> to /
> | > usr/
> | > | local/lib
> | >
> | > Hm. None of 1 to 4 should be necessary if the 'rpath' option works
> | > correctly.
> | > Does that exist / work on OS X? Otherwise I have a problem with
> | > Rcpp that is
> | > much larger than phylobase :-/ --- and I am surprised that has not
> | > come up
> | > before. As AFAIK the build bot on R-Forge doesn't use this
> either.
> | >
> | > I would still like to disentangle things, if possible.
> | >
> | > Could you have a look at the baby package 'earthmovdist' from R-
> | > Forge? I
> | > attach a tarball for your persusal. It simple uses Rcpp to get
> to a
> | > simple
> | > C++ class. No external library, but of course a dependency on
> Rcpp's
> | > library. On my system this builds as
> | >
> | > edd at ron:~/svn/earthmovdist$ R CMD INSTALL
> earthmovdist_0.1.0.tar.gz
> | > * installing to library ‘/usr/local/lib/R/site-library’
> | > * installing *source* package ‘earthmovdist’ ...
> | > ** libs
> | > g++ -I/usr/share/R/include -I/usr/local/lib/R/site-library/Rcpp/
> | > lib -fpic -g -O3 -Wall -pipe -c emdL1.cpp -o emdL1.o
> | > g++ -I/usr/share/R/include -I/usr/local/lib/R/site-library/Rcpp/
> | > lib -fpic -g -O3 -Wall -pipe -c emdL1_R.cpp -o emdL1_R.o
> | > g++ -shared -o earthmovdist.so emdL1.o emdL1_R.o -L/usr/local/
> lib/R/
> | > site-library/Rcpp/lib -lRcpp -Wl,-rpath,/usr/local/lib/R/site-
> | > library/Rcpp/lib -L/usr/lib/R/lib -lR
> | > ** R
> | > ** inst
> | > ** preparing package for lazy loading
> | > ** help
> | > *** installing help indices
> | > ** building package indices ...
> | > * DONE (earthmovdist)
> | > edd at ron:~/svn/earthmovdist$
> | >
> | > and I can then say
> | >
> | > edd at ron:~/svn/earthmovdist$ Rscript -e 'library(earthmovdist);
> | > example(emdL1)'
> | > Loading required package: Rcpp
> | > Loading required package: methods
> | >
> | > emdL1R> set.seed(42)
> | >
> | > emdL1R> x <- rnorm(100)
> | >
> | > emdL1R> y <- x + rnorm(100)/10
> | >
> | > emdL1R> emdL1(x, y, verbose=TRUE)
> | > 1D - EmdL1(h1,h2)=68.420658
> | > [1] 68.42
> | > edd at ron:~/svn/earthmovdist$
> | >
> | > as
> | >
> | > edd at ron:~/svn/earthmovdist$ ldd /usr/local/lib/R/site-library/
> | > earthmovdist/libs/earthmovdist.so
> | > linux-gate.so.1 => (0xb7f67000)
> | > libRcpp.so => /usr/local/lib/R/site-library/Rcpp/lib/
> | > libRcpp.so (0xb7f2c000)
> | > libR.so => /usr/lib/R/lib/libR.so (0xb7bfb000)
> | > libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0xb7b09000)
> | > libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7ae3000)
> | > libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb7ac6000)
> | > libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb797f000)
> | > libblas.so.3gf => /usr/lib/atlas/libblas.so.3gf
> (0xb7604000)
> | > libgfortran.so.3 => /usr/lib/libgfortran.so.3 (0xb753e000)
> | > libreadline.so.6 => /lib/libreadline.so.6 (0xb7509000)
> | > libpcre.so.3 => /lib/libpcre.so.3 (0xb74d9000)
> | > libbz2.so.1.0 => /lib/libbz2.so.1.0 (0xb74c9000)
> | > libz.so.1 => /usr/lib/libz.so.1 (0xb74b5000)
> | > libdl.so.2 => /lib/i686/cmov/libdl.so.2 (0xb74b1000)
> | > /lib/ld-linux.so.2 (0xb7f68000)
> | > libncurses.so.5 => /lib/libncurses.so.5 (0xb7478000)
> | > edd at ron:~/svn/earthmovdist$
> | >
> | > That would be the same for phylobase (in the sense that it would
> | > know where
> | > to find libRcpp.so), plus we still would have to deal with NCL.
> | >
> | > Now, I just checked on R-Forge on earthmovdist did NOT build
> cleanly
> | > the last
> | > time which has me puzzled. This should have worked.
> | >
> | > (Small) tarball for earthmovdist attached.
> | >
> | > <earthmovdist_0.1.0.tar.gz>
> | > This corresponds to the most recent revision r24.
> | >
> | > | On Jan 15, 2010, at 11:56 AM, Dirk Eddelbuettel wrote:
> | > | > [snip successful attempt]
> | > | > | >
> | > | > | > So it appears to be an issue of the dynamic linker not
> | > finding the
> | > | > | > right version of libRcpp.dylib
> | > | > | >
> | > | > | >
> | > | > | > I'm not sure how to fix it other than altering the
> user's env
> | > | > (which
> | > | > | > is obviously not ideal).
> | > | >
> | > | > The idea (originally suggested by Simon Urbanek who also
> supplied
> | > | > quite some
> | > | > help in getting the dylib business right) was that from an R
> | > prompt
> | > | > (or shell
> | > | > prompt via Rscript / littler) you could call Rcpp itself by
> | > using the
> | > | > functions
> | > | >
> | > | > Rcpp:::CxxFlags()
> | > | > Rcpp:::LdFlags()
> | > |
> | > | Sorry for the confusion -- I meant to say the "loader not
> finding"
> | > | rather than "linker not finding". It links, but the loading
> | > fails. So
> | > | I think that your Rcpp:::LdFlags() is working.
> | >
> | > Then I may well need some OS X help on learning you guys
> instrument
> | > the
> | > loader. I thought Simon and I had that squared away.
> | >
> | > | [snip]
> | > | >
> | > | > ( Building RQuantLib munges a lot CPU because QuantLib so
> huge.
> | > And
> | > | > you'd
> | > | > need Quantlib (Debian/ Ubuntu do. So this may be easier to
> look at
> | > | > than to
> | > | > try to run at your end. )
> | > |
> | > | You were not kidding -- I've never seen a library as big as
> | > QuantLib!
> | > | I did get it installed.
> | > |
> | > | I see the same errors with running RQuantLib from an unaltered
> env
> | > | that I do for phylobase:
> | > |
> | > | ###################################
> | > | $ R CMD check pkg
> | > | * checking for working pdflatex ... OK
> | > | * using log directory '/Users/mholder/builds/rin/rquantlib/
> | > pkg.Rcheck'
> | > | [snip]
> | > | * checking R files for syntax errors ... OK
> | > | * checking whether the package can be loaded ... ERROR
> | > |
> | > | Loading required package: Rcpp
> | > | Error in dyn.load(file, DLLpath = DLLpath, ...) :
> | > | unable to load shared library '/Users/mholder/builds/rin/
> | > rquantlib/
> | > | pkg.Rcheck/RQuantLib/libs/i386/RQuantLib.so':
> | > | dlopen(/Users/mholder/builds/rin/rquantlib/pkg.Rcheck/
> RQuantLib/
> | > | libs/i386/RQuantLib.so, 6): Library not loaded: /Builds/Rdev-
> web/QA/
> | > | Simon/packages/tiger-universal/Rlib/2.10/Rcpp/lib/i386/
> libRcpp.dylib
> | > | Referenced from: /Users/mholder/builds/rin/rquantlib/
> pkg.Rcheck/
> | > | RQuantLib/libs/i386/RQuantLib.so
> | > | Reason: image not found
> | > | Error in library(RQuantLib) : .First.lib failed for 'RQuantLib'
> | > | Execution halted
> | > |
> | > | ###################################
> | > |
> | > | If I can install and then I add the parent of libRcpp.dylib to
> my
> | > | DYLD_LIBRARY_PATH, then library(RQuantLib) succeeds.
> | > |
> | > |
> | > | I looked at https://r-forge.r-project.org/R/?group_id=111&log=check_x86_64_mac&pkg=RQuantLib&flavor=patched
> | > | which indicates no problems.
> | >
> | > Right. I do not know what R-Forge does but they don't cut any
> | > specials for
> | > Rcpp as far as I know.
> | >
> | > | Could it be that it is the
> | > | "Additional arguments to R CMD check: --install=fake"
> | > | line that explains why this build-bot is succeeding while
> | > phylobase's
> | > | is failing?
> | >
> | > AFAIK --fake does less rather than more and I would expect it to
> | > fail earlier
> | > / more often. R CMD INSTALL --help says:
> | >
> | > --fake do minimal install for testing purposes
> | >
> | >
> | > Dirk
> | >
> | > | I tried adding that --install-args="--install=fake" but that
> never
> | > | seemed to result in a "Additional arguments to R CMD check: ..."
> | > | statement in the output.
> | > |
> | > |
> | > | all the best,
> | > | Mark
> | > |
> | > |
> | > |
> | > |
> | > |
> | > |
> | > | Mark Holder
> | > |
> | > | mtholder at ku.edu
> | > | http://www.people.ku.edu/~mtholder/
> | > |
> | > | ==============================================
> | > | Department of Ecology and Evolutionary Biology
> | > | University of Kansas
> | > | 6031 Haworth Hall
> | > | 1200 Sunnyside Avenue
> | > | Lawrence, Kansas 66045
> | > |
> | > | lab phone: 785.864.5789
> | > |
> | > | fax (shared): 785.864.5860
> | > | ==============================================
> | > |
> | > |
> | > |
> | > |
> | > |
> | > |
> | > |
> | >
> | > --
> | > Three out of two people have difficulties with fractions.
> | > _______________________________________________
> | > Phylobase-devl mailing list
> | > Phylobase-devl at lists.r-forge.r-project.org
> | > https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/phylobase-devl
> |
> | Mark Holder
> |
> | mtholder at ku.edu
> | http://www.people.ku.edu/~mtholder/
> |
> | ==============================================
> | Department of Ecology and Evolutionary Biology
> | University of Kansas
> | 6031 Haworth Hall
> | 1200 Sunnyside Avenue
> | Lawrence, Kansas 66045
> |
> | lab phone: 785.864.5789
> |
> | fax (shared): 785.864.5860
> | ==============================================
> |
> |
> |
> |
> |
> |
> |
>
> --
> Three out of two people have difficulties with fractions.
Mark Holder
mtholder at ku.edu
http://www.people.ku.edu/~mtholder/
==============================================
Department of Ecology and Evolutionary Biology
University of Kansas
6031 Haworth Hall
1200 Sunnyside Avenue
Lawrence, Kansas 66045
lab phone: 785.864.5789
fax (shared): 785.864.5860
==============================================
More information about the Phylobase-devl
mailing list