[Rcpp-devel] RcppArmadillo inv() depends on Lapack, zgetri_ not available on CRAN / R-forge?
Douglas Bates
bates at stat.wisc.edu
Wed Jun 8 00:13:12 CEST 2011
On Tue, Jun 7, 2011 at 4:20 PM, baptiste auguie
<baptiste.auguie at googlemail.com> wrote:
> On 8 June 2011 09:04, Douglas Bates <bates at stat.wisc.edu> wrote:
>> On Tue, Jun 7, 2011 at 3:47 PM, baptiste auguie
>> <baptiste.auguie at googlemail.com> wrote:
>>> On 8 June 2011 08:03, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>> On Tue, Jun 7, 2011 at 2:34 PM, baptiste auguie
>>>> <baptiste.auguie at googlemail.com> wrote:
>>>>> Hi Doug,
>>>>>
>>>>> On 8 June 2011 03:43, Douglas Bates <bates at stat.wisc.edu> wrote:
>>>>>>
>>>>>> On Jun 6, 2011 4:17 AM, "baptiste auguie" <baptiste.auguie at googlemail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Thank you for the explanations below.
>>>>>>>
>>>>>>> On 5 June 2011 10:40, Dirk Eddelbuettel <edd at debian.org> wrote:
>>>>>>> >
>>>>>>> > On 5 June 2011 at 10:12, baptiste auguie wrote:
>>>>>>> > | Hi Dirk and all,
>>>>>>> > |
>>>>>>> > | On 4 June 2011 12:04, Dirk Eddelbuettel <edd at debian.org> wrote:
>>>>>>> > | >
>>>>>>> > | > Baptiste,
>>>>>>> > | >
>>>>>>> > | > On 4 June 2011 at 11:45, baptiste auguie wrote:
>>>>>>> > | > | Dear list,
>>>>>>> > | > |
>>>>>>> > | > | My package cda, which I was hoping to release on CRAN, fails to
>>>>>>> > | > | compile on R-forge with error,
>>>>>>> > | > |
>>>>>>> > | > | ** testing if installed package can be loaded
>>>>>>> > | > | Error in dyn.load(file, DLLpath = DLLpath, ...) :
>>>>>>> > | > | unable to load shared object
>>>>>>> > '/tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so':
>>>>>>> > | > | /tmp/RtmpbztUMm/Rinst1829c04c/cda/libs/cda.so: undefined symbol:
>>>>>>> > zgetri_
>>>>>>> > | > |
>>>>>>> > | > | It builds fine on my local machines (Mac OS 10.5, 10.6).
>>>>>>> > | > |
>>>>>>> > | > | >From an older discussion on this list <
>>>>>>> > | > |
>>>>>>> > http://www.mail-archive.com/rcpp-devel@lists.r-forge.r-project.org/msg00678.html>
>>>>>>> > | > | the issue seems to be that Armadillo's inv() relies on a function
>>>>>>> > that
>>>>>>> > | > | is not provided by R, only by LAPACK. I have just replaced inv()
>>>>>>> > by
>>>>>>> > | > | pinv() and solve() in my code; merely to see what happens, but
>>>>>>> > chances
>>>>>>> > | > | are they also require a full LAPACK.
>>>>>>> > |
>>>>>>> > | Indeed, the error on R-forge is now with zgels_, required to solve
>>>>>>> > | linear systems. It seems one cannot solve Armadillo linear systems
>>>>>>> > | without LAPACK in the current situation.
>>>>>>> >
>>>>>>> > Yes. Doug, Romain and myself should address that, or at least make it
>>>>>>> > clear
>>>>>>> > what feature of the full Armadillo are lacking in RcppArmadillo.
>>>>>>> >
>>>>>>> > | > Sometime relatively early in the RcppArmadillo development process,
>>>>>>> > Doug
>>>>>>> > | > convinced Romain and myself to go for a pure template solution with
>>>>>>> > Armadillo
>>>>>>> > | > as all / most things found during the configure (or in this case,
>>>>>>> > cmake)
>>>>>>> > | > stage can be assumed 'found' given that we have around us by design.
>>>>>>> > So no
>>>>>>> > | > testing, no local library and full reliance and what R gives us.
>>>>>>> > | >
>>>>>>> > | > That was a brilliant idea, and has freed us from having to rely on
>>>>>>> > building
>>>>>>> > | > and shipping a library, having to tell users how to set PKG_LIBS etc
>>>>>>> > pp and I
>>>>>>> > | > firmly believe that this helped tremendously in getting
>>>>>>> > RcppArmadillo more
>>>>>>> > | > widely used. So I would not want to revert this.
>>>>>>> > |
>>>>>>> > | It sounds like a good choice, I agree. Yet solving a linear system is
>>>>>>> > | quite a crucial task in linear algebra; it would be nice if we could
>>>>>>> > | come up with a tutorial recipe for dummies like me.
>>>>>>> > |
>>>>>>> > | >
>>>>>>> > | > In any event, it seems that you need more LAPACK than R has for you.
>>>>>>> > That is
>>>>>>> > | > likely to be a dicey situation as you per se do not know whether R
>>>>>>> > was built
>>>>>>> > | > and linked with its own (subset) copy of LAPACK, or whether it uses
>>>>>>> > system
>>>>>>> > | > LAPACK libraries (as e.g. the Debian / Ubuntu systems do). So you
>>>>>>> > may be in
>>>>>>> > | > a spot bother and I not sure what I can recommend --- other than
>>>>>>> > trying your
>>>>>>> > | > luck at some short configure snippets that will run at package build
>>>>>>> > time to
>>>>>>> > | > determine whether the system you want to build cda on it 'rich'
>>>>>>> > enough to
>>>>>>> > | > support it. I can help you off list with some configure snippets as
>>>>>>> > some of
>>>>>>> > | > my packages have configure code; adding a test for zgetri should be
>>>>>>> > feasible.
>>>>>>> > | >
>>>>>>> > | > | Does anybody have any experience
>>>>>>> > | > | dealing with such issues w.r.t releasing a package on R-forge /
>>>>>>> > CRAN?
>>>>>>> > | > | Is there any chance they would consider installing LAPACK on those
>>>>>>> > | > | servers, or is there no point in asking such things?
>>>>>>> > | >
>>>>>>> > | > I don't think it is a matter of fixing the R-Forge server. I think
>>>>>>> > it is a
>>>>>>> > | > matter of making your package installable on the largest number of
>>>>>>> > user
>>>>>>> > | > systems. Also try win-builder.r-project.org to see how it fares on
>>>>>>> > that
>>>>>>> > | > platform.
>>>>>>>
>>>>>>> Unsurprisingly, it fails, with the same complaint as R-forge.
>>>>>>>
>>>>>>> > |
>>>>>>> > | I was under the impression that R-forge or CRAN, if it had LAPACK
>>>>>>> > | installed, could produce binaries for the relevant platforms, and
>>>>>>> > | users would not have to build the package themselves and would not be
>>>>>>> > | required of having LAPACK on their machine. That's probably a
>>>>>>> > | misconception, isn't it?
>>>>>>> >
>>>>>>> > If and only statically linked binaries or libraries where produced,
>>>>>>> > which is
>>>>>>> > generally not the case. Many OSs (Linux incl) ship source only and
>>>>>>> > otherwise
>>>>>>> > link dynamically, others (Windoze) use dynamic linking and OS X is for
>>>>>>> > all I
>>>>>>> > know somewhere in the middle (as you can get prebuild packages with
>>>>>>> > dynamic
>>>>>>> > linking or build from source).
>>>>>>>
>>>>>>> I see; so basically the user will always need to have a full LAPACK
>>>>>>> installed. Here's one question then, if R-core didn't consider
>>>>>>> necessary to include those particular functions from LAPACK,
>>>>>>> presumably that means that R defines its own routines to solve linear
>>>>>>> systems and invert matrices. Is there any possibility to use those
>>>>>>> routines with Armadillo?
>>>>>>
>>>>>> One important point has been missed in this discussion, which is that the
>>>>>> particular Lapack subroutine that is not found in the subset of Lapack
>>>>>> shipped with R is for general, complex-valued matrices (just google the name
>>>>>> zgetri). The way that Armadillo is structured it is either all-in or
>>>>>> all-out with respect to complex-valued matrices If you allow for complex
>>>>>> matrices then you must have all the supporting subroutines for whatever you
>>>>>> are calling. If you call inv() you need to allow for all the [sdcz]getri
>>>>>> routines to be available.
>>>>>
>>>>> The matrix I need to invert is definitely always complex; in fact,
>>>>> convenient complex algebra is the main attraction of Armadillo for me.
>>>>
>>>> Argh. Of course, I was being foolish. Because Armadillo is a
>>>> header-only library it does not access any Lapack subroutine until the
>>>> call is instantiated. I still haven't quite gotten used to thinking
>>>> only having the headers.
>>>>
>>>>>>
>>>>>> So one possibility is to check the Armadillo sources to see if you can
>>>>>> suppress the use of complex-valued matrices when compiling your package. A
>>>>>> quick glance indicates that this might now be easy.
>>>>>>
>>>>>> Otherwise, remember the first rule of numerical linear algebra which is, "If
>>>>>> your algorithm involves computing the inverse of a matrix you should rethink
>>>>>> the algorithm because there is a better way of doing it". So why do you
>>>>>> need inv()?
>>>>>> If the answer is to solve a linear system of equations then you
>>>>>> use solve(), you do not use inv(). If, for some truly unusual reason you
>>>>>> actually need the inverse (and, remember in 99.99% of the cases where people
>>>>>> think they need the inverse, they don't) then you use a combination of
>>>>>> solve() and eye().
>>>>>
>>>>> I tried this, and in fact I do use solve() elsewhere in my code, which
>>>>> calls for another LAPACK routine (zgels.f) not present in R. The
>>>>> reason I still need inv() is that I have to solve about 300 times a
>>>>> linear system AX=B with the same matrix A but varying B. A quick
>>>>> timing last week revealed that using solve() 300 times was typically
>>>>> slower by a factor of two in my use case than using inv() once (or
>>>>> pinv() for that matter, it makes not appreciable difference). I'm
>>>>> happy to be shown otherwise though.
>>>>
>>>> I forgot that Armadillo doesn't provide a convenient way of using the
>>>> LU decomposition (that is one of the things that I find frustrating
>>>> about Armadillo). Did you try a single solve call in which the
>>>> right-hand side is an identity matrix? On looking at the Armadillo
>>>> sources it seems that it should call zgesv which is included in R's
>>>> subset of the Lapack routines.
>>>
>>> It seems curious, looking at the bestiary of LAPACK functions, that so
>>> many of them seem to perform a similar task. I wonder what are the
>>> practical downsides of using a code that solves AX=I rather than one
>>> inverting A. Also, when I replaced inv() by solve() and pinv(),
>>> R-forge still failed to build complaining that zgesv_ was not present.
>>> I assumed it was used by solve(), but perhaps it was pinv(). I'll give
>>> it a try anyway, but I'd rather hope we can figure out a less ad hoc
>>> solution.
>>
>> Once upon a time we actually counted the number of floating point
>> operations that were needed to perform a particular calculation, which
>> is the reason for all the special-case code in Lapack. The difference
>> between using the LU decomposition of a matrix to solve a system in
>> which the right hand side is an identity, versus a special-purpose
>> piece of code like zgetri is that zgetri can take advantage of the
>> fact that the identity matrix is diagonal, thereby saving a relatively
>> small number of operations.
>
> That makes sense, thanks.
>
>>
>> I'm not sure why there should be a complaint about zgesv_ not being
>> available. It's there in the R sources (src/modules/lapack/cmplx.f
>> starting at line 3944). That routine is trivial because it just
>> checks its arguments then calls zgetrf and zgetrs.
>>>
>
> Sorry, I meant zgels_ not zgesv_ (who came up with those names?!).
Fortran programmers. Just as the founders of Microsoft decided that
no one would ever have more than some amount of memory (128K, IIRC)
those defining the Fortran specification in the 1950's - 60's declared
that you would never need more than 6 upper-case characters in an
identifier.
You parse the name according to the first letter indicates the storage
type (z for double precision complex), the next two for specific
characteristics of the matrix (ge -> general, tr -> triangular, sy ->
symmetric, po -> symmetric and positive definite, ...) and the
remaining two or three for the operation (trf -> triangular
factorization, sl -> solution of linear systems, ls -> least squares
solution).
You should only have a request for zgels if you are trying to get
least squares solutions.
The complex Lapack routines that you can count on being available are
cmplx.f:96: SUBROUTINE ZBDSQR( UPLO, N, NCVT, NRU, NCC, D, E, VT, LDVT, U,
cmplx.f:838: SUBROUTINE ZDRSCL( N, SA, SX, INCX )
cmplx.f:952: SUBROUTINE ZGEBAK( JOB, SIDE, N, ILO, IHI, SCALE, M, V, LDV,
cmplx.f:1141: SUBROUTINE ZGEBAL( JOB, N, A, LDA, ILO, IHI, SCALE, INFO )
cmplx.f:1471: SUBROUTINE ZGEBD2( M, N, A, LDA, D, E, TAUQ, TAUP,
WORK, INFO )
cmplx.f:1721: SUBROUTINE ZGEBRD( M, N, A, LDA, D, E, TAUQ, TAUP,
WORK, LWORK,
cmplx.f:1989: SUBROUTINE ZGECON( NORM, N, A, LDA, ANORM, RCOND,
WORK, RWORK,
cmplx.f:2182: SUBROUTINE ZGEEV( JOBVL, JOBVR, N, A, LDA, W, VL,
LDVL, VR, LDVR,
cmplx.f:2578: SUBROUTINE ZGEHD2( N, ILO, IHI, A, LDA, TAU, WORK, INFO )
cmplx.f:2726: SUBROUTINE ZGEHRD( N, ILO, IHI, A, LDA, TAU, WORK,
LWORK, INFO )
cmplx.f:2999: SUBROUTINE ZGELQ2( M, N, A, LDA, TAU, WORK, INFO )
cmplx.f:3122: SUBROUTINE ZGELQF( M, N, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:3317: SUBROUTINE ZGEQP3( M, N, A, LDA, JPVT, TAU, WORK,
LWORK, RWORK,
cmplx.f:3610: SUBROUTINE ZGEQR2( M, N, A, LDA, TAU, WORK, INFO )
cmplx.f:3731: SUBROUTINE ZGEQRF( M, N, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:3927: SUBROUTINE ZGESV( N, NRHS, A, LDA, IPIV, B, LDB, INFO )
cmplx.f:4034: SUBROUTINE ZGESVD( JOBU, JOBVT, M, N, A, LDA, S, U,
LDU, VT, LDVT,
cmplx.f:7636: SUBROUTINE ZGETF2( M, N, A, LDA, IPIV, INFO )
cmplx.f:7784: SUBROUTINE ZGETRF( M, N, A, LDA, IPIV, INFO )
cmplx.f:7943: SUBROUTINE ZGETRS( TRANS, N, NRHS, A, LDA, IPIV, B,
LDB, INFO )
cmplx.f:8092: SUBROUTINE ZHEEV( JOBZ, UPLO, N, A, LDA, W, WORK,
LWORK, RWORK,
cmplx.f:8310: SUBROUTINE ZHETD2( UPLO, N, A, LDA, D, E, TAU, INFO )
cmplx.f:8568: SUBROUTINE ZHETRD( UPLO, N, A, LDA, D, E, TAU,
WORK, LWORK, INFO )
cmplx.f:8864: SUBROUTINE ZHSEQR( JOB, COMPZ, N, ILO, IHI, H, LDH,
W, Z, LDZ,
cmplx.f:9259: SUBROUTINE ZLABRD( M, N, NB, A, LDA, D, E, TAUQ,
TAUP, X, LDX, Y,
cmplx.f:9587: SUBROUTINE ZLACGV( N, X, INCX )
cmplx.f:9647: SUBROUTINE ZLACN2( N, V, X, EST, KASE, ISAVE )
cmplx.f:9868: SUBROUTINE ZLACPY( UPLO, M, N, A, LDA, B, LDB )
cmplx.f:10004: SUBROUTINE ZLAHQR( WANTT, WANTZ, N, ILO, IHI, H,
LDH, W, ILOZ,
cmplx.f:10474: SUBROUTINE ZLAHR2( N, K, NB, A, LDA, TAU, T, LDT, Y, LDY )
cmplx.f:10714: SUBROUTINE ZLAHRD( N, K, NB, A, LDA, TAU, T, LDT, Y, LDY )
cmplx.f:11678: SUBROUTINE ZLAQP2( M, N, OFFSET, A, LDA, JPVT,
TAU, VN1, VN2,
cmplx.f:11857: SUBROUTINE ZLAQPS( M, N, OFFSET, NB, KB, A, LDA,
JPVT, TAU, VN1,
cmplx.f:12123: SUBROUTINE ZLAQR0( WANTT, WANTZ, N, ILO, IHI, H,
LDH, W, ILOZ,
cmplx.f:12724: SUBROUTINE ZLAQR1( N, H, LDH, S1, S2, V )
cmplx.f:12821: SUBROUTINE ZLAQR2( WANTT, WANTZ, N, KTOP, KBOT,
NW, H, LDH, ILOZ,
cmplx.f:13258: SUBROUTINE ZLAQR3( WANTT, WANTZ, N, KTOP, KBOT,
NW, H, LDH, ILOZ,
cmplx.f:13706: SUBROUTINE ZLAQR4( WANTT, WANTZ, N, ILO, IHI, H,
LDH, W, ILOZ,
cmplx.f:14308: SUBROUTINE ZLAQR5( WANTT, WANTZ, KACC22, N, KTOP,
KBOT, NSHFTS, S,
cmplx.f:15117: SUBROUTINE ZLARF( SIDE, M, N, V, INCV, TAU, C, LDC, WORK )
cmplx.f:15237: SUBROUTINE ZLARFB( SIDE, TRANS, DIRECT, STOREV, M,
N, K, V, LDV,
cmplx.f:15845: SUBROUTINE ZLARFG( N, ALPHA, X, INCX, TAU )
cmplx.f:15990: SUBROUTINE ZLARFT( DIRECT, STOREV, N, K, V, LDV,
TAU, T, LDT )
cmplx.f:16214: SUBROUTINE ZLARFX( SIDE, M, N, V, TAU, C, LDC, WORK )
cmplx.f:16855: SUBROUTINE ZLARTG( F, G, CS, SN, R )
cmplx.f:17050: SUBROUTINE ZLASCL( TYPE, KL, KU, CFROM, CTO, M, N,
A, LDA, INFO )
cmplx.f:17317: SUBROUTINE ZLASET( UPLO, M, N, ALPHA, BETA, A, LDA )
cmplx.f:17431: SUBROUTINE ZLASR( SIDE, PIVOT, DIRECT, M, N, C, S, A, LDA )
cmplx.f:17794: SUBROUTINE ZLASSQ( N, X, INCX, SCALE, SUMSQ )
cmplx.f:17895: SUBROUTINE ZLASWP( N, A, LDA, K1, K2, IPIV, INCX )
cmplx.f:18014: SUBROUTINE ZLATRD( UPLO, N, NB, A, LDA, E, TAU, W, LDW )
cmplx.f:18293: SUBROUTINE ZLATRS( UPLO, TRANS, DIAG, NORMIN, N,
A, LDA, X, SCALE,
cmplx.f:19172: SUBROUTINE ZPOTF2( UPLO, N, A, LDA, INFO )
cmplx.f:19346: SUBROUTINE ZPOTRF( UPLO, N, A, LDA, INFO )
cmplx.f:19532: SUBROUTINE ZROT( N, CX, INCX, CY, INCY, C, S )
cmplx.f:19623: SUBROUTINE ZSTEQR( COMPZ, N, D, E, Z, LDZ, WORK, INFO )
cmplx.f:20126: SUBROUTINE ZTRCON( NORM, UPLO, DIAG, N, A, LDA, RCOND, WORK,
cmplx.f:20330: SUBROUTINE ZTREVC( SIDE, HOWMNY, SELECT, N, T,
LDT, VL, LDVL, VR,
cmplx.f:20716: SUBROUTINE ZTREXC( COMPQ, N, T, LDT, Q, LDQ, IFST,
ILST, INFO )
cmplx.f:20878: SUBROUTINE ZTRTRS( UPLO, TRANS, DIAG, N, NRHS, A,
LDA, B, LDB,
cmplx.f:21026: SUBROUTINE ZUNG2L( M, N, K, A, LDA, TAU, WORK, INFO )
cmplx.f:21154: SUBROUTINE ZUNG2R( M, N, K, A, LDA, TAU, WORK, INFO )
cmplx.f:21284: SUBROUTINE ZUNGBR( VECT, M, N, K, A, LDA, TAU,
WORK, LWORK, INFO )
cmplx.f:21529: SUBROUTINE ZUNGHR( N, ILO, IHI, A, LDA, TAU, WORK,
LWORK, INFO )
cmplx.f:21694: SUBROUTINE ZUNGL2( M, N, K, A, LDA, TAU, WORK, INFO )
cmplx.f:21830: SUBROUTINE ZUNGLQ( M, N, K, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:22045: SUBROUTINE ZUNGQL( M, N, K, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:22267: SUBROUTINE ZUNGQR( M, N, K, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:22483: SUBROUTINE ZUNGR2( M, N, K, A, LDA, TAU, WORK, INFO )
cmplx.f:22617: SUBROUTINE ZUNGRQ( M, N, K, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:22840: SUBROUTINE ZUNGTR( UPLO, N, A, LDA, TAU, WORK, LWORK, INFO )
cmplx.f:23024: SUBROUTINE ZUNM2R( SIDE, TRANS, M, N, K, A, LDA,
TAU, C, LDC,
cmplx.f:23225: SUBROUTINE ZUNMBR( VECT, SIDE, TRANS, M, N, K, A,
LDA, TAU, C,
cmplx.f:23513: SUBROUTINE ZUNML2( SIDE, TRANS, M, N, K, A, LDA,
TAU, C, LDC,
cmplx.f:23718: SUBROUTINE ZUNMLQ( SIDE, TRANS, M, N, K, A, LDA,
TAU, C, LDC,
cmplx.f:23985: SUBROUTINE ZUNMQR( SIDE, TRANS, M, N, K, A, LDA,
TAU, C, LDC,
If you look at the definition of zgels you will see that for least
squares solutions it calls ZGEQRF, ZUNMQR and ZTRTRS so you just need
to set up those calls instead (although admittedly it's a royal PITA
to get the Fortran calls set up properly).
An alternative would be to leave a copy of the Fortran sources for
ZGELS in the src directory for your package but that may croak on
trying to compile the whole routine, which allows for both
over-determined and under-determined systems.
> You can see the error here:
> https://r-forge.r-project.org/R/?group_id=160&log=build_src&pkg=cda&flavor=patched
>
> Best,
>
> baptiste
>
>>> Thanks,
>>>
>>> baptiste
>>>
>>>>
>>>>
>>>>> One option that I'd like to consider is whether the appropriate LAPACK
>>>>> routines could be wrapped and shipped in a separate package
>>>>> (discontinued rblas could provide a good starting point). Sadly, I
>>>>> know nothing about static/dynamic libraries and all this business..
>>>>>
>>>>> Thanks,
>>>>>
>>>>> baptiste
>>>>>
>>>>>>
>>>>>>> > | Sorry for being dense, I don't know anything about linking R to
>>>>>>
>>>>>>> > | external dependencies.
>>>>>>> >
>>>>>>> > It can be done; there are many examples -- for example every package
>>>>>>> > using
>>>>>>> > the GSL.
>>>>>>>
>>>>>>> I just checked how RcppGSL does it, and well, this configure magic is
>>>>>>> way above my head.
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> baptiste
>>>>>>> _______________________________________________
>>>>>>> Rcpp-devel mailing list
>>>>>>> Rcpp-devel at lists.r-forge.r-project.org
>>>>>>> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
>>>>>>
>>>>>
>>>>
>>>
>>
>
More information about the Rcpp-devel
mailing list