[Rcpp-devel] Patch for using Rcpp with Clang + libc++ AND Intel icpc C++11
Yan Zhou
zhouyan at me.com
Sat Dec 1 16:49:02 CET 2012
Hi Dirk,
Thanks for apply the patch so quick. I hope it does not break anything works perfectly before. Anyway I am sure if it does, it will be corrected before next release.
As for the bigger change, I can start working on it maybe next week. This weekend is not too good for me either. It will take some time and I shall approach it more careful. We don't want to break things already work!
As for c++11, I totally agree that we don't push for it. After all it is still very young. If someone like me want to use it, then he shall be able to build R himself and able to deal with such small issues.
About timing, I guess you mean the timing of R when using different compilers instead of some timing of Rcpp. Compiler optimization is always marginal (if not then the code has some serious problem, which R does not have). Intel compiler can be somehow faster by at most 20% from my experience. However the blas come with MKL is significant faster than other implementations. Clang has a performance roughly the same as GCC. However it provides far better diagnostic messages, especially for template. And it compiles faster. Also it does not accept many ill formed code as GCC does, nor does it reject well formed code as MSVC often do. In general I prefer develop in clang and deploy the final computation task with intel icpc
An unrelated side note about icc. In terms of compiler optimization, it is not much better than recent gcc and clang these days. However it does provide a significant advantage. Intel compiler does not use system libm, instead they use Intel's own math library which can be much faster than the glic or eglic used in Linux. I once observed 100% faster results because heavy use of three function, lgamma, exp and log. That was a Monte Carlo simulation and the 90% of time it was computing some likelihood.
Best,
Yan
On Dec 1, 2012, at 3:23 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
>
> Hi Yan,
>
> Thanks for such a detailed response -- much appreciated. Your patch is in SVN
> now, by the way.
>
> On 1 December 2012 at 15:02, Yan Zhou wrote:
> | Hi Dirk,
> |
> | Thanks for the quick reply.
> |
> |
> | > Nice, that does look indeed clean and proper. Can you tell me a bit more
> | > about the setup you have use:
> | >
> | > -- operating system and version
> | I have tested the patch on Mac OS X 10.8.2 and Ubuntu 12.10. R is built from scratch by myself using the compilers of concern instead of the official binaries.
>
> Great. We do need help with OS X, as "the world" needs help with OS X and the
> mess over the silly standoff with the FSF. Getting clang++ ready will help.
>
> I run more Ubuntu than Debian these days, usually, but I know that over on
> the Debian side it is pretty easy to get prebuilt g++ (now up to 4.8 in the
> experimental repo) and 3.2 for clang. And I do keep chroots around for quick
> tests.
>
> | > -- compiler(s) and version(s), at least clang++ and intel's icpc (wasn't it
> | > called icc ?)
> | Clang is clang 3.1 (though branded as clang 4.1 by Apple, it is actually LLVM 3.1) on Mac OS X, and the SVN version on Linux. Intel icpc is tested with, 12.0, 12.1 and 13.0 on Linux. icpc is the C++ compiler, icc is the C compiler. Just like g++ is actually gcc with another name.
>
> Right, thanks.
>
> | > -- where libc++ came from
> | libc++ can be obtained from http://libcxx.llvm.org
>
> I think it is in Ubuntu now too my apt-cache foo now fails me on the embedded
> regexp in libc++ ... :-/
>
> | It also comes with Apple Xcode 4.2 or later (maybe 4.3, not sure the oldest version come with libc++, I am using Xcode 4.5.2). It is a new standard C++ library implementation. It can only be used with clang at the time of writing. It is also part of the LLVM project, which clang belongs too. On Mac OS X, and FreeBSD, there have been an effort to replace the GCC tool chain with LLVM tool chain for quite a few years, mainly due to GCC 4.3 or later's GPL3 status and other technical considerations. Anyway, on Mac OS X and FreeBSD, the official build of GCC is 4.2, newer versions are not officially supported anymore. Currently Apple ship LLVM-GCC as a backward compatibility solution in Xcode. In the near future Apple will stop shipping this GCC 4.2 variant entirely, and leave the platform with only Clang. On FreeBSD, Clang is also set to replace GCC as the default tool chain in the next version.
>
> We are aware of this, and it is a constant issue.
>
> | libc++ is C++98 conforming and at the time of writing the only 100% feature complete C++11 implementation of standard library. But it does not come with TR1 mainly because when the project took off, C++11 is already on the horizon and they decided to support C++11 directly. Also this the only standard library one can use on Mac OS X if C++11 is desired, or otherwise one need to build a GCC from scratch themselves.
> |
> | The situation with Clang and Intel icpc is that, they do not bind to a specific version of standard library. They are just compilers. Unlike GCC, say we are using GCC 4.7, it is not possible for G++ to include headers from or link to a version of libstdc++ other than the one distributed with GCC 4.7 unless we do something tricky. With Intel icpc, at least on Linux and Mac OS X, icpc find whatever version of GCC on the system. For example, if the system comes with GCC 4.2, then icpc use libstdc++ 4.2. Even the version of icpc supports a lot of new C++11 features, you can not use them at all. To use C++11 features, one not only need a new version of icpc, but also a new version of GCC, which comes with a new version of libstdc++. So the situation is a mess. Clang is similar, it can use whatever version of libstdc++ or libc++ installed on the system.
> | >
> | > -- in case you timed this, what performance differences do you see?
> | I am not sure what kind of timing are suggested
>
> Any, really :) R itself has regression tests for which you can turn on
> timing. And in industry, the only time I meet people talking intel icc are
> usually those who want to shave a micro or nanosecond here or there :)
>
> You are more concerned with buildability first?
>
> | > This is very nice news for less-standard systems. Just so that I remain on
> | > the same page, these do still generate gcc-compatible code so what you
> | > generate does in fact interoperates with code on your system which may have
> | > been built by gcc, correct?
> | Almost YES. Clang and icpc generate GCC compatible binary code. But libc++ is not always libstdc++ compatible at binary level. They are mostly compatible except for some IO facilities. As long as <iostream> etc. are not involved, they are binary compatible. I don't think it will be an issue. Because if someone is going to built Rcpp with libc++, then the compiler flag -stdlib=libc++ shall be provided when buidling R itself, or he/she shall always remember to use this flag when incorporate other C++ code. The situation I am trying to solve is that, both Intel icc and clang can built R and pass all of R's test suites. It is better to have their corresponding C++ compiler able to build a popular R package. If a compiler does not built R at all, e.g., MSVC, we don't have to worry about if Rcpp works with it. In the case of clang++ and icpc, I think the cost is small, while the benefits is one can use many of their C++11 code while using Rcpp at the same time. We cannot possibly support all compilers that can build R. Besides building R only need C compiler while the CXXFLAGS are only captured by R build script for future use (for example when install.packages("Rcpp")). However, if it is not too much a trouble, support some C++11 implementation can be nice.
>
> Absolutely.
>
> Our attitude has been that as along as CRAN does not let us set C++11 as a
> build option for Rcpp, there is little point in pushing for it.
>
> | The current patch won't pass all unit test, because some unit test use the tr1:: namespace explicitly, while the solution is to have Clang and other compilers use std::unordered_map etc while it is available. If I may suggest, if we are willing to make some bigger change, we can solve the situation of compilers other than GCC more elegant. For example, we can have the following in RcppCommon.h, it may make the RcppCommon.h looks a little more tedious, but shall hopefully cleanup all the mess. My patch in the last email only touch RcppCommon.h and work around the fact that other headers like sugar/sets.h also test and set macros about std::unordered_set or tr1::unordered_set.
> |
> | #include <cmath>
> | #if defined(__INTEL_COMPILER)
> | #if defined(__GLIBCXX__) && __GLIBCXX__ >= 20090421
> | #define HAS_TR1_UNORDERED_MAP
> | #define HAS_TR1_UNORDERED_SET
> | #endif
> |
> | #if __cplusplus >= 201103L
> | #if defined(__GLIBCXX__) && __GLIBCXX__ >= 20090421
> | #define HAS_CXX11_UNORDERED_MAP
> | #define HAS_CXX11_UNORDERED_SET
> | #endif
> | #elif defined(__clang__)
> | #if __has_include<tr1/unordered_map>
> | #define HAS_TR1_UNORDERED_MAP
> | #endif
> |
> | #if __has_include<tr1/unordered_set>
> | #define HAS_TR1_UNORDERED_SET
> | #endif
> |
> | #if __cplusplus >= 201103L
> | #if __has_include<unordered_map>
> | #define HAS_CXX11_UNORDERED_MAP
> | #endif
> |
> | #if __has_include<unordered_set>
> | #define HAS_CXX11_UNORDERED_SET
> | #endif
> | #endif
> | #elif defined(__GNUC__)
> | // test GCC the last, as both clang and intel defines __GNUC__
> | #define GCC_VERSION \
> | (__GNUC__ * 10000 + __GNUC_MINOR__ * 100 + __GNUC_PATCHLEVEL__)
> | #if GCC_VERSION >= 40201
> | #define HAS_TR1_UNORDERED_MAP
> | #define HAS_TR1_UNORDERED_SET
> | #endif
> |
> | #ifdef __GXX_EXPERIMENTAL_CXX0X__
> | #if GCC_VERSION >= 40400 // maybe another earlier version also works
> | #define HAS_CXX11_UNORDERED_MAP
> | #define HAS_CXX11_UNORDERED_SET
> | #endif
> | #endif
> |
> | #endif
> |
> | #if defined(HAS_CXX11_UNORDERED_MAP)
> | #include <unordered_map>
> | #define RCPP_MAP std::unordered_map
> | #elif defined(HAS_TR1_UNORDERED_MAP)
> | #include <tr1/unordered_map>
> | #define RCPP_MAP tr1::unordered_map
> | #else
> | #include <map>
> | #define RCPP_MAP tr1::map
> | #endif
> |
> | #if defined(HAS_CXX11_UNORDERED_SET)
> | #include <unordered_set>
> | #define RCPP_SET std::unordered_set
> | #elif defined(HAS_TR1_UNORDERED_SET)
> | #include <tr1/unordered_set>
> | #define RCPP_SET tr1::unordered_set
> | #else
> | #include <set>
> | #define RCPP_SET tr1::set
> | #endif
> |
> | // The rest of the library does not need to test __cplusplus etc.
> | // They only need to know RCPP_MAP and RCPP_SET
> | // All test are taken care inside RcppCommon.h
> |
> | With the above additions, other headers and source files hall not test __cplusplus >= 201103L etc. They shall just used RCPP_MAP and RCPP_SET whenever they need them. And if later we find that some compilers version are not properly tested by the above code, we only need to fix it in RcppCommon.h without any change to other parts of the library.
>
> I am on board. RcppCommon.h is the place to do this, and we can (and are)
> pushing the defines down to good use. If you can help with testing I am
> absolutely in favour of supporting this.
>
> This weekend is bad, but let's keep this on the agenda. Please ping me again
> if you do not hear from me.
>
> Thanks again for all of this -- much appreciated!
>
> Dirk
>
>
>
> | Best,
> |
> | Yan Zhou
> |
> | >
> | > Dirk
> | >
> | > |
> | > |
> | > | Best,
> | > |
> | > | Yan Zhou
> | > |
> | > | On Dec 1, 2012, at 12:44 PM, Yan Zhou <zhouyan at me.com> wrote:
> | > |
> | > | > Dear Dirk,
> | > | >
> | > | > In addition to my last email which provides a path for clang++ with libc++, I updated the patch to also fix problems with intel icpc in C++11 mode.
> | > | >
> | > | > In the RcppCommon.h, there is comments says that Intel ICPC does not support C++11 or TR1. That is not entirely true. Intel compiler does not come with its own standard library. On Linux and Mac OS X it use the libstdc++ come with the system. On Windows it may use others. The current test has a problem when an Intel C++ compiler is used in C++11 mode. In that case, headers like sugar/sets.h test the C++11 macro and conclude it shall define SET to std::unordered_set. However, C++11 <unordered_set> is not included. So a compiler error happens.
> | > | >
> | > | > In general, I found Rcpp does not work with compilers in C++11 mode. The new path, in addition to the own test Clang with libc++, also test the followings,
> | > | > 1. If Intel compiler is used, test is it is used with libstdc++. If it is not, then we undef HAS_TR1_... etc. To test if libstdc++ is used, we need to at least include one standard library header before the testing, so I included <cmath>, which shall be harmless
> | > | > 2. After trying to include <tr1/unodered_map> etc, we also test if we are using C++11. If it is, then test if we are using libstdc++ (either gcc, clang, intel etc). If it is case, and the libstdc++ is recent enough, we include <unordered_map> and <unordered_set>. Otherwise, if we are using C++11 in Clang with libc++, we also included <unodered_map> etc.
> | > | >
> | > | > After this patch, Rcpp shall work seamlessly with GCC, Intel and Clang, in both C++98 and C++11 modes.
> | > | > <RcppCommon.h.diff>
> | > | >
> | > | > Best,
> | > | >
> | > | > Yan Zhou
> | > | >
> | > | >
> | > | >
> | > | > On Dec 1, 2012, at 12:12 PM, Yan Zhou <zhouyan at me.com> wrote:
> | > | >
> | > | >> Hi Dirk,
> | > | >>
> | > | >> Rcpp cannot be compiled with clang++ with libc++, even clang++ provides very good standard conforming in both C++98 and C++11 mode, and libc++ provides 100% C++98/11 features. The problems is Rcpp's use of TR1 instead of C++11, and does not perform some compiler checks properly. I made a small patch to the include/RcppCommon.h header, which makes Rcpp works with Clang++ and libc++ in C++11 mode.
> | > | >>
> | > | >> To summary the change, when macro __clang__ is defined, the header use clang's __has_include to check if <tr1/unordered_map> and <tr1/unordered_set> is present, if not, it undef HAS_TR1_UNORDERED_MAP etc. In addition, if __has_include<unordered_map> is tested to be true while HAS_TR1_... etc are not true, the C++11 header is included. So headers like sugar/sets.h can use C++11 header instead of TR1, since they already test __cplusplus >= 201103L.
> | > | >>
> | > | >> With this patch, nothing already works will be broken. This patch only affects clang++ with libc++ situation. Using clang++ with libstdc++ the situation will be exactly the same as before.
> | > | >>
> | > | >> At the end of the email is the path, it is also attached as a diff file
> | > | >>
> | > | >> Best,
> | > | >>
> | > | >> Yan Zhou
> | > | >> <RcppCommon.h.diff>
> | > | >>
> | > | >> --- Rcpp/inst/include/RcppCommon.h 2012-11-23 01:07:34.000000000 +0000
> | > | >> +++ ../Downloads/Rcpp/inst/include/RcppCommon.h 2012-12-01 11:46:48.000000000 +0000
> | > | >> @@ -107,6 +107,20 @@
> | > | >> // #endif
> | > | >> // #endif
> | > | >>
> | > | >> +#ifdef __clang__
> | > | >> + #if !__has_include(<tr1/unordered_map>)
> | > | >> + #undef HAS_TR1
> | > | >> + #undef HAS_TR1_UNORDERED_MAP
> | > | >> + #endif
> | > | >> + #if !__has_include(<tr1/unordered_set>)
> | > | >> + #undef HAS_TR1
> | > | >> + #undef HAS_TR1_UNORDERED_SET
> | > | >> + #endif
> | > | >> + #if !__has_feature(cxx_variadic_templates)
> | > | >> + #undef HAS_VARIADIC_TEMPLATES
> | > | >> + #endif
> | > | >> +#endif
> | > | >> +
> | > | >> #ifdef __INTEL_COMPILER
> | > | >> // This is based on an email by Alexey Stukalov who tested
> | > | >> // Intel Compiler 12.0 and states that is does support Cxx0x
> | > | >> @@ -149,6 +163,15 @@
> | > | >> #include <tr1/unordered_set>
> | > | >> #endif
> | > | >>
> | > | >> +#ifdef __clang__
> | > | >> + #if !defined(HAS_TR1_UNORDERED_MAP) && __has_include(<unordered_map>)
> | > | >> + #include <unordered_map>
> | > | >> + #endif
> | > | >> + #if !defined(HAS_TR1_UNORDERED_SET) && __has_include(<unordered_set>)
> | > | >> + #include <unordered_set>
> | > | >> + #endif
> | > | >> +#endif
> | > | >> +
> | > | >> std::string demangle( const std::string& name) ;
> | > | >> #define DEMANGLE(__TYPE__) demangle( typeid(__TYPE__).name() ).c_str()
> | > | >>
> | > | >>
> | > | >> _______________________________________________
> | > | >> Rcpp-devel mailing list
> | > | >> Rcpp-devel at lists.r-forge.r-project.org
> | > | >> https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
> | > | >
> | > | > _______________________________________________
> | > | > Rcpp-devel mailing list
> | > | > Rcpp-devel at lists.r-forge.r-project.org
> | > | > https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
> | > |
> | > |
> | > | ----------------------------------------------------------------------
> | > | _______________________________________________
> | > | Rcpp-devel mailing list
> | > | Rcpp-devel at lists.r-forge.r-project.org
> | > | https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/rcpp-devel
> | > --
> | > Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com
> |
>
> --
> Dirk Eddelbuettel | edd at debian.org | http://dirk.eddelbuettel.com
More information about the Rcpp-devel
mailing list