[Pw_forum] Libraries on Blue Gene/L

[Axel Kohlmeyer]

hi,

FWIW, the contact for optimized libraries for BG/L is alessandro curioni
at IBM research in zurich <cur at zurich.ibm.com>.

for optimal performance you do not need ESSL, but a specially tuned
LAPACK and mostly BLAS (i.e. DGEMM and related).

On Wed, Nov 5, 2008 at 12:18 PM,  <zhaofeng at princeton.edu> wrote:
> Dear members,
>
>    Currently I am porting CP code on BlueGene/L. I have met some problems
> that are
> related with libraries.
>    The compilation and linking options are shown below
>  ./configure MPIF90=mpxlf90 CC=mpxlc F77=mpxlf LD=mpxlf90
> LDFLAGS="-L/bgl/BlueLight/ppcfloor/bglsys/lib -lmpich.rts -lmsglayer.rts
> -lrts.rts -
> ldevices.rts"
> BLAS_LIBS="/bgl/local/lib/libblas440.a"
> LAPACK_LIBS="/bgl/local/lib/liblapack440.a
> /bgl/local/lib/scalapack/libscalapack.a"
> CFLAGS="-qarch=440 -qmaxmem=64000"
> FFLAGS="-qalias=noaryovrlp -I../include -O3 -qstrict -qarch=440d -qtune=440
> -qsuffix=cpp=f90 -qdpc -qalias=nointpt"
> FFT_LIBS="/bgl/local/fftw-3.1.2/lib/libfftw3.a"
>
>    There is not problem in obtaining cp.x when "make cp". We test 64 water
> using 256
> processors(128 nodes) with number of task group equal to 2, and the speed of
> cp is 6 sec
> per step, which is good(If some of you have better suggestions for
> configuration which can lead to good performance, that will be so great).
>    But we need to us cp-wf as well and find that wannier part takes also 6
> sec per step, which is really
> unacceptable.We look into it and find that there are calls of DGEMM and
> ZHPEV which are quite time consuming(in wf.f90). For instance, ZHPEV takes
> 1.9sec per step and ZGEMM is 0.14 sec per step. Since ZGEMM and ZHPEV are
> both the calls in the library of LAPACK, we conclude it is because of bad

no, ZGEMM is part of BLAS and not LAPACK.

nevertheless, this does not necessarily mean, that those are the culprit.
it could just be, that this part of the code is not as parallel as the rest
and thus is doing a lot of redundant work. i saw this quite often with
wannier implementations in different codes.

cheers,
   axel.

> choice of lapack library, is it correct?
>   Currently the awkward situation is that there aren't ESSL libraries on our
> Blue Gene, but some others provided by an IBM'er as a part of the original
> training class. So I have little to do with this lapack library. And I have
> also searched our forum content a little bit and found few related result. I
> am not sure whether it is enough to choose the correct lapack library. Or we
> need a complete ESSL library so that the computing time of ZGEMM and ZHPEV
> can be decreased a lot. If it is the former case, can anyone who has lapack
> library of ESSL shares with me?
>   Thank you in advance.
>
> best
> zhaofeng li
>
>
>
> _______________________________________________
> Pw_forum mailing list
> Pw_forum at pwscf.org
> http://www.democritos.it/mailman/listinfo/pw_forum
>
>



-- 
=======================================================================
Axel Kohlmeyer   akohlmey at cmm.chem.upenn.edu   http://www.cmm.upenn.edu
  Center for Molecular Modeling   --   University of Pennsylvania
Department of Chemistry, 231 S.34th Street, Philadelphia, PA 19104-6323
tel: 1-215-898-1582,  fax: 1-215-573-6233,  office-tel: 1-215-898-5425
=======================================================================
If you make something idiot-proof, the universe creates a better idiot.