[Pw_forum] Libraries on Blue Gene/L

[zhaofeng at Princeton.EDU]

  BODY { font-family:Arial, Helvetica, sans-serif;font-size:12px;
}Dear members,
    Currently I am porting CP code on BlueGene/L. I have met some
problems that are 
 related with libraries.
    The compilation and linking options are shown below
  ./configure MPIF90=mpxlf90 CC=mpxlc F77=mpxlf LD=mpxlf90 
 LDFLAGS="-L/bgl/BlueLight/ppcfloor/bglsys/lib -lmpich.rts
-lmsglayer.rts -lrts.rts -
 ldevices.rts"
 BLAS_LIBS="/bgl/local/lib/libblas440.a" 
 LAPACK_LIBS="/bgl/local/lib/liblapack440.a 
/bgl/local/lib/scalapack/libscalapack.a" 
 CFLAGS="-qarch=440 -qmaxmem=64000" 
 FFLAGS="-qalias=noaryovrlp -I../include -O3 -qstrict -qarch=440d
-qtune=440 -qsuffix=cpp=f90 -qdpc -qalias=nointpt" 
 FFT_LIBS="/bgl/local/fftw-3.1.2/lib/libfftw3.a"
    There is not problem in obtaining cp.x when "make cp". We test 64
water using 256 
 processors(128 nodes) with number of task group equal to 2, and the
speed of cp is 6 sec 
 per step, which is good(If some of you have better suggestions for
configuration which can lead to good performance, that will be so
great). 
    But we need to us cp-wf as well and find that wannier part takes
also 6 sec per step, which is really 
 unacceptable.We look into it and find that there are calls of DGEMM
and ZHPEV which are quite time consuming(in wf.f90). For instance,
ZHPEV takes 1.9sec per step and ZGEMM is 0.14 sec per step. Since
ZGEMM and ZHPEV are both the calls in the library of LAPACK, we
conclude it is because of bad choice of lapack library, is it
correct?
   Currently the awkward situation is that there aren't ESSL
libraries on our Blue Gene, but some others provided by an IBM'er as
a part of the original training class. So I have little to do with
this lapack library. And I have also searched our forum content a
little bit and found few related result. I am not sure whether it is
enough to choose the correct lapack library. Or we need a complete
ESSL library so that the computing time of ZGEMM and ZHPEV can be
decreased a lot. If it is the former case, can anyone who has lapack
library of ESSL shares with me?
   Thank you in advance.
 best
 zhaofeng li
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.democritos.it/pipermail/pw_forum/attachments/20081105/e699ae07/attachment.htm