<br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Eduardo Ariel Menendez Proupin</b> <<a href="mailto:eariel99@gmail.com">eariel99@gmail.com</a>><br>Date: 2008/5/7<br>
Subject: Re: [Pw_forum] openmp vs mpich performance with MKL 10.x<br>To: Axel Kohlmeyer <<a href="mailto:akohlmey@cmm.chem.upenn.edu">akohlmey@cmm.chem.upenn.edu</a>><br><br><br>Hi,<br><br><br>
<div class="gmail_quote"><div class="Ih2E3d"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
with serial MKL, serial FFTW-2.1.5 and OpenMPI with 4 mpi tasks.<br>
i get a wall time of 12m12s and cpu time of 10m40s.<br></blockquote></div><div>I GET <br>OMP_NUM_THREADS=1<br>MKL_NUM_THREADS=1<br> DFLAGS = -D__INTEL -D__FFTW -D__USE_INTERNAL_FFTW -D__MPI -D__PARA<br>mpiexec -n 4 /home/emenendez/ChemUtils/Espresso/espresso4.0cvs3/bin4/pw.x < <a href="http://cdteo0.2.md.in" target="_blank">cdteo0.2.md.in</a> >> cdteo0.2.md.out<br>
PWSCF : 12m33.73s CPU time, 12m43.31s wall time<br><br> </div><div class="Ih2E3d"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
changing MKL to threaded MKL using 4 threads and 1 mpi task<br>
i get a wall time of 18m8s and cpu time of 28m30s<br>
(which means that roughly 40% of the time the code<br>
was running multi-threaded BLAS/LAPACK).</blockquote></div><div>OMP_NUM_THREADS=4<br>MKL_NUM_THREADS=4<br>mpiexec -n 1 /home/emenendez/ChemUtils/Espresso/espresso4.0cvs3/bin4/pw.x < <a href="http://cdteo0.2.md.in" target="_blank">cdteo0.2.md.in</a> >> cdteo0.2.md.out<br>
PWSCF : 27m40.02s CPU time, 17m 2.73s wall time<br><br> </div><div class="Ih2E3d"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
with serial FFT, threaded MKL using 2 threads and 2 mpi tasks<br>
i get a wall time of 12m45s and cpu time of 14.42s</blockquote></div><div>UNTESTED IN THIS MACHINE, BUT WORSE IN OTHERS.<br></div><div class="Ih2E3d"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
now when i swap the serial FFTW2 against the<br>
intel MKL FFTW2 wrapper i get with 2 threads and 2 MPI tasks<br>
a wall time of 15m2s and a cpu time of 24m11s.</blockquote></div><div> UNTESTED IN THIS MACHINE, BUT WORSE IN OTHERS.</div><div class="Ih2E3d"><div><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
and with 4 threads and 1 MPI task i get<br>
a wall time of 0h19m and a cpu time of 1h 2m</blockquote></div><div><br>OMP_NUM_THREADS=4<br>MKL_NUM_THREADS=4<br>mpiexec -n 1 /home/emenendez/ChemUtils/Espresso/espresso4.0cvs3/bin5/pw.x < <a href="http://cdteo0.2.md.in" target="_blank">cdteo0.2.md.in</a> >> cdteo0.2.md.out<br>
PWSCF : 58m50.42s CPU time, 17m55.90s wall time<br><br></div><div class="Ih2E3d"><div><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
and finally when disabling threading and with<br>
4 MPI tasks i get 12m38 wall time and 11m14s cpu time.</blockquote></div><div> OMP_NUM_THREADS=1<br>MKL_NUM_THREADS=1<br> DFLAGS = -D__INTEL -D__FFTW -D__MPI -D__PARA (using fftw2_intel)<br>mpiexec -n 4 /home/emenendez/ChemUtils/Espresso/espresso4.0cvs3/bin5/pw.x < <a href="http://cdteo0.2.md.in" target="_blank">cdteo0.2.md.in</a> >> cdteo0.2.md.out<br>
PWSCF : 13m 2.54s CPU time, 13m16.11s wall time<br>IT IS WORSE THAN USING THE INTERNAL FFTW.<br></div><div><br>HOWEVER, RUNNING SERIAL<br><br>OMP_NUM_THREADS=4<br>MKL_NUM_THREADS=4<br>DFLAGS = -D__INTEL -D__FFTW<br>
BLAS_LIBS = -lfftw2xf_intel -lmkl_em64t<br>/home/emenendez/ChemUtils/Espresso/espresso4.0cvs3/bin2/pw.x < <a href="http://cdteo0.2.md.in" target="_blank">cdteo0.2.md.in</a> >> cdteo0.2.md.out<br> PWSCF : 36m58.97s CPU time, 11m36.11s wall time<br>
<br><br>OMP_NUM_THREADS=4<br>MKL_NUM_THREADS=4<br>DFLAGS = -D__INTEL -D__FFTW3<br>BLAS_LIBS = -lfftw3xf_intel -lmkl_em64t<br>/home/emenendez/ChemUtils/Espresso/espresso4.0cvs3/bin3/pw.x < <a href="http://cdteo0.2.md.in" target="_blank">cdteo0.2.md.in</a> >> cdteo0.2.md.out<br>
PWSCF : 36m44.64s CPU time, 11m29.59s wall time<br><br> </div><div class="Ih2E3d"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
obviously, switching to the intel fft didn't help.</blockquote></div><div>FOR ME, IT HELPS ONLY WHEN RUNNING SERIAL. <br></div><div class="Ih2E3d"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
your system with many states and only gamma point<br>
is definitely a case that benefits the most from<br>
multi-threaded BLAS/LAPACK.</blockquote></div><div>TYPICAL FOR BO MOLECULAR DYNAMICS. <br><br>I WOULD SAY, AVOID MIXING MPI AND OPENMP. ALSO AVOID INTEL FFTW WRAPPERS WITH MPI, EVEN IF OMP_NUM_THREADS=1.<br>USE THREADED BLAS/LAPACK/FFTW2(3) FOR SERIAL RUNS.<br>
<br>ANYWAY, THE DIFFERENCE BETWEEN THE BEST MPI AND THE BEST OPENMP IS LESS THAN 10% (11m30s vs 12m43s)<br></div><div class="Ih2E3d"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
<br>
i'm curious to learn how these number match up<br>
with your performance measurements.<br>
<br>
cheers,<br>
axel.<br>
<div><br>
<br></div></blockquote></div></div><br><br clear="all"><br>-- <br><font color="#888888">Eduardo Menendez
</font></div><br><br clear="all"><br>-- <br>Eduardo Menendez