[Pw_forum] scaling on clusters with different communication types

[Axel Kohlmeyer]

On Fri, 8 Dec 2006, Kara, Abdelkader wrote:

AK> Dear all,
AK> 
AK> Greatings.
AK> 
AK> I will appreciate it very much if you can share with me your experience
AK> of running pwscf on clusters with different communication hardware.
AK> I am interested in the scaling with the number of CPU's for the following 3
AK> different communication types:
AK> 1)gigabit ethernet
AK> 2) myrinet
AK> 3) InfiniBand

scaling depends a lot on the kind of jobs you intend to run.

pw.x scales almost independendly and very well across NEB 
images and k-points even with gigabit ethernet. on top of
that you can parallelize across g-space, which is much more
demanding in terms of communication bandwidth and latency.
in this case scaling across gigabit is limited to a few 
nodes. in-node performance is governed by available memory
bandwidth wich results in hyper-threading being conter-productive,
multi-core cpus having reduced efficiency (depending on job
size, i.e. cache efficiency) and opteron cpus due to dedicated
per-cpu memory busses scaling better than intel (xeon). only
very recent intel (woodcrest) xeon cpus have been demonstrated
to have a somewhat better performance and price/performance ratio.

please note, that these are some general trends observed from
some usage patterns that may not translate to your needs.
also presence of a per-node local scratch area or absence
impacts the performance. using a NFS filesystem for temporary
storage usually results in degraded performance.

basically, the larger your systems and the fewer k-points
you need use, the more important a fast interconnect becomes.
performance between infiniband and myrinet solutions is
more or less equivalent when compared with gigabit. thus
using older/obsolete hardware can be a bargain.

cheers,
     axel.

AK> 
AK> Thank you very much for your input on this matter
AK> 
AK> Kader Kara
AK> 
AK> Physics Department
AK> University of Central Florida
AK> _______________________________________________
AK> Pw_forum mailing list
AK> Pw_forum at pwscf.org
AK> http://www.democritos.it/mailman/listinfo/pw_forum
AK> 

-- 
=======================================================================
Axel Kohlmeyer   akohlmey at cmm.chem.upenn.edu   http://www.cmm.upenn.edu
   Center for Molecular Modeling   --   University of Pennsylvania
Department of Chemistry, 231 S.34th Street, Philadelphia, PA 19104-6323
tel: 1-215-898-1582,  fax: 1-215-573-6233,  office-tel: 1-215-898-5425
=======================================================================
If you make something idiot-proof, the universe creates a better idiot.