<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Yes, but only just in the last few days. I also CC'd this to the PWSCF list.</div><div><br></div><div>I travelled out to a meeting at Argonne's LCF (where the BG is hosted), and was able to begin to work more closely with Nichols Romero on the issue.</div><div><br></div><div>I was able to try the following systems (a pw BOMD run on Li4BN3H10 crystal with an initial temperature and periodic boundaries, no vacuum in the supercell) all run in vn mode.<div><br></div><div>432 atoms (960 electrons) on 1024, 4096, 8192 procs<br><div>576 atoms (1280 electrons) on 1024, 2048, 4096 procs</div><div>864 atoms (1920 electrons) on 1024, 2048, 4096 procs</div><div><br></div><div>What I found is (with with -ntg 8 and -ndiag 1):</div><div><br></div><div>- In all cases, the 1024 proc case was able to get into at the electronic loop and complete at least one SCF step (before the runtime got too long and it was killed)</div><div><br></div><div>- In all cases, the runs on greater than 1024 procs died before outputing SCF step data. The complaint found in the stderr files was the 'from n_plane_waves : error # 1' and 'no PWs!' error. In the 8192 proc case (ntg 64 ndiag 1), the error was 'wrong ngm' - but I think that is actually the same error (data spread too thin) - just manifest in a different way.</div><div><br></div><div>- All of the systems on all processors had the following complaints in the beginning of stdout:</div><div><br></div><div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>Iterative solution of the eigenvalue problem</div><div> <span class="Apple-tab-span" style="white-space: pre; "> </span>Too few electrons for parallel algorithm</div><div> <span class="Apple-tab-span" style="white-space: pre; "> </span>we need at least as many bands as SQRT(nproc)</div><div><br></div><div> <span class="Apple-tab-span" style="white-space: pre; "> </span>a serial algorithm will be used</div><div><br></div><div> <span class="Apple-tab-span" style="white-space: pre; "> </span>Message from routine data_structure:</div><div> <span class="Apple-tab-span" style="white-space: pre; "> </span>some processors have no planes </div><div> <span class="Apple-tab-span" style="white-space: pre; "> </span>Message from routine data_structure:</div><div> <span class="Apple-tab-span" style="white-space: pre; "> </span>some processors have no smooth planes</div><div><br></div><div><span class="Apple-tab-span" style="white-space: pre; "> </span>.... Regardless of if they were able to successfully enter the SCF steps or not. The error about the number of bands I think may be erroneous - but would have to dig through the code a bit more to be sure. </div><div><br></div><div>- ndiag > 1 doesn't work in any case is pointing toward the idea that the parallel orthogonalization isn't working. The error spat out are like this (from the 432 atom case on 1024 procs with -ntg 8 -ndiag 128:</div><div><div>from pdpotf : error # 1</div><div> problems computing cholesky decomposition</div></div><div><br></div><div><br></div><div>I can try a 1008 atom case (2240 electrons) and the above system as a 1152 atom case (2560 electrons) - I have a feeling that may break down because of memory issues even at 1024 procs, but I have a feeling getting to the bottom of why the above cases fail so spectacularly will be required if I want to get to a larger system on more procs. I also haven't touched the taskgroup thing to see what, if any, trouble that is causing. </div></div></div></div><div><br></div><div>I also tried the above with the CVS version with SCALAPACK and seemed to get the same behavior. If I am able to get to run say ~2500-3000 electrons on 4192 cores, I will consider that a win. But right now, it seems that 1024 is about as high as I can go.</div><div><br></div><div>Dave</div><div><br></div><br><div><div>On Feb 11, 2009, at 7:18 AM, Paolo Giannozzi wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi, any news on your BG problem? Paolo<br><br>-- <br>Paolo Giannozzi, Democritos and University of Udine, Italy<br></div></blockquote></div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; font-size: 12px; ">David E. Farrell</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px; ">Post-Doctoral Fellow</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px; ">Department of Materials Science and Engineering</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; font-size: 12px; ">Northwestern University</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; font-size: 12px; ">email: <a href="mailto:d-farrell2@northwestern.edu">d-farrell2@northwestern.edu</a></font></div></div></div></span> </div><br></body></html>