<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>Dear Paolo and Nichols,</div><div><br></div> as a follow up, I had a brief meeting with the sysadmin of our local BGP. It looks like the timings I was reporting actually correspond to the maximum I/O throughput of that specific rack, which depends on the number of I/O nodes present on the rack itself (in that case, 4 I/O nodes per midplane, each of them capable of 350 MB/s, corresponding to 1.7 GB/s for that midplane).<div>In the example I was reporting:</div><div> davcio : 1083.30s CPU 1083.30s WALL ( 38 calls)</div><div><br><div>I've been running on just 128 nodes (512 cores in VN mode), therefore I had only one I/O node (1 midplane = 4 x 128 nodes, for the non-BG/P-ers). Now, the total size of the .wfc files was around 9200 MB, which cannot be written in less than 9200/350 = 26.3 sec, according to the figures that the sysadmins gave me. </div><div>In my case the timings give: 1083.30s/38=28.5s, which is close to the theoretical maximum. </div><div>I will perform more testing and I will take into consideration the suggestion of Nichols about the number of files per node. In our machine we have one rack with 16 I/O nodes per midplane, I will try to see if the I/O performance scales accordingly.</div><div><br></div><div>As a side effect, I met a problem in the timing procedure. I found very different davcio timings (i.e. 3 orders of magnitude!) for two jobs where the size of the wavefunctions differed by a factor 2 only (the jobs have been executed on the same rack and with the same number of processors and same parallelization scheme). </div><div>The sysadmins replied that I/O bandwidth measured in the fastest case is not attainable on BG/P, and should be imputed to an inaccurate measurement of cputime/walltime. </div><div>I'm going to investigate this anyway.</div><div><br></div><div>I'm not aware of anyone working on MPI I/O porting.</div><div><br></div><div>Thanks so far for your suggestions,</div><div><br></div><div><br></div><div><br></div><div>Gabriele</div><div><br></div><div><br></div><div><br><div><div>Il giorno 11/apr/2011, alle ore 20.02, Nichols A. Romero ha scritto:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Sorry for not replying earlier, but I missed this e-mail due to the<br>APS March Meeting.<br><br>The GPFS file system on BG/P does a poor job at handling writes to more than<br>one file per node. My guess is that Gabriele was running QE in either dual<br>or VN mode (2 and 4 MPI tasks per node, respectively). So on BG/P,<br>you basically<br>want to write one file per node (which GPFS is designed to handle) or<br>one big file<br>using MPI-I/O.<br><br>At ANL, we are thinking about re-writing some of the I/O<br>using parallel I/O (e.g. HDF5, Parallel NetCDF). The simplest<br>approach, though highly<br>unportable, is to use the MPI I/O directly.<br><br>Has anyone on this list worked on parallel I/O with QE? Or have any<br>strong opinions<br>on this issue?<br><br><br>On Wed, Mar 30, 2011 at 11:57 AM, Paolo Giannozzi<br><<a href="mailto:giannozz@democritos.it">giannozz@democritos.it</a>> wrote:<br><blockquote type="cite"><br></blockquote><blockquote type="cite">On Mar 30, 2011, at 11:20 , Gabriele Sclauzero wrote:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">Do you think that having an additional optional level of I/O<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">(let's say that it might be called "medium")<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I propose 'rare', 'medium', 'well done'<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">would be too confusing for users?<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">some users get confused no matter what<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">I could try to implement and test it.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">ok: just follow the "io_level" variable. Try first to understand<br></blockquote><blockquote type="cite">what the actual behavior is (the documentation is not so<br></blockquote><blockquote type="cite">clear on this point) and then think what it should be, if you<br></blockquote><blockquote type="cite">have some clear ideas<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">P.<br></blockquote><blockquote type="cite">---<br></blockquote><blockquote type="cite">Paolo Giannozzi, Dept of Chemistry&Physics&Environment,<br></blockquote><blockquote type="cite">Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br></blockquote><blockquote type="cite">Phone +39-0432-558216, fax +39-0432-558222<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">Pw_forum mailing list<br></blockquote><blockquote type="cite"><a href="mailto:Pw_forum@pwscf.org">Pw_forum@pwscf.org</a><br></blockquote><blockquote type="cite"><a href="http://www.democritos.it/mailman/listinfo/pw_forum">http://www.democritos.it/mailman/listinfo/pw_forum</a><br></blockquote><blockquote type="cite"><br></blockquote><br><br><br>-- <br>Nichols A. Romero, Ph.D.<br>Argonne Leadership Computing Facility<br>Argonne, IL 60490<br>(630) 447-9793<br>_______________________________________________<br>Pw_forum mailing list<br><a href="mailto:Pw_forum@pwscf.org">Pw_forum@pwscf.org</a><br>http://www.democritos.it/mailman/listinfo/pw_forum<br></div></blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div><span class="Apple-style-span" style="color: rgb(126, 126, 126); font-size: 16px; font-style: italic; "><br class="Apple-interchange-newline">§ Gabriele Sclauzero, EPFL SB ITP CSEA</span></div><div><font class="Apple-style-span" color="#7E7E7E"><i> PH H2 462, Station 3, CH-1015 Lausanne</i></font></div></span>
</div>
<br></div></div></body></html>