Email this article Parallel Processor Design and Implementation for Molecular Dynamics Simulations on a FPGA-Based Supercomputer
Abstract
The design and implementation of an FPGA core that parallelises all the necessary operations to compute the non-bonded interactions in a MD simulation with the purpose of accelarating the LAMMPS MD software is presented in this paper. Our MD processor core comprised of 4 identical pipelines working independently in parallel to evaluate the non-bonded potentials, forces and virials was implemented on the nodes of a FPGA-based supercomputer named Maxwell. Implementing our FPGA core on multiple nodes of Maxwell allowed us to produce a special-purpose parallel machine for the hardware acceleration of MD simulations. The timing performance figures of this machine for the pairwise LJ and short-range Coulombic (via PPPM) interaction computations in the MD simulations of the solvated Rhodopsin protein systems with various numbers of atom show performance gains over the pure software implementation by factors of up to 13 on two nodes of the Maxwell machine. Furthermore, our MD machine is highly scalable, yielding higher computational power with the additional Maxwell nodes. To our knowledge, this is the first attempt to port an existing production-grade MD software to a FPGA-based parallel computer.
Keywords
References
[1] M. P. Allen and D. J. Tildesley, “Computer Simulation of Liquids”, Oxford University Press, 1987.
[2] G. D. Fasman, “Prediction of Protein Structure and the Principles of Protein Conformations”, Plenum Press, New York, 1989.
http://dx.doi.org/10.1007/978-1-4613-1571-1
[3] Z. R. Wasserman and C. N. Hodge, “Fitting an inhibitor into the active site of thermolysin: A molecular dynamics case study”, J. Proteins: Structure, Function, and Bioinformatics, vol. 24, no. 2, pp. 227-237, Feb. 1996.
http://dx.doi.org/10.1002/(SICI)1097-0134(199602)24:2<227::AID-PROT9>3.0.CO;2-F
[4] N. R. Taylor and M. Itzstein, “A structural and energetics analysis of the binding of a series of N-acetylneuraminic-acid-based inhibitors to influenza virus sialidase”, J. Computer-Aided Molecular Design, vol. 10, no. 3, pp. 233-246, June 1996.
http://dx.doi.org/10.1007/BF00355045
PMid:8808739
[5] D. I. Liao, E. Silverton, Y. J. Seok, B. R. Lee, A. Peterkofsky and D. R. Davies, “The first step in sugar transport: crystal structure of the amino terminal domain of enzyme I of the E. coli PEP: sugar phosphotransferase system and a model of the phosphotransfer complex with HPr.”, J. Structure, vol. 4, no. 7, pp. 861-872, July 1996.
http://dx.doi.org/10.1016/S0969-2126(96)00092-5
[6] N. L. Greenbaum, I. Radhakrishnan, D. J. Patel and D. Hirsh, “Solution structure of the donor site of a trans-splicing RNA”, J. Structure, vol. 4, no. 6, pp. 725-733, June 1996.
http://dx.doi.org/10.1016/S0969-2126(96)00078-0
[7] D. C. Rapaport, “The Art of Molecular Dynamics Simulation”, Cambridge University Press, New York, 2004.
http://dx.doi.org/10.1017/CBO9780511816581
[8] P. P. Ewald, “Evaluation of optical and electrostatic lattice potentials”, Ann. Phys Leipzig, vol. 64, pp. 253-287, 1921.
http://dx.doi.org/10.1002/andp.19213690304
[9] L. Verlet, “Computer "Experiments" on Classical Fluids. I. Thermodynamical Properties of Lennard-Jones Molecules”, Physical Review, vol. 159, no. 1, pp. 98-103, July 1967.
http://dx.doi.org/10.1103/PhysRev.159.98
[10] D. Beeman, “Some Multistep Methods for Use in Molecular Dynamics Calculations”, J. Computational Physics, vol. 20, pp. 130-139, Feb. 1976.
http://dx.doi.org/10.1016/0021-9991(76)90059-0
[11] M.E. Tuckerman, G.J. Martyna and B.J. Berne, “Reversible multiple time-scale molecular dynamics”, J. Chemical Physics, vol. 97, no. 3, pp. 1990-2001, March 1992.
http://dx.doi.org/10.1063/1.463137
[12] D. V. D. Spoel, E. Lindahl, B. Hess, G. Groenhof, A. E. Mark and H. J. Berendsen, “GROMACS: Fast, flexible, and free”, J. Computational Chemistry, vol. 26, no. 16, pp. 1701-1718, Oct. 2005.
http://dx.doi.org/10.1002/jcc.20291
PMid:16211538
[13] GROMACS-4.0.7, “Download website for GROMACS 4.0.7”, available at http://www.gromacs.org, Dec. 2009.
[14] J. C. Phillips, R. Braun, W. Wang, J. Gumbart, E. Tajkhorshid, E. Villa, C. Chipot, R. D. Skeel, L. Kale and K. Schulten, “Scalable molecular dynamics with NAMD”, J. Computational Chemistry, vol. 26., no. 16, pp. 1781-1802, Oct. 2005.
http://dx.doi.org/10.1002/jcc.20289
PMid:16222654 PMCid:2486339
[15] NAMD-2.7b2, “Download website for NAMD 2.7b2”, available at http://www.ks.uiuc.edu/Research/namd, Nov. 2009.
[16] LAMMPS, “Download website for LAMMPS”, available at http://lammps.sandia.gov, Jan. 2010.
[17] F. Toshiyuki, T. Makoto, M. Junichiro, E. Toshikazu and S. Duiichiro, “A Highly Parallelized Special-Purpose Computer for Many-Body Simulations with an Arbitrary Central Force: MD-GRAPE”, Astrophysical Journal, vol. 468, pp. 51-61, Sep. 1996.
http://dx.doi.org/10.1086/177668
[18] Y. Komeiji, M. Uebayasi, R. Takata, A. Shimizu, K. Itsukashi and M. Taiji, “Fast and Accurate Molecular Dynamics Sumulation of a Protein Using a Special-Purpose Computer”, J. Computational Chemistry, vol. 18, no. 12, pp. 1546-1563, Sep. 1997.
http://dx.doi.org/10.1002/(SICI)1096-987X(199709)18:12<1546::AID-JCC11>3.0.CO;2-I
[19] S. Toyoda, H. Miyagawa, K. Kitamura, T. Amisaki, E. Hashimoto, H. Ikeda, A. Kusumi and N. Miyakawa, “Development of MD Engine: High-Speed Accelerator with Parallel Processor Design for Molecular Dynamics Simulations”, J. Computational Chemistry, vol. 20, no.2, pp. 185-199, 1999.
http://dx.doi.org/10.1002/(SICI)1096-987X(19990130)20:2<185::AID-JCC1>3.0.CO;2-L
[20] C. Wolinski, F. Trouw and M. B. Gokhale, “A preliminary study of molecular dynamics on reconfigurable computers”, Proc. International Conf. Engineering Reconfigurable Systems and Algorithms, June 2003.
[21] R. Scrofano and V. K. Prasanna, “Computing Lennard-Jones potentials and forces with reconfigurable hardware”, Proc. International Conf. Engineering Reconfigurable Systems and Algorithms, June 2004.
[22] R. Scrofano, M. B. Gokhale, F. Trouw and V. K. Prasanna, “Accelerating Molecular Dynamics Simulations with Reconfigurable Computers”, IEEE Trans. on Parallel and Distributed Systems, vol. 19, no. 6, pp. 764-778, June 2008.
http://dx.doi.org/10.1109/TPDS.2007.70777
[23] Y. Gu, T. VanCourt and M. C. Herbordt, “Improved interpolation and system integration for FPGA-based molecular dynamics simulations”, Proc. International Conf. Field Programmable Logic and Applications, pp. 21-28, 2006.
http://dx.doi.org/10.1109/FPL.2006.311190
[24] Y. Gu, T. VanCourt and M. C. Herbordt, “Explicit design of FPGA-based coprocessors for short-range force computations in molecular dynamics simulations”, Elsevier Parallel Computing, vol. 34, no. 4, pp. 261-277, May 2008.
http://dx.doi.org/10.1016/j.parco.2008.01.007
PMid:19412319 PMCid:2440579
[25] N. Azizi, I. Kuon, A. Egier, A. Darabiha and P. Chow, “Reconfigurable Molecular Dynamics Simulator”, Proc. IEEE Symp. Field-Programmable Custom Computing Machines, pp. 197-206, Apr. 2004.
http://dx.doi.org/10.1109/FCCM.2004.48
[26] Y. Gu, T. VanCourt and M. C. Herbordt. “Accelerating molecular dynamics simulations with configurable circuits”, Proc. International Conf. Field Programmable Logic and Applications, pp. 475-480, Aug. 2005.
[27] LAMMPS, “LAMMPS manual”, available at http://lammps.sandia.gov/doc/Manual.html, Jan. 2010.
[28] L. Greengard and V. Rokhlin, “A Fast Algorithm for Particle Simulations”, J. Computational Physics, vol. 73, no.2, pp. 325-348, Dec. 1987.
http://dx.doi.org/10.1016/0021-9991(87)90140-9
[29] H. Q. Ding, N. Karasawa and W. A. Goddard, “Atomic level simulations on a million particles: The cell multipole method for Coulomb and London nonbond interactions”, J. Chemical Physics, vol. 97, no. 6, pp. 4309-4315, Sep. 1992.
http://dx.doi.org/10.1063/1.463935
[30] R. W. Hockney and J. W. Eastwood, “Computer Simulation Using Particles, Adam Hilger, 1988.
http://dx.doi.org/10.1887/0852743920
[31] T. Darden, D. York and L. Pedersen, “Particle mesh Ewald: An N⋅log(N) method for Ewald sums in large systems”, J. Chemical Physics, vol. 98, no. 12, pp. 10089-10092, June 1993.
http://dx.doi.org/10.1063/1.464397
[32] S. Plimpton, “Fast Parallel Algorithms for Short-Range Molecular Dynamics”, J. Computational Physics, vol. 117, no. 1, pp. 1-19, Mar. 1995.
http://dx.doi.org/10.1006/jcph.1995.1039
[33] S. J. Plimpton, R. Pollock, M. Stevens, “Particle-Mesh Ewald and rRESPA for Parallel Molecular Dynamics Simulations”, Proc. SIAM Conf. Parallel Processing for Scientific Computing, Mar. 1997.
[34] E. L. Pollock and J. Glosli, “Comments on P3M, FMM, and the Ewald method for large periodic Coulombic systems”, Computer Physics Communications, vol. 95, no. 2, pp. 93-110, June 1996.
http://dx.doi.org/10.1016/0010-4655(96)00043-4
[35] R. Baxter, S. Booth, M. Bull, G. Cawood, J. Perry, M. Parsons, A. Simpson, A. Trew, A. McCormick, G. Smart, R.Smart, A. Cantle, R. Chamberlain and G. Genest, “Maxwell—a 64 FPGA supercputer”, Proc. NASA/ESA Conf. Adaptive Hardware Systems, pp. 287-294, 2007.
[36] FHPCA, Edinburgh, U.K., “The FHPCA website”, available at http://www.fhpca.org, 2010.
[37] Alpha Data Ltd., Edinburgh, U.K., “ADM-XRC-4FX Datasheet”, available at http://www.alphadata.co.uk/adm-xrc-4fx.html, May 2007.
[38] Nallatech Ltd., Glasgow, U.K., “H100 Series Datasheet”, available at http://www.nallatech.com/meadiLibrary/images/english/5595.pdf, May 2007.
[39] R. Baxter, S. Booth, M. Bull, G. Cawood, J. Perry, M. Parsons, A. Simpson, A. Trew, A. McCormick, G. Smart, R. Smart, A. Cantle, R. Chamberlain and G. Genest, “The FPGA HPC alliance parallel toolkit”, Proc. NASA/ESA Conf. Adaptive Hardware Systems, pp.301-310, 2007.
[40] FHPCA, Edinburgh, U.K., “PowerPoint presentation”, available at http://www.fhpca.org/download/MRSC07-Mar07.ppt, Mar. 2007.
[41] Argonne National Lab, Argonne, IL, “MPI manual”, available at http://www.unix.mcs.anl.gov/mpi/www/www3/MPI_Wtime.html, 2009.
[42] OpenCores website, “Floating Point Adder and Multiplier”, available at http://opencores.org/project,fpuvhdl, 2009.
[43] G. Marcus, P. Hinojosa, A. Avila and J. N. Flores, “A Fully Synthesizable Single-Precision, Floating-Point Adder/Subtractor and Multiplier in VHDL for General and Educational Use”, Proc. IEEE International Caracas Conf. Devices, Circuits and Systems, pp. 319-323, Nov. 2004.
http://dx.doi.org/10.1109/ICCDCS.2004.1393405
[44] Xilinx Inc., San Jose, CA, “Virtex-4 datasheets”, available at http://www.xilinx.com/products/silicon_solutions/fpgas/virtex/virtex4/index.htm, May 2007.
[45] T. Amisaki, T. Fujiwara, A. Kusumi, H. Miyagawa and K. Kitamura, “Error evaluation in the design of a special-purpose processor that calculates nonbonded forces in molecular dynamics simulations”, J. Computational Chemistry, vol. 16, no. 9, pp. 1120-1130, Sep. 1995.
http://dx.doi.org/10.1002/jcc.540160906
[46] M. Chiu, M.C. Herbordt, “Efficient particle-pair filtering for acceleration of molecular dynamics simulation”, Proc. International Conf. Field Programmable Logic and Applications, pp. 345-352, Aug. 2009.
http://dx.doi.org/10.1109/FPL.2009.5272272
[47] V. Kindratenko, D. Pointer, “A case study in porting a production scientific supercomputing application to a reconfigurable computer”, Proc. IEEE Symp. Field-Programmable Custom Computing Machines, pp. 13-22, April 2006.
http://dx.doi.org/10.1109/FCCM.2006.5
Full Text: PDF