Light-weight parallel Python tools for earth system modeling workflows
Paul, K., Mickelson, S., Dennis, J. M., Xu, H., Brown, D.. (2015). Light-weight parallel Python tools for earth system modeling workflows.
| Title | Light-weight parallel Python tools for earth system modeling workflows |
|---|---|
| Genre | Conference Material |
| Author(s) | Kevin Paul, Sheri Mickelson, John M. Dennis, Haiying Xu, David Brown |
| Abstract | In the last 30 years, earth system modeling has become increasingly data-intensive. The Community Earth System Model (CESM) response to the next Intergovernmental Panel on Climate Change (IPCC) assessment report (AR6) may require close to 1 Billion CPU hours of computation and generate up to 12 PB of raw data for post-processing. Existing post-processing tools are serial-only and impossibly slow with this much data. To improve the post-processing performance, our team has adopted a strategy of targeted replacement of the "bottleneck software" with light-weight parallel Python alternatives. This allows maximum impact with the least disruption to the CESM community and the shortest delivery time. We developed two light-weight parallel Python tools: one to convert model output from time-slice to time-series format, and one to perform fast time-averaging of time-series data. We present the motivation, approach, and results of these two tools, and our plans for future research and development. |
| Publication Title | |
| Publication Date | Oct 29, 2015 |
| Publisher's Version of Record | |
| OpenSky Citable URL | https://n2t.org/ark:/85065/d7s1847f |
| OpenSky Listing | View on OpenSky |
| CISL Affiliations | TDD, ASAP, IOWAG, DVAT |