Editing Portal:HPC - High Performance Computing (section)

=== HPC file formats and data movement ===

HPC systems produce large amounts of data. Perl’s file-processing strengths make it useful for metadata extraction, indexing, verification, and format conversion, even when the largest binary arrays are handled by specialized libraries.

PDL supports scientific data workflows, and Perl can also interface with libraries and command-line tools for formats such as NetCDF, HDF5, FITS, GRIB, CSV, binary arrays, logs, and custom model formats. In many workflows, Perl does not need to load every byte into memory. It may only need to inspect headers, extract time ranges, verify dimensions, create manifests, or call external tools.

This is important because HPC output is often too large for casual handling. A good workflow needs automation around data selection, compression, archiving, and validation. Perl is effective in these tasks because it combines file-system control, pattern matching, databases, and reporting.