Research IT

Nova Research Cluster Memorandum of Understanding

Background

Recognizing that High Performance Computing (HPC) is an enabler of discovery and innovation, ISU is investing in the HPC@ISU initiative with the following goal of leveraging campus HPC resources and expertise by combining separate activities into a unified HPC effort to ensure that needed computational resources are available to the campus research community. This sustainable model for HPC is supported through a combination of university and grant support.

We acquired a large research cluster, called CyEnce, through an NSF-MRI grant received in 2012. The Condo cluster was acquired in 2015.  Nova, acquired in 2018 with an MRI grant and expanded in 2021 with another MRI grant, incorporates the MRI cluster into the condo mode.  In this model, faculty use grant funds (or departmental funds, startup funds, institutional education support, etc.) to purchase compute nodes in a coordinated all-campus purchase.  The university sources (called the base funds) will fund costs associated with space, utilities, racks (to house compute processors and storage), network, software, and systems support staff.  This in effect represent almost a one-to-one match to funds being spent by researchers, departments, and colleges for their researchers. The base funds are provided through a combination of HPC funds provided by the VPR office and colleges. This approach effectively splits the cost evenly between institutional and faculty sources.  The cluster is maintained by the central staff. Participants will be able to use computational cycles proportional to the investment, however, at any given time, researchers will be able to also use idle cycles across the cluster.

 

Equipment and purchase

Research groups will be able to purchase both nodes and storage.  Purchases will be made through ITS.  

 

Compute nodes

  1. Compute nodes configuration will be defined by the cluster management staff. 
  2. All nodes are purchased with 5 years warranty. 
  3. Research groups can split node costs with other research groups as long as this is allowed by the funding source.

Storage

  1. Long-term production storage (work storage) is purchased in TB increments.  
  2. A production storage volume will be created for each research group.
  3. For each production storage volume, there will be an option of having equivalent sized non-production volume of cheaper storage.  Each production volume will be synced nightly to the non-production volume.  
  4. In case of hardware or file system failure on a production volume, the files from the non-production volume will be copied back to the production system.  This will provide access to files which have not changed since the last sync.  Though not enterprise level backup, it does provide a level of data integrity.  It is limited by synchronization transfer rates and utilization. Users should store a copy of any mission-critical data off-site.
  5. Research groups can select to purchase only production storage without the equivalent non-production storage. In this case no local backup described in item (3) will be provided.
  6. The one-time cost provides storage valid for 5 years. At the end of the 5 year period research groups will have to purchase new storage or move data off the cluster.
  7. After initial storage purchase more storage can be purchased later. If agreed by the research group, old purchase can be pro-rated to coincide with the end date of the new purchase.
  8. New research groups on the cluster may start with 1TB of free group work storage on older hardware. Paid storage will replace the free storage and existing data will be moved to the new equipment.
  9. Note: Storage amounts are after ZFS RAID-3 is applied, in TB where TB=10^12 bytes. Final storage space will be somewhat smaller than these round figures as some of this space will be used by the file system and the amount of file system overhead will depend on the pattern of use. Automatic on-the-fly compression is used both to allow more storage and speed up access to spinning disks.

Purchasing

  1. Purchases may be made as a collaboration of researchers or grants.

Hardware Return request

  1. Purchasers of full sets of storage that can be taken off the cluster may have the storage returned to them should they so wish.
    1. The warranty will no longer apply.  
    2. No infrastructure will be returned.
  2. Purchasers of a complete set of compute resources (a whole node that can be taken off the cluster) may have those resources returned to them should they wish 
    1. The warranty will no longer apply. 
    2. ITS staff will format the hard drives of returned systems to ensure we abide by software licensing restrictions. 

End of Warranty

  1. Warranty-expired compute nodes will no longer count toward the purchaser's contribution to the cluster.
  2. These nodes can still be used on the cluster.

End of life

  1. Equipment falls under the policy at https://www.policy.iastate.edu/policy/equipment/disposal.

Operation

Shared Resource

  1. This cluster is a shared resource, the cluster infrastructure components are purchased with shared funds.  The compute nodes will become part of that shared resource pool. All of the nodes will be utilized by all research groups. This sharing of resources is fundamental to a well-functioning cluster.  The primary benefit of this method is that it allows researchers to access a larger resource than they would otherwise be able to afford consequently allowing researchers to solve larger problems than they otherwise could.  
  2. To prevent one research group from dominating the cluster, the group limits are based on their contribution of computing resources. The contribution is calculated as three times the sum of all node purchases, where each purchase is linearly depreciated over five years from its purchase date.  

Connectivity

  1. Access to the cluster will require the use of two-factor authentication. This will provide an additional layer of security to help prevent a system compromise and avoid down time associated with recovery from such an event.
  2. The login node will be used for job submission, not for running programs. Such programs should be submitted to the job queuing system.
  3. Data transfer will be through a data transfer node to maintain interactive performance on the login node.   
  4. The cluster will be only accessible from on campus or via Iowa State VPN.

Queue

  1. A job scheduling system (SLURM) is used to reserve compute resources (CPU cores, memory, GPU cards) for the exclusive use of the jobs running on them.  It also enforces fair-sharing policies.  
  2. Complete utilization of the cluster is not possible, particularly when not using job cancelation policies. Typically job queuing delays do not present a problem until the system utilization is at about 67% utilization. At this level of cluster utilization, some jobs do not have enough resources to start immediately, but will start later when the system becomes less busy, typically during the off-peak night and weekend hours.  Due to this we will consider the system to be fully loaded at a utilization of 67% of all available node hours.
  3. Compute nodes purchased by research groups will be placed into nova SLURM partition and can be used by all researchers on the cluster.
  4. A smaller set of nodes will be placed in the interactive partition with a shorter runtime limit, allowing faster access to compute resources for short jobs.
  5. Usage is managed by the fair share scheduler in SLURM. Group's fairshare value is based on the contribution of the compute equipment as described in the section Shared Resource.   Jobs submitted into queue are assigned a priority to run, and the jobs with highest priority run first.  Additionally, there are limits on compute resources that can be used by all members of the group at the same time. These limits are based on the group contribution plus Community Tier allocation and exceed total capacity of the compute equipment purchased by the group. 
  6. Some grants require that nodes are available at any time for grant projects. These nodes are placed in restrictive partitions and are available to other users on the cluster through scavenger partition that will need to be explicitly specified when submitting a job. Jobs in the scavenger partition do not affect groups' usage but can be killed if the nodes are needed in the restrictive partitions.
  7. The classrom cluster has been combined with the research cluster. Classroom nodes are used by students taking classes via instruction partition. These nodes are available to other users on the cluster through scavenger partition.

Prioritization

  1. Groups which have used a smaller percentage of their contribution over the last 90 days will have a higher priority.  
  2. The fair share algorithm assigns lower priority to jobs from groups which have used larger portion of their contribution than jobs from those groups which have not.  This balances out usage over time to the percentage of the resources that each group has provided to the cluster.  

Community Tier

  1. ISU researchers can use the cluster in Community Tier without having to purchase compute equipment. Nodes out of warranty provide additional compute resources on the cluster to accommodate for community usage.