Access to High-end Systems

Systems

IBM iDataplex x86 system – Barcoo

  • Peak performance of 20 teraFLOPS – with Xeon Phi cards* running nominally at 1 teraFLOP each.
  • 1120 Intel Sandybridge compute cores running at 2.7GHz.
  • 67 nodes with 256GB RAM and 16 cores per node.
  • 3 nodes with 512GB RAM and 16 cores per node.
  • 20 Xeon Phi 5110P cards installed across 10 nodes.
  • Connected to a high speed, low latency Mellanox FDR14 InfiniBand switch for inter-process communications.
  • The system runs the RHEL 6 operating system, a variety of Linux.

Lenovo x86 system – Snowy

  • Peak performance – compute nodes currently performing at 30 teraFLOPS.
  • 992 Intel Haswell compute cores running at 2.3GHz.
  • 29 nodes with 128GB RAM and 32 cores per node.
  • 2 nodes with 512GB RAM and 32 cores per node.
  • Connected to a high speed, low latency FDR Mellanox InfiniBand switch for inter-process communications.
  • The system runs the RHEL 6 operating system, a variety of Linux.

Storage infrastructure (shared by all systems)

  • 700TB GPFS Parallel Data Store
  • 1PB HSM tape system, made available through GPFS

Access

Applying for resources in 2017

Melbourne precinct and associated researchers may submit a ‘light-touch’ application at any time of the year, detailing their project and evidence of project merit.

APPLY NOW

In 2016 we introduced a new Resource Application process and a ‘fair-share’ compute resource scheduling system.  They have been very successful and we trust they have simplified your access and use of our compute resources.

Based on this experience, we have now changed the application process so that:
  • existing projects do not need to reapply
  • new projects can apply any time
  • you do not need to estimate your resource requirements
  • any project which has been ‘inactive’ for 6 months will be closed.
As announced, after extensive discussions with our funding organisations and major compute project leads, our supercomputer, the IBM BlueGene/Q ‘Avoca’ was retired at the end of 2016. We are working with Avoca-based projects to help with data migration. We are also monitoring and modifying some resource allocations and compute job scheduling to ensure all projects have reasonable resources and responsive scheduling. This may involve limiting large jobs.

Project management and reporting

  • Applicants allocate personnel to various roles on an application for resources.
  • Project Managers are responsible for keeping records of account holders on the systems up to date.
  • Project Supervisors are required to submit one annual report on research outcomes (in February of the year following the year when systems were used).

Storage

Melbourne Bioinformatics has limited data storage capacity. Currently, data storage is intended only for current work, not long term archive. Data storage and backup capacity is becoming a serious problem and we need your assistance. By regularly removing your completed data from the system you make the system more responsive, minimise backups, and help us from running out of storage space.  Could you please ensure all project members remove any data not needed for their immediate compute needs.

Systems documentation (including detailed terms and conditions)

Costs of Access

There will be no charge for use of computational resources. Usage of data storage beyond that granted for a project may incur a charge.

There is some budget to instal application software and data sets that are commonly used by the life sciences research community. Commercial software required by only one project may be purchased at the Facility Manager’s discretion.

Users are expected to access the Facility via AARNet and charges are covered but users must pay for any communications charges billed directly to them from AARNet and any other data communications organisations that they may use.

Principles governing resource access

The governing principles of the new application and allocation processes are informed by the requirements and priorities of any member institutes and our observations of how researchers use our systems. In summary, we are:

  • simplifying the application and reporting processes to reduce the burden on researchers
  • implementing flexible application submission and evaluation timelines that recognise the often unpredictable and intermittent demands for research-driven computing
  • introducing a more flexible allocation process for the majority of our users
  • maximising productivity of our systems by finding cloud and local cluster solutions for smaller jobs, if appropriate, and continuing to offer support and training to ensure jobs run efficiently
  • prioritising resource requests from member institutions and stakeholders.

Access from 1 January 2018 – notice to holders of legacy projects

State Government funding of VLSCI ceased at the end of 2016. As part of the transition to Melbourne Bioinformatics, we continued in goodwill to support existing projects from all Victorian institutes during 2017. From 2018, these legacy projects will no longer be eligible to run on our systems – compute jobs can only run up until 11:59pm, 31 December 2017.

All data must be removed by 31 March 2018 and any remaining data will then be deleted. Project owners will need to allow plenty of time to remove data. We estimate that it can take a full day to move 1 TB of data, however, please be mindful that as we get closer to the close off date, we can expect speeds to slow due to bottlenecks, so starting as soon as possible is advised.

Future options for non-members

Applications for the National Computational Merit Allocation Scheme (NCMAS) are currently open: https://ncmas.nci.org.au/2018/.

From 2018, Deakin, RMIT, La Trobe and Melbourne Universities, as partners on a successful LIEF grant to procure a new high-performance GPU cluster, will have access to this system being hosted by the University of Melbourne’s Research Platform Services (not Melbourne Bioinformatics). Storage and access details for the new cluster have not yet been finalised. No direct project migration to this new resource is possible, so it is essential that all data from these projects is removed from Melbourne Bioinformatics systems as noted above.

Thank you for your support and understanding. It has been a pleasure to host such wonderful and diverse research, and hopefully there will be opportunities to collaborate in the future.