+44 40 8873432 [email protected]

Data Mover Challenge 2020 (DMC20)

Update: Submission for DMC20 entries has closed. Thank you

The sharing and transfer of huge amounts of data across borders and continents are key foundations for the advancement of global science. Such data is essential in advanced, leading-edge research that run extreme data-intensive calculations to solve today’s most complex research puzzles for tomorrow’s innovations. Applications for the data are wide-ranging including the Internet of Things (IoT), Artificial Intelligence (AI), advanced manufacturing, drug discovery, weather monitoring, big data analysis and many others.

Over the past decade, such transfers have progressed beyond research and into applications, and at increasingly larger data amounts as innovations and standards of the data resolution being captured continues to evolve. The network bandwidth to support the growing demand has gone from Mega to Giga, and to hundreds of Gbps in just a few years. Despite these advancements in high bandwidth capacity, scientists and researchers are still not experiencing the corresponding increase in the “speed” of data transfer. This is due to multiple factors, including server and storage hardware limitations, operating system tuning, and data transfer tools used, to name a few.

The Data Mover Challenge (DMC), which is organised by the National Supercomputing Centre (NSCC), Singapore brings together experts from industry and academia, to push the envelope in high speed, big data transfer capabilities, and across vast distances. Participants for DMC19 included international teams from USA, Japan, Taiwan, Indonesia, Thailand, Vietnam and South Korea. The Data Transfer Nodes (DTNs) for DMC19 that were set up globally in Singapore, Australia, USA, South Korea and Japan were connected by 100Gbps international research and education networks.

The next edition of the Data Mover Challenge, DMC20, has been launched. This year’s edition will challenge participants to send test data effectively around the globe!

Submissions for DMC20 are open from 01 June 2019 till 31 July 2019.

The lead of the winning team will have the opportunity to showcase the team’s solutions at the SupercomputingAsia conference in Singapore, which includes return coach airfare and free admission passes to the event that gathers the best of HPC in Asia!

DMC20 Participating Teams

DMC20 – “Data for Science”

The capability to effect the sharing of large amounts of research data quickly and securely between collaborating entities or computational resources will rise in demand as science becomes more data-intensive with more precise and larger instruments, and global research cooperation increases.

The theme for DMC20, “Data for Science”, places an emphasis on the need for better coordination and speed in moving large amounts of data between multiple sites with Data Transfer Nodes (DTNs) as well as to HPC centres around the globe.

DMC20 will focus on optimising point-to-point data transfer between sites – a crucial step forward in advancing research collaboration and sharing.

DMC20 will extend the network testbed and location of DTNs to cover the entire world. The intention will be for the test data to circle the globe. Partners in Asia, Europe and USA will be supporting the competition.

DMC20 at SupercomputingAsia 2020

  • The winner of DMC20 will be announced at SupercomputingAsia 2020 (SCA20)!
  • SCA20 will be held from 24-27 February 2020 in Singapore (www.sc-asia.org). The lead of the winning team will be invited to SCA20 to receive their award and to present the team’s winning solution at the conference.

The inaugural DMC19 Overall Winner: Zettar Inc.

(Left-Right: Andrew Howard, Cloud Team Manager, National Computational Infrastructure Australia, Dr Chin Fang, Founder & CEO, Zettar Inc., Prof Tan Tin Wee, Chief Executive, National Supercomputing Centre (NSCC) Singapore)

Scope

Network Infrastructure

  • Each team will leverage on the global NREN infrastructure for the Data Mover Challenge.
  • The deployed DTNs will connect all locations, including Australia, Europe, Japan, Singapore and USA.
  • Teams must determine the connection type that will be used, for example, point-to-point, multipoint or distributed.
  • Multiple Virtual Local Area Networks (VLANs) will be set up, per location, to provide inter-connectivity.
  • Network monitoring will be set up to capture data transfer statistics.

Server Hardware sponsorship

  • Purpose-built DTN servers will be provided by partners based on baseline minimum specifications
    – e.g. 100/10 Gb capability, local storage, connection to high performance file systems
  • Existing DTN systems at each partner sites must be used.
  • Singularity container software will be used on the DTN servers.

Applications

  • Data transfer tools & software will be provided by the competing team.
  • Teams will need to implement their DTN transfer tools & software within Singularity.
  • A separate note will be sent to the shortlisted DMC20 participants at the start of the challenge with details on how to access the nodes, and other operational processes.

Data Mover Challenge Topology

Eligibility

  • DMC20 is open to all organisations, companies, research institutions, academia, researchers, post-graduate students and undergraduate students.
  • The final number of Participating Teams for the challenge will be assessed and shortlisted at the discretion of the DMC20 organising committee.
  • The organiser, co-organisers and representatives involved in organising DMC20 are not eligible to participate in the challenge.

Judging Criteria

1. Error-free data transfer performance (average transfer rate and time taken) of the data set
(Weightage 20%) Memory to Memory Test for both directions
(Weightage 40%) Disk to Disk Test for both directions

2. Technological Innovation base on solution submission and interview with judges
(Weightage 20%)

3. End-user Experience including user interface, ease of installation, licensing model, and integration with services such as access federation etc.
(Weightage 20%)

DMC20 Challenge Period

  • DMC20 competition period will run from 26 August 2019 to 23 January 2020.
  • Each Participating Team will be given 5 days for the challenge, which includes:
  1. Monday SGT (UTC+8) 0000Hrs – Wednesday SGT (UTC+8) 2359Hrs: Software Setup
  2. Thursday SGT (UTC+8) 0000Hrs – Friday SGT (UTC+8) 2359Hrs: Running and demonstrating of Software to judges (Statistics and results will be captured)
  • The winning team’s leader will be invited to attend SCA20 in Singapore from 24-27 February 2020 for the award presentation and solution showcase.

DMC20 Participants' Information

  • Click here to download the document.

Leader of winning team will receive

Return coach airfare to Singapore for team leader of the winning team, from the country he/she is based

Subsistence Allowance (including accommodation) during SCA20

Exclusive complimentary all-access conference pass for SCA20

Registration

Update: Submission for DMC20 entries has closed. Thank you

For enquiries about the challenge or any other details, please contact [email protected].

Important Dates

  • 1 JUN

    2019

    Launch of DMC20

  • 1 JUN

    31 JUL

    2019

    Submission period of entries for DMC20

  • 31 JUL

    2019

    Close of entry submissions for DMC20

  • 7 AUG

    2019

    Announcement of DMC20 Shortlisted Participants

  • 26 AUG
    2019

    23 JAN
    2020

    Start - Challenge to begin

  • 23 JAN

    2020

    End Challenge to conclude

  • 24 FEB

    27 FEB

    2020

    SupercomputingAsia 2020 (SCA20) - Gathering the Best of HPC in Asia!

  • 26 FEB

    2020

    DMC20 Award Ceremony - Announcement of Winner(s)

Supporting Partners