Student Autonomous Maritime Vehicle Competitions: International Coordination and Initial Benchmarking

EarthzineAutonomous Underwater Vehicles 2017, Themed Articles

IEEE OES Singapore Chapter

The Autonomous Marine Vehicles Competition Committee is envisioned to be a forum for the exchange of knowledge and ideas among all involved in sponsoring, managing and supporting autonomous maritime vehicle student competitions around the world.

Part I: International Coordination

The Autonomous Marine Vehicles Competition Committee (AMVC) was formed in March 2017. The committee is international in scope, comprised of organizers of current student autonomous maritime competitions, sponsors and educators. The competitions provide students with hands-on experience in system engineering of autonomous maritime vehicles and provide the organizers an opportunity to refine the metrics of robotic autonomy. The principal goal of the committee is seeking consensus on stimulating creativity, benchmarking baseline tasks, and establishing parity among existing and planned global competitions.

IEEE OES Singapore Chapter

Image Credit: IEEE OES Singapore Chapter

The idea for the committee grew out of informal discussions at recent oceans conferences about the growth of robotic competitions over the past 20 years, and the demonstrated value of the autonomous maritime events. The U.S. Office of Naval Research (ONR) has supported the competitions since their inception. Recently, the IEEE Oceanic Engineering Society (OES) has restructured internally to focus more on marine robotics competitions, reflecting the interests of OES membership.

The realization that existing competitions are evolving somewhat independently motivated the concept of bringing as many involved people as possible together for a face-to-face meeting. The plan developed into a two-day meeting, the first day to discuss the merits of a common set of standards to bring parity to competitions and among the groups operating them. The second day would be dedicated to chartering the committee and subcommittees and their operation, and defining value-added outputs beneficial to the community. Together, ONR-funded principal investigators and OES officers submitted a proposal to OES for funds to support such a meeting. Later at the AUV Symposium (AUV 2016) at the University of Tokyo in the fall of 2016, the plan was adopted and University of Porto was selected as the meeting location.

Those invited to participate included competition organizers and technical directors, industry experts, research engineers, government laboratory personnel and past participants. Attendees spanned the globe with representatives from India, Singapore, Japan, China, Australia, Canada, Scotland, France, Germany, Spain, Croatia, Scotland, Sweden, Ireland, Italy, Chile, Argentina, Portugal and the United States. Attendees included representatives from the following competitions:


Working together, the group formed a governing board and drafted a charter:

“the international Autonomous Marine Vehicle Competition (AMVC) Committee is a group with a committed interest in coordinating, improving and expanding the quality and value of autonomous maritime systems student competitions.”

Autonomous maritime systems include vehicles in, on or above the water. Members of the newly formed board identified efforts for the group to initially focus on: a common lexicon, benchmarking fundamental tasks of autonomy, guides for uniformly scoring tasks, common procedures and metrics for static judging, technical resources for competitors and mentors, and mechanisms for knowledge transfer as teams turn over. Part II describes an initial task benchmarking framework.

The Autonomous Marine Vehicles Competition Committee is envisioned to be a forum for the exchange of knowledge and ideas among all involved in sponsoring, managing and supporting autonomous maritime vehicle student competitions around the world. Involvement of OES strengthens the connection to professional development and provides a place for peer-reviewed and other recognized technical publications.

Participation in the competitions has clearly impacted the careers of a generation of engineering students to date, and that trend is likely to continue. The committee is committed to supporting student professional development and community interaction through special sessions and events at conferences and a website hosted by OES. Consistent with this approach, the committee plans to meet at existing symposiums, conferences and competitions each year and capitalize on intellectual investments in those activities.

Part II: Initial Task Benchmarking Framework

Since there are no quantitative units of autonomy, benchmarking of building-block tasks is essential for common baselining across competitions. Some key questions we are addressing include:

  • What are best tasks to test autonomy?
  • Is there any taxonomic structure to fundamental tasks?
  • Can benchmarks tasks be identified for all maritime autonomy competitions?
  • What other aspects of competitions might be managed in common?
  • Should there be a common qualifying procedure (e.g., maneuvers, papers)
  • What resources should be provided to support participants?
  • Should there be a common scoring system incentivizing creativity and innovation?
  • What are the potential venues for future competitions?
  • What is the best way to manage growth (size, complexity, multi-media)?
  • How can local chapters best complement national efforts?

An initial framework mapping fundamental autonomous behaviors to functional  tasks was constructed (Figures 1, 2, 3). Some promising structure emerged in this mapping in which functional tasks are organized by the component behaviors needed to complete the task. This framework is a work in progress and will be refined with community input.

Figure 1: Fundamental autonomous behaviors mapped onto functional tasks. Examples of functional tasks corresponding to the columns above are shown in Figure 2.

Figure 2: Functional tasks (columns in Figure 1) requiring a combination of fundamental behaviors (rows in Figure 1).

Additional questions include:

  • Should competitions be structured by division based on demonstrated skill (e.g., novice, intermediate, expert)?
  • What is the best way to align competition design efforts with academic classes and credit?  
  • How should potential users be involved in the definition of tasks?
  • Do we need to consider a series of sequentially more challenging task challenges?
  • How do we measure impact and success?
  • What are the informal and formal methods of cooperation?

The committee will seek answers to these questions in the coming months and years. For more information, visit the committee website.

Figure 3: Skill levels associated with functional tasks.

Acknowledgments: The University of Porto, Dr. Joao Sousa and Catarina Morais; OES, Drs. Rene Garello, Chris de Moustier and Malcom Heron.

William Kirkwood is a senior engineer at the Monetery Bay Aquarium Research Institute and treasurer of the IEEE OES.

Thomas Curtin is a senior scientist at the Applied Physics Laboratory, University of Washington, and the founder of the RoboSub competition.  

Yvan Petillot is a professor at Heriot-Watt University and a principal in the European Robotics League.  

João Sousa is a professor at Porto University and the host of the Workshop, Portugal.

Esteban Escobar is the chief executive officer of 5:00 Films. 

Zoz Brooks is a consultant with RoboNation.

All members of the committee contributed to the workshop discussions and output. They are acknowledged here in alphabetical order: Greado Acosta (U. Buenos Aries),  Gianluca Antonelli (U. di Cassino),  D. Atmanand (NIOT), Ralf Bachmayer (Memorial U.), Anne Bajart (EU), Julie Banner (U.S. Navy),  Dick Blidberg (consultant), Marc Carreras (U. Girona),  Kelley Cooper (U.S. Navy), Nuno Cruz (Porto U.), Janelle Curtis (AUVSI Foundation), Daryl Davidson (AUVSI Foundation), Daniel Deitz (U.S. Navy), Vladimir Djapic (consultant),  Tom Drake (U.S. Navy),  Shuangshuang Fan (Zhejiang U./ U. Tasmania),  Fausto Ferreira (CMRE),  Gabriele Ferri (CMRE), Hayato Kondo (Tokyo U.),  David Lane (Heriot-Watt U.), Richard Mills (Kongsberg), Rick Nagle (consultant and workshop scribe), David Novick (RoboSub), Felix Pageau (RoboBoat),  Venugopalan Pallayil (Nat. U. Singapore),  Bill Porter (U.S. Navy),  Aamir Qaiyumi (Robot-X),  Pere Ridao (U. Girona), Robert Simmons (U.S. Navy), Hanu Singh (WHOI),  Asgeir Sorenson (U. Norway),  Jason Stack (U.S. Navy), Ivan Stenius (KTH), Andy Stewart (APL/U. Washington), Tom Swean (consultant), Doug Todoroff (U.S. Navy), Giancarlo Troni (Pontifica U.), Zoran Vukic (U. Zagreb), Christoph Waldmann (U. Bremen), Alan Winfield (U.West England).