Blizzard Challenge organisation

From SynSIG
Revision as of 14:51, 3 January 2019 by Simon.King (talk | contribs)

Who organises the Blizzard Challenge?

The challenge was conceived by Alan Black and Keiichi Tokuda, as described in their Interspeech 2005 paper. In 2007, they were joined by Simon King. Together these three people form the steering committee with overall control of the challenge.


Each annual challenge has an organiser who is responsible for the main tasks of data preparation and release, planning and executing the evaluation, and arranging the workshop. In the first years of the challenge, this organiser was one of the steering committee.

Steering Committee

Annual organisers

Who provides the data?

The data are donated by external organisations. Where possible, we ask for the data to be made available for longer term research use either only to the participating teams, or more generally.

Data providers for each challenge

How to get involved in the organisation of the challenge

Providing data

We are always looking for new an interesting data. The amount of data used in the Challenge has varied widely over the years, from 1 hour to 100s of hours. If external data is allowed, or a multi-speaker corpus is made available, then the amount data from the target speaker(s) can be small. However, for a conventional single-speaker corpus, most participants would probably like a minimum of 5 hours (e.g., Blizzard Challenges 2017 and 2018).

The data can be raw and unprocessed (e.g., complete audiobooks) or partially segmented by the provider. There must normally be a transcript, but this may not be aligned with the data, and small mismatches can be handled by most participants. We have a system for sharing the effort of preparing data across the participating teams, with a Git repository of cleaned transcripts, alignments, etc. This system works best for data that is used for two or three consecutive challenges.

Specifying the task and designing the evaluation

There is a default design of task and evaluation, and we have not deviated from this much over the years. But we are always looking for constructive criticism of all aspects, including the task for participants, the materials to be synthesised, the listening test, and the statistical analysis.

Organisers' checklist

Here are the key tasks and timing for organising a Blizzard Challenge. Please can organisers update this after each Challenge to make it as useful as possible to future organisers.

Data

  • Prepare the data for distribution
  • Obtain a clear written statement from the owner of the data about usage rights, and store this in a safe place.
  • Make an initial guess at materials for the listening test, and hold out sufficient natural data from the distribution. In general, plan for 3 consecutive years using the same data, so hold out at least enough data for three listening tests.
  • Choose a license (a good default starting point is a recent license). Make sure the data owner is happy with the license. If you are able to release the data under liberal terms (e.g., allowing commercial use) get this agreed now, and don't leave it for later.
  • We restrict access to the data to registered teams. But you still need to determine now whether you will be able to make the data more widely available after completion of the Challenge. Make sure the license is consistent with what you decide.
  • Decide how to distribute the data. By default, it will be hosted at the University of Edinburgh and Simon King will take care of the online license form, and issuing of passwords to registered teams. If you are able to make the data more generally available after the Challenge is finished, then be sure to host it in a suitable location.

Website

  • Create pages for the Challenge, the rules, and the workshop, on the SynSIG website. It's easiest to copy pages from previous years, then edit.
  • Decide on a timeline, and publish this on the website

Participating teams

  • Write the call for participation, probably based on one from a previous year. Create a PDF version.
  • Write the rules for the challenge and publish on the SynSIG website.
  • Publish the call on the SynSIG website and send to appropriate mailing lists: blizzard-discuss, synsig, Festival, HTS.

Listening test

  • Depending on the language, the listening test facility at the University of Edinburgh is a good choice for running around 100-150 "gold standard" listeners under ideal conditions.
  • Decide whether you also want to use paid crowd-sourcing to obtain further listeners.
  • Ask Simon King to approach sponsors to cover some of the costs of the listening test, including payments to listeners and to assistants running the test. Agree with him how this funding will be used (e.g., do you need money to pay your own assistants or to pay for crowd-sourcing?). Normally, sponsors make a donation to the University of Edinburgh, and funds are then distributed as needed.

Workshop

Simon King is willing to do most of this part of the organisation, so try to delegate this task to him!

  • Choose a date and location for the workshop. The default is a satellite of Interspeech or SSW.
  • Find a sponsor for the workshop to cover: venue cost, event catering and evening drinks.
Retrieved from "http://synsig.it.helsinki.fi/index.php?title=Blizzard_Challenge_organisation&oldid=2766"