Blizzard Challenge organisation: Difference between revisions

From SynSIG
Line 77: Line 77:
=== Data ===
=== Data ===


* Prepare the data for distribution
* Obtain a clear written statement from the owner of the data about usage rights, and store this in a safe place.
* Obtain a clear written statement from the owner of the data about usage rights, and store this in a safe place.
* Make an initial guess at materials for the listening test, and hold out sufficient natural data from the distribution. In general, plan for 3 consecutive years using the same data, so hold out at least enough data for three listening tests.
* Make an initial guess at materials for the listening test, and hold out sufficient natural data from the distribution. In general, plan for 3 consecutive years using the same data, so hold out at least enough data for three listening tests.
* Choose a license (a good default starting point is a recent license). Make sure the data owner is happy with the license. If you are able to release the data under liberal terms (e.g., allowing commercial use) get this agreed now, and don't leave it for later.
* Choose a license (a good default starting point is a recent license). Make sure the data owner is happy with the license. If you are able to release the data under liberal terms (e.g., allowing commercial use) get this agreed now, and don't leave it for later.
* Prepare the data for distribution
* Access to the data is initially restricted to registered teams. But you still need to determine now whether you will be able to make the data more widely available after completion of the Challenge. Make sure the license is consistent with what you decide.
* Access to the data is initially restricted to registered teams. But you still need to determine now whether you will be able to make the data more widely available after completion of the Challenge. Make sure the license is consistent with what you decide.
* We normally use a click-through license form which requires manual checking and issuing of passwords. This is necessary during the Challenge to restrict access to only registered teams. It is also the standard method for wider distribution after the Challenge, but if you are also to remove the need for manual checking that would save a lot of work in the long run - you need to agree this with the data owner now.
* We normally use a click-through license form which requires manual checking and issuing of passwords. This is necessary during the Challenge to restrict access to only registered teams. It is also the standard method for wider distribution after the Challenge, but if you are also to remove the need for manual checking that would save a lot of work in the long run - you need to agree this with the data owner now.

Revision as of 15:32, 3 January 2019

Who organises the Blizzard Challenge?

The challenge was conceived by Alan Black and Keiichi Tokuda, as described in their Interspeech 2005 paper. In 2007, they were joined by Simon King. Together these three people form the steering committee with overall control of the challenge.


Each annual challenge has an organiser who is responsible for the main tasks of data preparation and release, planning and executing the evaluation, and arranging the workshop. In the first years of the challenge, this organiser was one of the steering committee.

Steering Committee

Annual organisers

Who provides the data?

The data are donated by external organisations. Where possible, we ask for the data to be made available for longer term research use either only to the participating teams, or more generally.

Data providers for each challenge

How to get involved in the organisation of the challenge

Providing data

We are always looking for new and interesting data. The amount of data used in the Challenge has varied widely over the years, from 1 hour to 100s of hours. If external data is allowed, or a multi-speaker corpus is made available, then the amount data from the target speaker(s) can be small. However, for a conventional single-speaker corpus, most participants would probably like a minimum of 5 hours (e.g., Blizzard Challenges 2017 and 2018).

The data can be raw and unprocessed (e.g., complete audiobooks) or partially segmented by the provider. There must normally be a transcript, but this may not be aligned with the data, and small mismatches can be handled by most participants. We have a system for sharing the effort of preparing data across the participating teams, with a Git repository of cleaned transcripts, alignments, etc. This system works best for data that is used for two or three consecutive challenges.

Specifying the task and designing the evaluation

There is a default design of task and evaluation, and we have not deviated from this much over the years. But we are always looking for constructive criticism of all aspects, including the task for participants, the materials to be synthesised, the listening test, and the statistical analysis.

Organisers' checklist

Here are the key tasks for organising a Blizzard Challenge. Please can organisers update this after each Challenge to make it as useful as possible to future organisers.

Preliminaries

  • Form an organising committee for the current Challenge. You should include at least one member of the permanent steering committee
  • Check that you and all your committee are on the blizzard-discuss mailing list, so you can post to it
  • Get yourself added to the blizzard@festvox.org alias, managed by Alan Black
  • Get an account on the SynSIG website, which is hosted by Thierry Dutoit's group at U Mons. Simon King, or any member of the SynSIG steering committee can request this for you.

Data

  • Obtain a clear written statement from the owner of the data about usage rights, and store this in a safe place.
  • Make an initial guess at materials for the listening test, and hold out sufficient natural data from the distribution. In general, plan for 3 consecutive years using the same data, so hold out at least enough data for three listening tests.
  • Choose a license (a good default starting point is a recent license). Make sure the data owner is happy with the license. If you are able to release the data under liberal terms (e.g., allowing commercial use) get this agreed now, and don't leave it for later.
  • Prepare the data for distribution
  • Access to the data is initially restricted to registered teams. But you still need to determine now whether you will be able to make the data more widely available after completion of the Challenge. Make sure the license is consistent with what you decide.
  • We normally use a click-through license form which requires manual checking and issuing of passwords. This is necessary during the Challenge to restrict access to only registered teams. It is also the standard method for wider distribution after the Challenge, but if you are also to remove the need for manual checking that would save a lot of work in the long run - you need to agree this with the data owner now.
  • Decide how to distribute the data. By default, it will be hosted at the University of Edinburgh and Simon King will take care of the online license form, and issuing of passwords to registered teams. If you are able to make the data more generally available after the Challenge is finished, then be sure to host it in a suitable permanent location (the University of Edinburgh is a good option).

Website

  • Create pages for the Challenge, the rules, and the workshop, on the SynSIG website. It's easiest to copy pages from previous years, then edit.
  • Decide on a timeline, and publish this on the website. Refer to previous years to see what is realistic. The key dates (working backwards) are: date of the workshop (typically August or September), period where listeners are available for the listening test (e.g., at the University of Edinburgh you would need to start by late April and finish by early June), date of submission by participants, date of data release, date of first announcement of the Challenge.
  • Update the list of data providers, to keep the historical record complete
  • Update the Roadmap

Participating teams

  • Write the call for participation, probably based on one from a previous year. Create a PDF version.
  • Write the rules for the challenge and publish on the SynSIG website.
  • Decide on the entry fee and method of payment. The University of Edinburgh is able to accept online credit card payments, so this should be your default choice. In this case, agree with Simon King how the received funds will be used (see also Listening test)
  • Publish the call on the SynSIG website and send to appropriate mailing lists: blizzard-discuss, synsig, Festival, HTS.
  • Receive team registrations via the blizzard@festvox.org alias
  • Create a shared Google spreadsheet to store team information (you could ask Simon King to create this from a previous year's version, as a template)
  • Issue data download passwords to teams who have both registered and completed the data license (see also Data)

Listening test

  • Depending on the language, the listening test facility at the University of Edinburgh is a good choice for running around 100-150 "gold standard" listeners under ideal conditions. This will cost around GBP 1200 for listener payments, plus around GBP 1500 for an assistant to recruit and run the listeners (and more, if you want this assistant to do other tasks such as building the listening test or running the analysis).
  • Decide whether you also want to use paid crowd-sourcing to obtain further listeners. You might decide to only use crowdsourcing and have no "in person" listeners. It's your choice.
  • Ask Simon King to approach sponsors to cover some of the costs of the listening test, including payments to listeners and to assistants running the test. Agree with him how funding will be used (e.g., do you need money to pay your own assistants or to pay for crowd-sourcing?). Normally, sponsors make a donation to the University of Edinburgh, and funds are then distributed as needed.
  • Run the listening test
  • Run the statistical analysis
  • Distribute results to teams. Expect at least one team to query the results, and to have to double-check them (usually, we are right and the team is wrong, but we always check anyway)

Concluding tasks

  • Write a summary paper
  • After the final challenge using your data, make it generally available (if allowed by the data owner)
  • If your tour of duty is complete, have yourself removed from the blizzard@festvox.org alias

Workshop

Simon King is usually willing to organise this, so try to delegate this task to him!

  • Choose a date and location for the workshop. The default is a satellite of Interspeech or SSW.
  • Find a local helper for the logistics. If you are lucky, the SSW or Interspeech organisers will help you.
  • Book the venue (always ask your local helper to visit it first to check it is suitable). If you are lucky, then using the same venue as SSW or Interspeech may be possible (although Interspeech venues are usually expensive conference centres, so this is rarely a good option).
  • Find a sponsor for the workshop to cover: venue cost, event catering and evening drinks.
  • Book a venue for the evening drinks.
  • Get the workshop approved as an official satellite of the larger event
  • Decide how to handle paper submissions and reviewing. Normally, this is informal and done by the organising committee, but sometimes the larger event wants a more formal procedure.
  • Decide how papers will be published. The default is festvox.org
Retrieved from "http://synsig.it.helsinki.fi/index.php?title=Blizzard_Challenge_organisation&oldid=2780"