Blizzard Challenge 2016 Rules

DATABASE ACCESS

• After registration and completion of the required licenses, download passwords will issued, as described on the main Blizzard 2016 page.

REGISTRATION FEE

• A registration fee of 500 GBP is payable by all participants who wish to submit synthetic speech for evaluation, to offset the costs of running the challenge, including paying local assistants and listeners. The fee must be paid by Friday 29th April 2016. You can pay this fee using Edinburgh University's online payments system at http://www.epay.ed.ac.uk/browse/extra_info.asp?compid=1&modid=2&catid=42&prodid=2210 where you should register for the event called 'Blizzard Challenge 2016'. After doing this, please email blizzard@festvox.org to notify us that you have paid. If you are absolutely unable to use the online payments system, please contact blizzard@festvox.org for assistance with a bank transfer. However, we strongly prefer the epay system because it reduces the costs and admin work for us. If you must pay by bank transfer, please contact us in plenty of time (at least 3 weeks before the payment deadline); an additional fee of 100 GBP will be added for any payments not made using the epay system.

EXPERT LISTENERS

• Each participant should try to recruit at least ten volunteer listeners who have some professional knowledge of synthetic speech, for the evaluation tests.

NAIVE LISTENERS

• Each participant should try to recruit as many naive listeners (with no professional knowledge of synthetic speech) as possible
• The organisers would also appreciate assistance in advertising both the Challenge and the listening test as widely as possible (e.g., to your students or colleagues).
• We are also seeking assistance with conducting a listening test using children. These can be native speakers, or learners of English as an additional language. Children with reading ages of either 4, 5, or 6 years (approximately) would be most appropriate, but slightly older children would also be useful. Please contact blizzard@festvox.org to discuss this. We may be able to offer some financial support with the costs, or reduce the entry fee if you are also submitting synthetic speech.

MATERIALS PROVIDED

All participants will have access to the following material after signing the license:

• An estimated 5 hours of speech from one native British English female professional speaker (this includes the 2 hours released as the 2015 pilot data) from 50 children's audiobooks
• Publisher's text for all speech material, as either text or PDF files
• Cleaned-up text and sentence-level alignments with the audio, for part of the material
• A shared repository of cleaned-up text and alignments contributed by other participants (see below)

The material has been very kindly provided by Usborne Publishing

THE CHALLENGE

Participants involved in joint projects or consortia who wish to submit multiple systems (e.g., an individual entry and a joint system) should contact the organisers in advance to agree this. We will try to accommodate all reasonable requests, provided the listening test remains manageable.

Task

Build a voice from the provided data, suitable for reading children's audiobooks. There is just a single task, designated as 2016-EH1.

Sharing the work of data pre-processing

Some, but not all, of the material has been aligned with the book text. Because some of the books provided by the publisher are in PDF format, and the audio for each book is typically a sequence of audio tracks, it is non-trivial to extract a clean version of the text and then to align it with the speech (e.g., at sentence level).

Therefore, we ask all participants to collaborate and share the effort of creating clean, sentence-aligned transcripts of the speech. This will be co-ordinated by Sebastien Le Maguer. Please contact Sebastien before embarking on any manual cleanup of the text, or alignment with the audio, so that he can eliminate duplicate work.

The shared repository for cleaned-up text and alignments with the audio can be found at http://blizzard.coli.uni-saarland.de/root/blizzard_training_data/blob/master/README.md - please use the same username and password as for the data download.

Please note that any alignments of the text that you commit to the repository should be with respect to the publisher's original complete audio files, as distributed for the Blizzard Challenge 2016.

USE OF EXTERNAL DATA

• "External data" is defined as data, of any type, that is not part of the provided database.
• You are allowed to use external data in any way you wish, subject to any exclusions given in these rules
• Use of external data is entirely optional and is not compulsory
• You must use the provided audio files
• You must not use any additional speech data from the same speaker
• You may exclude any parts of the provided databases if you wish.
• Use of any provided segmentations, transcriptions or labels is optional.
• If you are in any doubt about how to apply these rules, please contact the organizers immediately.

SYNTHESISING THE TEST EXAMPLES

• The exact nature of the test set will not be revealed in advance, but is likely to include both sentence, paragraph and short book-length texts from a similar domain to the provided corpus, as well as texts from other domains.

• Synthetic speech may be submitted at any standard sampling rate (but always at 16 bits per sample). Waveforms will not be downsampled for the listening test.

RETENTION OF SUBMITTED SYNTHETIC SPEECH SAMPLES

• Any examples that you submit for evaluation will be retained by the Blizzard organisers for future use.
• You must include in your submission of the test sentences a statement of whether you give the organisers permission to publically distribute your waveforms and the corresponding listening test results in anonymised form. In the past, all participants have agreed to this and we strongly encourage you to give this consent.

LISTENING TEST

Formal listening tests will be conducted to evaluate the synthetic speech submitted. Whilst the task is to synthesise speech suitable for reading an audiobook to children, the listening test will likely also evaluate the performance of the voice in terms of naturalness and intelligibility on other types of material (i.e., as in most previous Blizzard Challenges).

PAPER

• Each participant will be expected to submit a six-page paper (using the Interspeech 2016 template) describing their entry for review. Please email your paper to blizzard@festvox.org .
• One of the authors of each accepted paper should present it at the Blizzard 2016 Workshop
• In addition, each participant will be expected to complete a form giving the general technical specification of their system, to facilitate easy cross-system comparisons (e.g. is it unit selection? does it predict prosody? etc. etc)

HOW ARE THESE RULES ENFORCED?

• This is a challenge, which is designed to answer scientific questions, and not a competition. Therefore, we rely on your honesty in preparing your entry.