by Mary E. Beckman & Gayle M. Ayers
More on the Tone Tier | More on the Break Index Tier
The Miscellaneous Tier | Bibliography
The ToBI Annotation Conventions
How this HTML document was created
What are the "Guidelines for ToBI Labelling"?
ToBI (for Tones and Break Indices) is a system for transcribing the intonation patterns and other aspects of the prosody of English utterances. It was devised by a group of speech scientists from various different disciplines (electrical engineering, psychology, linguistics, etc.) who wanted a common standard for transcribing an agreed-upon set of prosodic elements, in order to be able to share prosodically transcribed databases across research sites in the pursuit of diverse research purposes and varied technological goals. Silverman et al. (1992) describe the motivation for and development of the ToBI system. If you ask for this handbook in hard copy, that paper will be appended as Appendix B. Appendix A (which is included both in the hard copy and in the ASCII file version of this labelling guide) is "The ToBI Annotation Conventions", the definitive summary statement of the symbols and marks used in ToBI transcriptions, and of the conventions that we have agreed upon for their use. The rest of this labelling guide is a more detailed description of the system, with reference to accompanying utterances of two types: example utterances to illustrate points made in the text and exercise utterances to give labellers practice on the points made in the text. These utterances are set off in the text of the labelling guide using the following typographic conventions.
EXAMPLE <<basename>>: orthographic transcription tonal transcription and/or break index values
EXERCISE <<basename>>: orthographic transcription
The utterances that accompany this labelling guide can be obtained in two formats: as digitized computer files (with electronic record of the f0 contour) or as an audio tape (with paper record of the f0 contour). These utterances and this labelling guide were developed at the Ohio State University with partial support from the National Science Foundation, and New Mexico Institute of Mining and Technology has generously provided a distribution site for the electronic records (see Section 0.2). They are available to any interested user, but only for non-commercial use. The National Science Foundation, the Ohio State University, and New Mexico Institute of Mining and Technology make no warranty and accept no liability associated with the use of these materials. These materials may be obtained only as described in Sections 0.2 and 0.3, and are not to be redistributed by other user sites. Users may not redistribute these materials from their own sites, but should instead tell interested people how to obtain their own copy from the distribution sites.
Getting and using the digitized utterances and f0 tracks
If you have waves(tm) (an Entropic Research Laboratory product) or a similar computer display system, obtain the speech files, electronic record of the f0 contour, and label files by ftp from Colin Wightman at the New Mexico Tech distribution site.
There are two options for obtaining the ToBI materials depending upon how much disk space users have available. For those with sufficient disk space there is a single large tarfile for convenience. This option requires about 40 MB available during the installation process; the full materials occupy about 20 MB once the installation is completed and the tarfile is removed. If you do not have enough space to have both the complete tarfile and all the installed files at the same time, use the second option. There are four smaller tarfiles which together contain all the materials contained in the single large tarfile. The Guidelines for ToBI Labelling, scripts, and tools are contained in the "essential" tarfile, and the speech files, f0 records, and label files are divided into three parts by order of occurrence in the Guidelines.
On your home system, enter the command: "ftp kiwi.nmt.edu" (or use the internet address for kiwi, which is 22.214.171.124) from the directory where you would like to have the materials (the installation process will create a directory called TOBI-TRAINING where all the files will be put). When prompted, type the login name "ftp" and your user name on your home system as a password. If you now list the files available ("ls"), you will see several directories as listed below. Users should feel free to explore these directories.
docs - contains documentation files such as this labelling guide. pub - used for temporary storage of files uploaded by users. tarfiles - contains compressed sets of files for easy retrieval. tools - contain shellfiles and tools for transcription. utts - contains utterances and transcriptions for reference.Further descriptions of the ToBI ftp site at kiwi.nmt.edu and some guidelines can be found in the README file. If you run into serious problems send email to Colin Wightman at email@example.com.
All the files for the training materials have been placed in one large tarfile and in four smaller files as described above. To get it/them, change to the tarfile directory ("cd tarfiles"). Since tarfiles are binary files, enter the command "binary". You can transfer the complete materials by typing "get complete_tobi_release_2.0.tar.Z". When the transfer is complete, return to your home system with the command "quit". Alternatively, to get the smaller files with part of the training materials only, type "get tobi_release_2.0_essentials.tar.Z" This will give you the Guidelines for ToBI Labelling, and the scripts and tools that you will need. Then to get the actual examples and exercises speech files, f0 contours, and label files, get one or more of the "part" files. You can take as much as you have room for and then delete things after you are through with them to make room for the next set of examples and exercises to work through. Part 1 has the files for Section 1 to Practice 2, Part 2 has the files for Section 2.6 to Practice 4, and Part 3 has the files for Section 3.2 to Practice 6. Transfer these by typing "get tobi_release_2.0_part1.tar.Z", "get tobi_release_2.0_part2.tar.Z", and "get tobi_release_2.0_part3.tar.Z" respectively. When you have all the files you want, return to your home system with the command "quit".
Back on your home machine, make sure the tarfile(s) is/are in whatever directory you would like the ToBI files to reside under. The installation process will create a directory underneath the one in which you put the tarfile(s) and all the ToBI material will go in the new directory. To install the files, first enter the command "uncompress complete_tobi_release_2.0.tar.Z" (or uncompress the "essentials" and "parts" files). Once the file(s) is/are uncompressed (it will take a few minutes), enter the command "tar -xvf complete_tobi_release_2.0.tar" (or tar -xvf the other .tar files to extract all of the subdirectories and files). Don't forget to delete the tarfile once you're happy that everything got installed correctly.
Summary instructions are listed below.
1) ftp kiwi.nmt.edu (or ftp 126.96.36.199) Name: ftp Password: username on home system 2) binary 3) cd tarfiles 4a) get complete_tobi_release_2.0.tar.Z or, if disk space is a concern, chose a relevant subset of 4b) get tobi_release_2.0_essentials.tar.Z (Guidelines for ToBI Labelling, scripts, tools) get tobi_release_2.0_part1.tar.Z (Section 1 to Practice 2) get tobi_release_2.0_part2.tar.Z (Section 2.6 to Practice 4) get tobi_release_2.0_part3.tar.Z (Section 3.2 to Practice 6) 5) quit 6a) uncompress complete_tobi_release_2.0.tar.Z or relevant subset of 6b) uncompress tobi_release_2.0_part1.tar.Z uncompress tobi_release_2.0_part2.tar.Z uncompress tobi_release_2.0_part3.tar.Z 7) tar -xvf complete_tobi_release_2.0.tar (or relevant tobi_release_2.0 parts) 8) rm complete_tobi_release_2.0.tar (or relevant tobi_release_2.0 parts)The directory structure that you should recover by the untarring all the files is described below. The top level directory is TOBI-TRAINING. It includes an ASCII version of this labelling guide (called "labelling_guide_v2.ASCII"), two subdirectories called EXAMPLES and PRACTICE (which hold the speech, f0, and label files), and two script files called "examples" and "exercises". The last two are waves(tm) scripts for displaying the examples and exercises and for labelling the exercises. These two scripts assume this directory structure. Non-waves(tm) transcriptions of the examples are given in the ASCII file Nonwaves-transcriptions, and more detailed instructions about how to use the scripts and useful shortcuts for the mechanics of labelling using the waves(tm) scripts are given in README-transcriber.
TOBI-TRAINING EXAMPLES (where speech, f0, and "answers" are kept) AND1.breaks (break index label file) AND1.d (speech) AND1.f0 (f0) AND1.misc (misc label file) AND1.tones (tones label file) AND1.words (words label file) . . . PRACTICE (where user transcriptions are kept) I-mean.breaks I-mean.words I-mean.tones . . . examples (script for displaying examples and "answers") exercises (script for practice labelling) labelling_guide_v2.ASCII (ASCII version of Guidelines and Conventions) Nonwaves-transcriptions (ASCII version of non-waves(tm) labelling) README-transcriber (instructions and shortcuts for scripts)After you have untarred the files, you will have to type a few commands to use the utterances as intended. To make the scripts executable and to protect the speech files and "answer" label files which are kept in the directory EXAMPLES from being overwritten by mistake, type the following three commands at the unix command line from within the TOBI-TRAINING directory.
chmod +x examples chmod +x exercises chmod -w EXAMPLESThere are also a few other tools included in the complete tobi release and the "essentials" file. These are compressed tarfiles which must be uncompressed and untarred (as above) to be installed. They are not strictly necessary for working through the Guidelines for ToBI Labelling, but are helpful tools to have. "cardinals.tar.Z" contains the files necessary for displaying cardinal examples of ToBI label categories, "transcriber.tar.Z" contains the files necessary for transcribers to transcriber their own data, and "checker.tar.Z" contains the files necessary to invoke the John Pitrelli's checking program, which checks transcriptions and reports errors.
"cardinals.tar.Z" allows the user to display and play cardinal examples of ToBI label categories by pushing buttons in a menu display. If this tool is installed, the button menu with these examples is called up automatically each time the "examples", "exercises", or "transcriber" (see next tool description) scripts are invoked. Read the README-transcriber file for more information. Install by uncompressing and untarring as described above.
cardinals.tar.Z (for displaying cardinal examples of ToBI)When uncompressed and untarred yields:
README-transcriber aux_examples/ (additional examples of ToBI labelling) cardinals docardLabellers who wish to label their own data should use the script "transcriber" which is included in "transcriber.tar.Z". This uses the same format as the training materials. Cardinal examples are available if they are installed. Read the README-transcriber file for more information. Install by uncompressing and untarring as described above.
transcriber.tar.Z (for transcribing your own examples)When uncompressed and untarred yields:
breakindexmenu miscmenu tonemenu transcriber (script for doing transcriptions) wordsmenuLabellers should check their transcriptions of their own data to see that they have a "legal" ToBI transcription. The script "check-transcription" checks the label files for consistency and adherence to the ToBI conventions for labelling. Read the README-checker file for more information. Install by uncompressing and untarring as described above.
checker.tar.Z (for checking transcriptions)When uncompressed and untarred yields:
README-checker check-and-behead-breaks.awk check-and-behead-misc.awk check-and-behead-tones.awk check-and-behead-words.awk check-transcription (script for checking transcriptions) check-transcription.awk
Now you are ready to use the two waves(tm) scripts. Both of them take as their argument(s) the basename(s) of the utterance(s) you want to display. For example, as you are reading about example utterance <<jam1>> in the labelling guide, you can listen to the speech and look at the associated transcription by typing:
examples jam1This will call up xwaves with two data windows to display the speech waveform and f0 trace, and a third window for the ToBI labels of our "answer" transcriptions. This script is set up to just display the information and play the speech; it does not allow the user to change the labels.
In order to practice transcribing one of the example utterances (say the first exercise in PRACTICE ONE), type:
exercises amelia-p2This will display the speech waveform and f0 with only the word labels and placeholders for the break index labels in the labelling window. You can then use the labelling menus to add the tones and substitute break index values for the place holders in the break index section of the label window.
Both of these scripts can be used to display several utterances in a series. For example, to queue up the first four examples in the Guidelines, type:
examples jam1 cough made1 made2After you have finished looking at <<jam1>>, push the CONTINUE button in the waves(tm) control panel. The example <<cough>> will then be the displayed.
A note on where the label files are stored: The "answer" label files (basename.tones, basename.breaks, basename.misc) are kept in the directory EXAMPLES along with the speech, f0, and word labels files (basename.d, basename.f0, basename.words). The label files that the user creates when labelling with the script "exercises" are stored in the directory PRACTICE.
A note on multi-transcriber sites: Setting up a site to be a multi-transcriber site is fairly straightforward. The main idea is that instead of all labellers working with the "exercises" script and having their labels saved in the directory PRACTICE, each labeller will have a personal "exercises" script and directory where the labels will be stored. Take the name USER as our demonstration (where any name can substitute for USER).
For each user, make a separate directory within the top level directory TOBI-TRAINING, copy the "exercises" script to a personal copy for the user, edit the "user" script, and then invoke the script "user" exactly as one would invoke the "exercises" script (make sure the "user" script is executable -- "chmod +x user" if necessary.) Additionally, one may want to copy the break index placeholders into the USER directory.
1) mkdir USER 2) cp exercises user 3) Change line in script "user" which says: PRACTICEDIR=PRACTICE to say: PRACTICEDIR=USER 4) cp PRACTICE/*.breaks USER/ 5) chmod +x user (if necessary) 6) E.g., start Practice 1 by typing: user amelia-p2
To get an audio tape of the utterances and a printed paper copy of the labelling guide and f0 tracks, send your request along with a check for $25.00 made out to The Ohio State University to:
ToBI Labelling Guide, c/o Mary Beckman Ohio State University, Linguistics Dept. 222 Oxley Hall, 1712 Neil Ave. Columbus, OH 43210-1298 USA(The $25.00 just barely covers the cost of making copies of the tape and booklet and of the mailing to a North American location.)
On the audio tape each utterance is played twice in a row. The utterances occur in the order that they are listed in the Labelling Guide text, with two more repetitions if an utterance is mentioned again later in another section. However, it is far easier to use the utterances if you can play each one as many times as you want and if you can zero in on some section of an utterance at will, and we recommend that you find some way of doing so. For example, you might use a tape recorder with a recordable tape loop device (and loops of several lengths). Or if you have a Kay DSP 5500 or some other kind of computer system with fast A/D capabilities, digitize each utterance from the tape into a buffer where you can play repeatedly while looking at the paper record of the f0 contour and the accompanying labels.
To transcribe the example utterances, you will need to use the non-waves(tm) conventions described in Section 9 of The ToBI Annotation Conventions. The last section of the booklet is a listing of an ASCII file containing the non-waves(tm) format labelling of each example and exercise utterance corresponding to the waves(tm)-format labelling displayed on the sheet with the f0 contour. They are given in alphabetical order by basename. This ASCII file and another ASCII file containing the orthographic labels and field placeholders of all of the exercise utterances file can be obtained by anonymous ftp from Colin Wightman by doing the following:
On your home system, enter the command: "ftp kiwi.nmt.edu" (or use the internet address for kiwi, which is 188.8.131.52). When prompted, type the login name "ftp" and your user name on your home system as a password. If you now list the files available ("ls"), you will see several directories:
pub - used for temporary storage of files uploaded by users. docs - contains documentation files such as this labelling guide. tarfiles - contains compressed sets of files for easy retrieval. tools - contain shellfiles and tools for transcription. utts - contains utterances and transcriptions for reference.To get them, change to the docs directory ("cd docs") and transfer the files by entering the commands "get Nonwaves-transcriptions" and "get Nonwaves-exercises-templates". When the transfers are complete, return to your home system with the command "quit".
If you have comments on this Labelling Guide -- particularly, if you have suggestions for improvements or better example utterances you would like to give to us -- we would be very grateful if you would direct the commments to us at:
e-mail: firstname.lastname@example.orgThis e-mail address is also the place to send us your e-mail address if you want to be added to our list of "subscribers" to be notified of any future editions of the Labelling Guide.
other-mail: ToBI Labelling Guide, c/o Mary Beckman Ohio State University, Linguistics Dept. 222 Oxley Hall, 1712 Neil Ave. Columbus, OH 43210-1298 USA
The ToBI labelling system was originally developed to cover the three most widely used varieties of spoken English -- namely, general American, standard Australian, and southern British English. We do not claim to cover other varieties. Indeed, we have already determined that ToBI proper does not adequately cover many other British varieties such as the Glasgow dialect, and modified variants need to be developed by users who want to use it in transcribing utterances in these other dialects. By the same token, we must stress that ToBI was not intended to cover any language other than English, although we endorse the adoption of the basic principles in developing transcriptions systems for other languages, particularly languages that are typologically similar to English. More general comments about using the ToBI system for other dialects of English or about adapting ToBI labelling principles to develop comparable systems for the transcription of other languages may also be addressed to the tobi e-mail address listed above for forwarding to appropriate interested members of the larger ToBI group.