## **Manual for Coders Tagging CDS/ADS Segments** ## ---------- ## Introduction ## You will be given a set of conversational blocks to label. These blocks are chunks of speech that LENA has labeled as being a conversational exchange (a set of speech segments) from one or more speakers bounded by a silence of at least 5 seconds on each end – in other words, what LENA thinks is a conversation! Each of these conversations is further broken down into a set of labeled speech segments (utterances, roughly speaking) and non-speech segments (noises, etc.). You will find that LENA’s chunking and labeling process is far from perfect. Just do the best you can to label these as CDS/ADS, following the instructions below. Your task as a labeler is to: 1) Judge whether the segment sounds like speech that is directed to an infant or young child (child-directed speech; **CDS**) or another adult (adult-directed speech; **ADS**). If it's neither CDS nor ADS (e.g. mixed talkers, or just noise, or just baby sounds, or silence) call it **junk**. Some examples: - [ADS FAN/MAN clip][1] : code as ADS - [CDS FAN/MAN clip][2] : code as CDS - [Television FAN/MAN clip][3] : code as junk - [Toy FAN/MAN clip][4] : code as junk - [Muffled FAN/MAN clip][5] : code as junk 2) Judge whether the speaker was a **male** or **female**. You will label only those segments labeled by LENA as FAN or MAN (Female/Male Adult Near). You should ignore any segments that are labeled as anything other than FAN or MAN (the IDSlabel program won’t let you label them anyway.) **Note**: sometimes you can't tell the CDS/ADS/Junk or Male/Female distinction from just the FAN/MAN segment you're listening to. You can use what you hear in the rest of the block when making this decision. In addition to this labeling, you will also play a critical role in determining whether the block is suitable to share with a broader set of users. Blocks that contain highly identifying or sensitive material should be tagged as such, so that we can protect the participants by avoiding sharing those portions of the data. ## Installing and Setting up IDSLabel ## 1. create a folder called **idslabel-v1** 1. place the idslabel executable you download from [this](https://github.com/SeedlingsBabylab/idslabel/releases/tag/v1.0.0) page into that folder. The file titled idslabel_v1.0.0_osx.zip (under Downloads) contains the Mac OS X executable. The "Source code (zip)" contains the code if you're running idslabel.py from the command line. - if you're running idslabel as a script from the command line, the folder it's already in is good enough, no need for a new folder. 1. copy your lab's config.json file into that folder - if you're running idslabel as a script from the command line, copy the config.json into the folder that contains the idslabel.py file. **DO NOT** separate the idslabel.py file from the other .py files in this folder. All the .py files need to be in this folder. - make sure the config.json has 092616 in the filename. This is the date the config.json was made. It's the latest thus far. 1. create another folder and call it **idslabel_blocks**. - this will be your clips directory. **DO NOT** move or delete anything in this folder. This is where idslabel will download blocks and load them from. It will delete blocks when you're done with them, so no need to worry about filling up hard drive space. Let idslabel deal with everything in this folder. 1. If you're using the executable, double click it to start the program. - if you're running the script from the command line, run: ```$: python idslabel.py``` from the command line, within the idslabel directory which contains all the source code. 1. Type your name into the CODER_NAME box **and press enter**. - it will ask you to set the clips directory. - navigate to the **idslabel_blocks** folder you originally created in step 4, and choose that - idslabel will automatically load the config.json file that you placed alongside the executable in step 3. - if you've never used this name before it'll say that the server doesn't recognize this user as belonging to that lab, and that you should add it by going to File -> Add User to Server - do that 1. You should be ready to go. The rest of the usage is the same as before. There are a bunch of new features though, listed below. ### **note** The server is now hosted in Merced, California, so it might take considerably longer for blocks to reach you when you request them (and basically every other network operation in idslabel). Here in Durham, NC it takes 6-10 seconds to get a group of 3 blocks. ## Basic Program Use ## 1. **Open IDSLabel**. Open up the IDS Label program on your computer. (You would do this by double clicking the icon, or launching it from the command line.) *(See 'Open.JPG' in Files folder named "Screenshots".)* 2. **Coder Name**. Type your name in ***CODER_NAME*** box and **press enter**. 2. **Clips Directory and User**. Follow directions of the pop-ups that follow. - The first one will ask you to *set the clips path* to the blocks directory. This is the directory where the conversation blocks will be downloaded. (This should be the **same** directory each time you run the application.) i.e. **idslabel_blocks** directory created in step 4 of previous section. - Do **not** delete the contents in that blocks directory. The IDS Label program will delete them automatically after you submit or send back the blocks. - If you're running this for the first time, the next thing that should pop up is a message saying: "User "xyz" doesn't exist on the server". - In the menu, go to **File > Add User to Server**. This will register that username as being a member of your lab. - Next time when you type that name into the CODER_NAME box, the server will recognize it as a registered name. 4. **Get some data!**. Once completed, you may click '**Get Training Blocks**', **'Get Reliability Blocks'** or **'Get Blocks'**. - for *Training* blocks read directions here: [IDSLabel Training Blocks][6] - for *Reliability* and *Real* blocks: This will ask the server to send X number of blocks to your computer - the default "# blocks" is **3 blocks.** - if you change X you **must hit enter** or you'll still just get 3! 5. **Load Block**. Now there should be a list of blocks sitting in the small box labeled "Load Block". Double click one of these blocks to load it. - Blocks that were reloaded from previous sessions will have "old" next to them. Blocks you have received in this current session will have "new" next to them. ! *(See 'Load Blocks,JPG' in Files folder named "Screenshots".)* 6. **Listen & Label**. You can now select clips within this block to play and decide what they should be labeled as. (*See the next section called, "Primary Criteria to Use to Decide Between CDS/ADS" and "Steps and Considerations"*) - Make sure to **listen to the whole block first** by clicking "Play Block" or press shift+spacebar. - Then click/highlight the desired FAN/MAN clip, press "Play clip" or press spacebar. Then **label** it as **CDS (c), ADS (a), or Junk (j), and if it was male (m), or female (f)** ! *(See 'Shortcuts.JPG' in Files folder named "Screenshots".)*. Shortcuts are also available in the program itself, under the Help menu. - 7. **Submit + Save**. Once you label every FAN/MAN clip within a given block, press the "**Submit + Save**" button (or shift+enter). This will submit your classifications to the server, delete the block's data from the blocks directory, remove it from the "Load Block" list, and save its contents to a session.csv file you need to specify. - You must **label all FAN/MAN clips** to submit a block. - You must select whether a block can be shared in order to submit it (see **vetting** section below) 8. **Save**. When you press 'Save Block', the program will prompt you to title your local csv, and save that block's labeled data to the csv. - you should name it with a user and date specific convention e.g. **elika_6_11_16_rel.csv** - every session should have a csv for each user (otherwise you overwrite the content) - you can **open this csv in excel** to make sure it's saving your information properly and file an [issue on github][7] 9. **Other Saving Methods**. This is mostly for PIs: - go to **file/Get Lab Info**![enter image description here][8] - click the relevant save button, and save a csv with your results, e.g. elikalab_reliability.csv. - **save reliability blocks**: for your reliability blocks - **NB** PI save functions will be updated after v.0.9. 10. **Monitoring Progress** - You can monitor the progress of the whole system (i.e. all the labs) by looking at the output of "Save All Blocks" from the Get Lab Info page. To see the progress of a single user (who's part of your lab), there's an easier way. Just double click on the user's name in the first column of the Get Lab Info page. A box should pop up showing the current counts of all the blocks that this user has submitted.![enter image description here][9] ## Primary Criteria to Use to Decide Between CDS/ADS ## 1. If the segment sounds like it is directed to an infant or child under 8 years of age it should be tagged as CDS. We are forcing you to make a binary choice here, but in real life CDS/ADS is more of a continuous variable. So sometimes you will really be saying that a segment sounds more like CDS than like ADS with your codes. Because this is a “forced choice”, you need to make a choice even if you are really unsure. Just make your best guess. 2. You may use context as well as acoustic and prosodic (melodi/rhythmic) characteristics to help your decision-making. Some common characteristics of CDS include higher pitch, greater pitch variability, and slowed speech. CDS usually also has a “happy” sound and/or sounds very loving. 3. Don’t expect that all adult talkers will use the same way of talking for CDS, but do trust your ears. You are likely better at making this distinction in speech register than you think you are and certainly better than any machine! Keep in mind that: - Some adults may just raise their pitch, but not make it more variable when addressing children. - Some adults may lower their pitch or use whispered speech when addressing children. - Some adults may pronounce speech in a funny way (alter certain phonemes) when addressing children. - Some adults may do things we haven’t mentioned here. You can ask your PI or staff for clarification. 4. As a secondary consideration, ask yourself who the participants are based on what you learned from listening to the block and the content of what is being said. This information may help you to determine the addressee if you are unsure. - Use this whole block context to guide you in borderline cases. Consider whether the whole block "feels like" CDS or ADS, and classify borderline cases accordingly unless there is a clear register shift that distinguishes an utterance from the rest of the speech by that speaker in the block. - You should code CDS whenever the acoustics strongly suggest it even when the speech is not actually directed to a child. (E.g., if an adult talks to a dog or to another adult the way they would talk to a baby, this should be tagged as CDS.) ## Vetting Guidelines for Sharing ## Sometimes material that shows up in the recordings should be removed because it is private or compromising information. If this happens we would like you to tag it as such, so we can make sure that portion of the recording does not get shared widely. However, you do still need to label all blocks, including the ones that need to be vetted. The program will not let you submit the block otherwise. Note that even if you mark it as **do not share** we may use the labeling data for some purposes, so please label them accurately. There are 2 ways to **not share** a block: - click the **'do not share' checkbox** under # blocks - click **yes/no when prompted** with 'Is block ok to share? ! *(See 'Share.JPG' in Files folder named "Screenshots".)* Material should be excluded if there is a possibility that the information could breach confidentiality and/or be used maliciously against the person. In most circumstances, embarrassing or unflattering statements can be left in. However, if you run across something that you think really shouldn't be heard by strangers, please err on the side of marking it **do not share**. Some examples of what should be marked are listed below. This is not a complete list, so in questionable cases err on the side of marking. Please be conservative: "When in doubt, hold it out." - Last/family names (first names, including uncommon ones, are okay) - Addresses or other contact information - Birthdates - Social security, credit card, etc.. numbers - Serious discussion of illegal activity (if they are clearly just joking, this is fine in most cases) - Speech that is in another language if you can't understand what is being said (e.g., "hola" is fine to include, but some sentence whose content you can't understand should be excluded). ## Steps and Considerations ## - First *listen to the whole block from start to finish*, but don’t tag anything yet. Ask yourself: - Does this block contain material that should be *removed* based on our vetting criteria? If so, mark it as such. Proceed without coding that block. - How many participants are there? - Who are the participants (adult vs. child/infant)? - Outsiders, such as cashiers, can still be labeled. *But be careful!* Cashier-speak may sound like CDS, but should be coded as ADS: [Cashier-Speak Block][10] code as ADS - Is the conversation as a whole directed toward an infant/child or toward an adult, or a mixture of the two? - Then listen to each FAN and MAN speech segment in turn and ask yourself: - Is there a single adult speaker? - If so, tag the speech as CDS/ADS for that speaker following criteria described above. - If not, is there a foregrounded adult speaker? This is a speaker who is easier to perceive than other talker(s) or noises in the segment and/or who is the speaker for the bulk of the segment if there is a succession of speakers in the segment. - If so, tag the speech as CDS/ADS for that foregrounded speaker - If there is a succession of non-overlapping/foregrounded speakers who are consistently using CDS/ADS, you should also label the segment ADS/CDS, as appropriate. - If none of the above criteria apply (i.e., there is no speaker, the speaker(s) is/are an infant or child under 8 years of age (Ex. [Child FAN/MAN clip][11]), the utterance is not in English, or there is no identifiable single speaker), do not tag the segment as CDS/ADS, but tag it as junk (you will use the j on your keyboard for this code; see 'Help' > 'Show Shortcuts' for keyboard shortcuts) - You will only tag a segment with overlapping speech as junk if the segment you listen to is really mixed (e.g., 50/50 or 30/30/30) with multiple simultaneous speakers and there is no discernable foregrounded talker. - [Baby Crying FAN/MAN clip][12] : code as junk - [Laughter][13] : code as junk (because it is mostly laughter with very little speech) - Special cases/considerations: - ***What counts as speech?*** Some kinds of noises that we make are not speech. These kinds of sounds should be classified as “junk”. How do you know if it’s speech? - If you hear an acoustic vowel, classify it as CDS/ADS - Ex.) Oooh, uhh, umm, weee! - [Ohh FAN/MAN clip][14] - [Awh FAN/MAN clip][15] - [Woo FAN/MAN clip][16] - If you don’t hear a vowel code it as junk - Ex.) Shh, kissy-sound, tongue- ‘click’, hmm, laughter - [Brr FAN/MAN clip][17] - [Kisses FAN/MAN clip][18] - [Mmm FAN/MAN clip][19] - [Shh FAN/MAN clip][20] - [Humming FAN/MAN clip][21] - [Sound Effects FAN/MAN clip][22] - ***What about reading and singing?*** Code these as speech. If the utterance has no vowel sound, label it as junk - [Reading FAN/MAN clip][23] : code as CDS - [Singing (boop) FAN/MAN clip][24] : code as CDS - [Singing (dun dun) FAN/MAN clip][25] : code as CDS - [Singing a Song FAN/MAN clip][26] : code as CDS - [Humming clip][27]: code as junk - ***What if a speaker switches from CDS to ADS or vice versa partway through a segment?*** If the bulk of the utterance is one register, code it that way. If it is really balanced, and you can’t make a judgment that it is primarily one or the other, select CDS. - ***What if someone is clearly speaking like they would to an infant or child, but it is VERY obvious from context that the target of the speech is an adult, themselves or a pet (or vice versa)?*** This happens! We are interested in coding the CDS and ADS registers, so if your subjective judgment of what it sounds like conflict with who the actual target of the speech is from context, go with how it sounds. In other words, if someone is speaking to another adult, but they are using the CDS register, code it as CDS. Some examples: - [Talking to Pets FAN/MAN clip][28] - Someone has been talking to an infant for a while, and then switches over to say something to the adult, but fails to change their register properly. - A caregiver is ostensibly saying something “to the infant” but really is just talking to herself out loud. e.g. “and then mommy’s going to pay the bills!” - see Files folder called 'Examples for Manual' > 'Talking to Themselves' However, sometimes people’s CDS just isn’t all that animated, or conversely, they may be speaking to an adult in an especially lively fashion (say, because they are communicating something they are very excited about). Less-lively CDS is still CDS and animated ADS is still ADS. Obviously there is a grey area between CDS directed at an adult, and lively ADS (and conversely, between ADS directed at a child, and flat-affect CDS). As with all the criteria above, you should go with your “best judgment”. - ***What if there is too little context for me to judge whether it is CDS or ADS?*** If you really can’t tell, even with the block context, code it as junk. However, if it’s unclear from the individual segment alone, but you can tell from the block context that it is one or the other, give it the appropriate CDS or ADS label. - ***What about phone conversation?*** Code the person on this end of the phone, but not the one on the other end (i.e. code them as Junk), even if you can make out what they’re saying and know whether it’s CDS/ADS. - [Phone Call FAN/MAN clip][29] : code as junk ## Canonical Examples to Illustrate Above Issues ## - linguistic sounds from television, phone, computer, radio, iPad, iPhone, electronic toys, electronic ‘reader’ books, whether intentionally directed to kid or on in the background - code as junk - non-linguistic sounds like musical toys, poppers, rainsticks, pianos, guitars, etc. - code as junk - coder cannot tell what’s happening because baby is on his/her belly where the mic is so there’s rustling or dampening of the input - code as junk - people recorded via telephone - coded as junk because it is electronic - when the MAN/FAN is actually a child - code as junk - soothing/other sounds that are quasi-linguistic (e.g. ‘shhhhhhh’, “hm”) (TBR) - These are junk without a vowel, but classifiable with a vowel - talking happening while baby is crying - based on foregrounding as above - singing/reading (TBR) - Codeable as long as there is a vowel - block context makes clear child is asleep - code as appropriate CDS/ADS/junk - adult is speaking CDS to another adult - code as CDS - talking to pets - code based on prosodic/paralinguistic features as CDS if appropriate [1]: https://osf.io/9wsfx/ [2]: https://osf.io/zqmt9/ [3]: http://osf.io/b8drt/ [4]: https://osf.io/5p8gm/ [5]: https://osf.io/rn6m9/ [6]: https://osf.io/d9ac4/wiki/IDSLabel%20Training%20Blocks/ [7]: https://github.com/SeedlingsBabylab/idslabel/issues [8]: https://osf.io/n5s4r/?action=download&mode=render [9]: https://mfr.osf.io/export?url=https://osf.io/gc4cq/?action=download&direct&mode=render&initialWidth=848&childId=mfrIframe&format=1200x1200.jpeg [10]: https://osf.io/dfyu3/ [11]: https://osf.io/5cnh6/ [12]: https://osf.io/wg4js/ [13]: https://osf.io/2sgft/ [14]: https://osf.io/g6u79/ [15]: https://osf.io/5h6x4/ [16]: https://osf.io/f7sak/ [17]: https://osf.io/2jx73/ [18]: https://osf.io/hfgp6/ [19]: https://osf.io/48kjy/ [20]: https://osf.io/cjts5/ [21]: https://osf.io/h9csp/ [22]: https://osf.io/hvste/ [23]: https://osf.io/p69xm/ [24]: https://osf.io/hnaub/ [25]: https://osf.io/ycdg9/ [26]: https://osf.io/q6s2n/ [27]: https://osf.io/h9csp/ [28]: https://osf.io/yjs8z/ [29]: https://osf.io/7q2cn/