Project Listen (capstone proposal)
For my final project, I’ll be working with Jack Mostow and the Project LISTEN team (http://www.cs.cmu.edu/~listen/). They’ve developed a software called the Reading Tutor, a software that supplements the individual attention of teachers to teach children how to read better. For my final project, they’ve asked me to create some design sketches that explore a new feature to perhaps eventually incorporate into the Reading Tutor software. The basic question I will be investigating is: Can visual cues and game-mechanisms help children read more fluently, in real-time?
The software they’ve already developed does some speech analysis that assesses the prosody of a child’s spoken reading. Prosody is the stresses, pacing, and changes in intonation within spoken language. Fluent & non-fluent readers speak with measurably different prosody.
My responsibility is to use canned speech samples of both fluent adults and non-fluent children, and the speech analysis results of these samples, to create some design explorations. I’ll investigate if kids can read a sentence more fluently (with the correct prosody) if they are given some visual cues as to how to say the sentence. I’ll create a few (goal: 3 to 4) sketches exploring both different visualization techniques of prosodic features, as well as different gameplay mechanisms to engage children & encourage improvement.
Eventually, I will work with the team to go from canned samples to real interactivity. I’d love to have this interactivity possible by the time of the final show, but that will depend on the ease of integrating my code into their software, which is uncertain. User testing will be done with children in pittsburgh schools over the summer.
David – I’m really glad you’ve picked up on this opportunity, and I’m just totally delighted you’ve made such quick progress and lovely early results. KEEP GOING!
comments from the piratepad
================================================
How about some smoothing, and some inertia on the plane.
How about an animation of the plane “waiting” (perhaps twirling in circles?) while the kid is pausing?
How about the circles (clouds) evaporate or “pop” when the plane passes through them?
In openFrameworks, you can set the circle resolution (currently set to 20 pie segments); I recommend increasing the circle resolution to 40 or 64
Regarding the computing of smoothed curves — try a running average of a running median within some window. The running median will help eliminate the extreme outliers.
What about enhancing the display of the next target cloud — having it vibrate or throb, or grow in size, or have a wiggly edge.
In my English pronounciation class we’re acctually asked to do the paper plane motion on hand, to help smooth the intonation. but those guestures were more dramatic; going up and down with steep slop.
for lines, if you draw them in bigger lineweight, and add blur, would that visually smooth out lines?
I don’t see how exactly this changed from last time. I think what would really help for us is a project statement that clearly shows “this is where this project used to be at” (before you began working on it for this class) and “this is where I want to go”. Otherwise for projects like this where you’re not starting from scratch, it’s really hard for the rest of us to evaluate.
–The speech analysis algorithms are not mine and were done before I started on the project, but all that I’m displaying is my work and I started it at the start of this project, just to clear things up -david
I can see that he’s ported it to openFrameworks and that he processed the XML files to display but I’m a little unclear what the end goal of the project is.
for the curve, you might be able to just average adjacent points and then use ofCurveVertex to create something smoother looking.
If you want some oF code for smoothing or resampling lines, let me know – KDDW
I think you may want to think more about the game play. College students have about the mentality of children, so having them play it would be a good test sample. I would think about the reward structure, what happens if they hit a balloon, what if they miss. Can they try again? Does it adjust for the pitch/tone of their voice (would someone with a very deep or very high voice just always lose?)