Monday, September 25, 2017

It's 2017, and speech technology is still primitive

By Louise Kinross

I like our Dear Everybody campaign because the messages, like this one above, are real, and come from our children and families.

Most people assumes technology is an equalizer for people with disabilities, and in some cases it is.

But in the 20 years I’ve followed the use of voice devices, and more recently, voice apps that can be used on iPads, I’ve never seen a product that’s nimble, intuitive and fast.

In fact, you may have heard our Dear Everybody radio ad with Gavi, who uses a communication device. What you may not know is that Gavi had to pre-program her comments. She couldn’t express them live, because it isn’t possible to use a device at the speed people speak.

Indeed, “the process is often extraordinarily arduous and fatiguing for the person using the device,” writes McGill University scientist Gail Teachman in a study in Qualitative Inquiry this month.

Last week I had Marna-Rose Minett in my office. She’s raising her seven-year-old granddaughter Rayne, who has cerebral palsy. Marna-Rose noted that Rayne uses her communication device at school, but “we don’t use it much at home because we can understand her speech, and she has to go through so many different screens” to put words together on her device.

“It's slow,” Marna-Rose said. If she wants something to eat, first she presses ‘I’ and ‘want’ on the first screen, then she picks the icon for ‘food,’ then within food she has to choose between ‘breakfast,’ ‘lunch,’ ‘dinner’ or ‘snack,’ then she chooses what she wants.’” 

I likened this cumbersome process to a person needing to look a word up in the dictionary every time they wanted to express it. “Can you imagine if you wanted to say a word, having to flip through a dictionary and find it first?” I said. “And then, when you're finished with that word, you have to look up the next one?”

We both laughed wildly. But it wasn't “ha ha” funny, it was sad and despairing.

There is an app on the market—Speak for Yourself—which tries to keep the number of clicks needed to find a word to two. That’s an improvement.

But anyone who uses mainstream business or consumer technology would gasp in horror if asked to play around with the setup, capabilities and speed of these speech apps and devices.

That’s why a high number of children and youth abandon them. 

Recently, I was struck by how many people are instead using using a letterboard and spelling out words, letter by letter, which are then spoken by a partner, to communicate.

That’s how Japanese author Naoki Higashida, who has autism and is largely non-verbal, wrote his two books: The Reason I Jump and Fall Down 7 Times, Get Up 8.

In ABC’s hit Speechless, actor Micah Fowler communicates by selecting letters to form words with a laser pointer attached to his glasses.

One of our clients spoke to me the other day by pointing to letters to form words on his letterboard, while I spoke the words to confirm accuracy.

Over the years, I’ve seen scientists study why children and youth abandon voice devices. And I’ve always thought: “Why wouldn’t they?”

“I keep hoping that Apple will develop a voice app or device that is as intuitive and user-friendly as its other products,” I wrote in a blog in 2014. “However, I’ve been hoping that for more than 15 years. I even began a small campaign of tweets to Apple CEO Tim Cook last year that went unanswered. I think the big computer makers have let our families down in not bringing their expertise to the AAC table.”

Why is it okay, in 2017, that people with limited speech have to endure an “extraordinarily arduous and fatiguing process” as their only option for self-expression?

If current technology was sophisticated and speedy, why would a famous author choose to write his books by pointing to letters on a letterboard?

Why would my son prefer to use sign language, even though it drastically restricts who he can communicate with?

I have an idea for a scientist. I want a researcher to use a voice device for one week—24 hours a day—and not use the keyboard, because many young kids using devices can’t spell. Then, write a paper about how “easy” or “hard” it is to use, and how it compares to an Apple device.

I e-mailed my idea to scientist Gail Teachman, and this is how she responded: “I think a really important aspect of that researcher’s learning would be that NOT ONLY are devices slow, hard to use and clunky, but using an AAC device can be stigmatizing. The researcher would very likely experience social interactions where suddenly they are not seen as an expert, not smart…”

There would be “lots of being spoken to ‘like a child’ and an absence of the respect they are used to receiving from listeners. In short, what they have to say would suddenly be judged less valuable, less worthy of another person’s attention, less important and therefore, not worth the time it takes to listen.”

Why is that experience acceptable for children who struggle to speak?

Years ago, I spoke to the head of our research department about applying for one of Bill Gates’ Grand Challenges grants to develop an intuitive voice device. But the grant needed to create a life-changing solution for large markets of people around the world

The number of people using voice technology is relatively small. But it still
makes zero sense to me why Apple or Microsoft wouldn’t choose to enter the market for the goodwill they would generate alone. The media stories generated would be solid gold.

Gail Teachman is a postdoctoral fellow at McGill University with the Views on Interdisciplinary Childhood Ethics team. Her research examines the moral experiences of childrentheir views on what it right and wrong, good and bad, and just and unjustparticularly in the context of childhood disability. 


My son has used different AAC devices over the year and while the iPad has made a huge difference in how quickly he is able to use it, all your points are spot on! What he has now is easier, but it's still hard compared to somebody who can just say what they are thinking as they are thinking it. He resorts to 2-3-4 word phrases instead of sentences because it is faster and that reinforces many people's expectation/impression that he is not as smart as he really is. If you don't mind I am going to share this essay with my AAC group!

Love this article, Louise! I'm going to tweet it out to the universe. I was so happy to see an AAC user on TV when the sitcom, Speechless came out last year. I'm hoping the show will help bring awareness to the challenges AAC users experience in everyday communication. I'm right there with you in wanting a faster/easier system for Carter's communication. Come on Apple!! Help us out!

It's been a long time since Max has used anything but the Proloquo2Go speech app and his iPad, so i can't speak to other apps and devices. But I do know that he has gotten pretty quick at using it, and that it is relatively simple and also intuitive—e.g., all he has to do is type a couple of letters and couple of word options jumps up on screen for him to choose from. I have been excited about the possibilities of a speech recognition app in development, Talkitt. Perhaps the P2Go will evolve in that direction as well. What's missing for me from this post is acknowledgment that even though we have a ways to go in terms of communication devices, we have come such a long, long way when the only choice Max—now 14 years old—was a Dynavox that was too heavy for him to carry around and really hard to program.

It's an interesting conversation, and one worth having. I don't think it's a very complete one, though. For example, it doesn't take into account one of the most important developments in AAC, namely the addition of word prediction to most speech apps. When the author disregards using the keyboard, frankly she's throwing out a component of how many, many users utilize their tech. Of all the things that have sped up my daughter Schuyler's rate of synthetic speech, that's been the game changer. Now, she's an ambulatory user (polymicrogyria), so she's not representative of that population with more serious impairments that make using a keyboard impractical or impossible. But she is a part of that huge group, including so many with autism, for whom a combination of icons and typed words is a crucial part of their toolbox. It seems a little disingenuous to leave them out of the conversation.

Also, the article feels bereft of solutions. The delay in artificial communication has always been a huge issue. It's probably the one nut that developers have worked the hardest to crack. Beyond some kind of direct interface between brain and tech, it's probably always going to be an issue. So I'm not entirely sure what kind of technical solution she's hoping for from Apple. I do think that a direct interface is not totally in the realm of science fiction, but it's definitely future tech.

I'll ask Schuyler what she thinks.

Thanks so much Beth, Stacey, Ellen and Robert! I am thrilled for Max and Schuyler if they can communicate quickly and in the way they want to with their current voice apps. I don't recall our own experience with an app with word prediction. We did use WordQ, which is a separate tool, which uses word prediction to speed up keyboarding. The last time we really tried with our son was 2 years ago, when he was 21, and we used Speak for Yourself. The reason my piece is bereft of solutions is that for us personally, as a family, that is how it feels.

I do believe strongly that the expertise of Apple and other large computer makers should be brought to bear on voice technology. It doesn't make sense to have many tiny companies trying to produce apps or devices. In my mind, there shouldn't be a lag in the sophistication of voice technology when compared to mainstream business and consumer technology.

Children and adults who can't speak should't have to work with less than the most sophisticated, intuitive technology. For example, in our last try with Speak for Yourself, when programming the device, it was not possible to flip the placement of one word on a screen, for another (I believe this feature is on Proloquo). In order to reorganize the screen, you had to manually reorganize it. That is the kind of inefficiency that I'm talking about. I don't see why our families have to work so much harder than others do just to try to give their child the opportunity to communicate.

I am thrilled for any child or adult who is freed to communicate easily through technology. To be honest, I've worked in a large children's rehab hospital since 1999. I've seen many children use voice devices/apps, and even those who are very good at it are not able to do it quickly. I'm not saying they should stop. I am pleading with the major computer players to make this technology easier to use.

When I think of Apple, I think of products you can figure out intuitively, simply by playing with them. My own personal experience is that that isn't the case with voice apps. Perhaps there has been a huge surge in development I'm not aware of over the last 2 years.

I am writing from my personal experience -- which started with a Dynamite for my son when he was 4 (and which he couldn't carry), and has included Proloquo and Speak for Yourself. He doesn't want to use the voice apps. He is 23 now. Maybe if there was an app that was easier to use, he would get on board.

Yes, there has been progress. But I don't believe it's been progress at the rate we've seen in mainstream business and consumer technology. To me, that is wrong.

I would love to hear from Schuyler or Max or Beth or Stacey's kids about how they would like to see voice technology improved.

One more thought. Word prediction and the keyboard is great for kids who can spell. I was suggesting that when a young child first uses a voice device, they usually aren't able to spell. So if a researcher was assessing how "easy" it is to use a device (compared to mainstream tech), they should assess its use when you can't yet use a keyboard. After all, that is where a child starts at ground zero.

I think it's also worth pointing out that Gail Teachman's point about the social stigma of using a device was a very real concern and one that we also ran into... up until the introduction of the iPad onto the consumer electronic market and the development of apps like Proloquo-2-Go, Speak for Yourself and the system Schuyler prefers, PRC's LAMP Words for Life. For the kids who can use an iPad, the social stigma piece is greatly diminished; they're using the same technology as their typical peers. For those who can't, I'm not sure what sort of tech solution you would like to see from companies like Apple.

Think for a moment about the expense of developing some new hardware solution (rather than software development, which is where the bulk of the effort is going currently). Dedicated speech devices are incredibly expensive, which makes sense because they are building for a small market. Hundreds or thousands of units, not the millions like Apple sells. Imagine what that development cost would look like for Apple, and how expensive it would be to sell it to our small market.

And again, I'm not sure what kind of intuitive hardware solution you're looking for. Companies like Apple have developed a product, the iPad or whatever similar tablet, on which AAC is a natural fit for development. Language systems like Unity have met with a lot of success in developing language acquisition via icons and mapping patterns that increase speed for young users, and word prediction once they're older.

Is it perfect? Absolutely not. Is there a delay? There is a delay, and it is a huge source of frustration. But it's getting better all the time, developers and therapists continue to work towards faster communication, and the iPad continues to put this inherently unnatural way to speak into a socially accepted and even cool platform.

I'd love it if my daughter could put on a headset that would read her thoughts and speak through some kind of cool tech. I'm can imagine the day that could happen. But I'm not sure it's fair to find fault with tech companies like Apple for not making that a reality today. I've had the good fortune to meet and work with therapists and developers associated with a number of companies, including Apple, and there's one thing I can say with certainty. They care about this stuff, and they are moving that technology needle as fast as they can. (I guess technically that's two things.)

I'm also concerned that a blog post like this, as limiting and frankly out-of-date as it reads, is hosted on a site endorsed and hosted by a hospital. I know there's a disclaimer at the very bottom, but the overall impression that the site's branding gives is that it represents the hospital in a significant way.

A hospital that works this closely with children with disabilities should be concerned when incomplete editorial content is put out under its banner. I know that the author is expressing opinions based on her experiences as a parent, not a professional, but that doesn't excuse her unfamiliarity with some very critical elements of her topic. Indeed, much of the disagreement and frustration I'm reading online with this post is coming from self-educated parents advocates, including myself.

I understand the author's frustration like only a fellow parent of a kid with a communication disability can. But frankly, I feel like this essay is damaging to all the work being done, both to develop and advocate for assistive technology.

Hi Robert -- Your comment got stuck in our quarantine, which is why I only just posted it.

I'm sorry that you find this piece has many holes in it. I'm glad to hear through your and Ellen's comments that your children are benefitting from recent advances with voice apps, like word prediction. I've also received comments from parents who share frustration with what's available.

This is an opinion piece that I wrote based on my experience as a parent, but also as someone who has edited BLOOM for 16 years. None of the pieces on the BLOOM blog are hospital positions, hence the disclaimer. An opinion piece is one person's opinion.

That said, the red Dear Everybody image at the top, with the line about how exhausting using a communication device is, came not from me, but from our youth and families, as we were developing our new anti-stigma campaign. These messages were based on real words used by our children and families who wanted the general public to better understand their experience.

As mentioned, we have a Dear Everybody radio ad by a child using a device (not sure if iPad or dedicated device) that had to be preprogrammed. I think that's an indication of where the technology is, that it can't be used quickly enough to be "live."

Gail Teachman, who I quoted, worked for years here with children with AAC, and now does research on inclusion at McGill. Her studies are full of comments from young adults who use voice tech.

This piece is in no way a comment on the hard work of clinicians and families, or of the work and progress that has been made by developers. It is a call to the major players in the computer world to leverage their knowledge. The reason I mentioned Apple was that I think of Apple as being the most intuitive of products. It did seem to me, that there were many very small players and I wondered what a company like Apple could bring to the table, knowing what it's done for consumer technology.

The point of BLOOM is to open conversation and I genuinely thank you for taking the time to point out holes in this piece. At the end of the day, I think we all want better products that make it easier for our kids to express themselves.