Science Report | Biology News, Economics News, Computer Science News, Mathematics News, Physics News, Psychology News
The Multiverse According to Ben
Today I tried out Siri, the new AI “personal assistant” app for the iPhone. It has some very smart people behind it, and is based on some code and ideas from the DARPA-funded CALO project. Siri’s earlier prototype version impressed me with its integration of dialogue and maps, so I was eager to check it out.
The Siri website says:
Just like a real assistant, Siri understands what you say, accomplishes tasks for you and adapts to your preferences over time.
Ahem….
You may want to scroll to the end of this post, and read my dialogue with Siri, before reading the rest of what I have to say about the app.
This is the first release, and one doesn’t want to judge the whole Siri project too harshly based on a first impression. But all I can report on now is my reaction to the product I just downloaded on to my phone and chatted with: and … well … let’s just say that
- Indeed, this version of Siri may be very useful for carrying out a very limited set of very specific functionalities
- It’s not anything like a real assistant; and worse than that, it gives no appearance of being an AI system making any kind of effort to really understand anything
- The basic “chatbot” functionality seems unnecessarily crude and quirky
As an AI developer I’m well aware that sometimes you can make mediocre (or worse) products or demos based on deeply powerful technology. So I’m open to the possibility that there is some profound or at least interesting tech underlying Siri. But I was unable to find it via playing with the product for an hour or so.
What I found, rather, is
- a rather crude keyword based chatbot (i.e. crude even by the standards of keyword based chatbots), without much attempt at dialogue management
- straightforward, rule-based integration with a very small set of knowledge bases (about restaurants and movies, for instance) and with a map engine
- straightforward integration with TrueKnowledge for answering of factual questions
- decent speech-to-text with a very nice interactive interface
What surprised me most was the crudity of the dialogue management, which you’ll see in the transcript below, of my initial conversation with Siri. So often Siri’s responses had nothing to do with the questions I asked.
And its persistence of information between questions is rudimentary and awkward. Once you ask one question about New York, it pretty much assumes all your subsequent questions are about New York … but it doesn’t understand linguistic references to previous queries, not even simple ones.
Is Siri useful as a virtual assistant? I suppose I might use it to find restaurants or movies, or to check flight status. And just the other day, in the midst of a conversation in the car with the kids, I wanted to know Hitler’s birth year, and I asked Wikipedia on my iPhone — it would have been nicer to ask Siri instead.
So, yeah, for a few specific functionalities, where Siri’s language engine and database integration are well-tuned — yeah, it may be genuinely useful.
But the useful functionality is really VERY narrow and brittle. If you go even slightly beyond what the application has been specifically tweaked for, the results seem to be useless and annoying.
As a single example, consider the following snippet from my first conversation with Siri, given in full at the end of this post:
Ben: What is Kate Braverman’s latest book?
Siri: OK, here are some businesses named “Kate” a few miles from here
This is really an unnecessary gaffe’, but it’s not exceptional; Siri does that sort of thing quite frequently. It makes this mistake because the query is about books and authors, rather than about stuff it’s tuned for: restaurants, movies, flights, TrueKnowledge facts. And even for some things it’s tuned for, like flights, the results …