Science Report | Biology News, Economics News, Computer Science News, Mathematics News, Physics News, Psychology News
The Multiverse According to Ben
Today I tried out Siri, the new AI “personal assistant” app for the iPhone. It has some very smart people behind it, and is based on some code and ideas from the DARPA-funded CALO project. Siri’s earlier prototype version impressed me with its integration of dialogue and maps, so I was eager to check it out.
The Siri website says:
Just like a real assistant, Siri understands what you say, accomplishes tasks for you and adapts to your preferences over time.
It also describes Siri using metaphors of human learning, e.g. “like a child taking its first steps” ….
Ahem….
You may want to scroll to the end of this post, and read my dialogue with Siri, before reading the rest of what I have to say about the app.
This review has been edited in response to some comments (which you’ll see below this post) by Dag, one of the Siri creators. If you’re curious to see the original version of my review, it’s here. There are no huge changes but I hope this revised version is an improvement.
This is the first release, and one doesn’t want to judge the whole Siri project based on a first impression. But all I can report on now is my reaction to the product I just downloaded on to my phone and chatted with….
Two Perspectives on Siri
Before giving my detailed comments, I’d like to distinguish two different perspectives on Siri
- Considered as a freebie iPhone app, is it funky? Is it worth downloading and playing with? Might it be useful for some purposes?
- How well does it live up to the “AI Personal Assistant” label, and the description of being “like a human assistant”, “like a child taking its first steps”, etc.
Plenty of others can assess Siri as a freebie iPhone app as well or better than I can, so I’ll make a few comments in that regard, but focus most of my attention here on the AI aspect, since that’s my own area of expertise.
Overall, my take is that
- Indeed, this version of Siri may be very useful for carrying out a very limited set of very specific functionalities
- It’s not anything like a real assistant; and worse than that, its attempts to really understand anything you say seem very limited and domain-specific at this point
- The basic “chatbot” functionality seems unnecessarily crude and quirky
As an AI developer I’m well aware that sometimes you can make mediocre (or worse) products or demos based on deeply powerful technology. So I’m open to the possibility that there is some profound or at least interesting tech underlying Siri. But, to be quite blunt, I was unable to find it via playing with the product for an hour or so.
Siri from an AI Perspective
Looking at Siri from the perspective of someone who has built a bunch of AI systems, including chatbots and more serious natural language processing and reasoning systems, what I see here is:
- a rather crude keyword based chatbot (i.e. crude even by the standards of keyword based chatbots), without much attempt at dialogue management
- straightforward, rule-based integration with a very small set of knowledge bases (about restaurants and movies, for instance) and with a map engine
- straightforward integration with TrueKnowledge for answering of factual questions
- decent speech-to-text with a very nice interactive interface
What surprised me most was the crudity of the dialogue management, which you’ll see in the transcript below, of my initial conversation with Siri. So often Siri’s responses had nothing to do with the questions I asked.
And Siri’s persistence of information between questions is rudimentary and awkward. Once you ask one question about New York, it pretty much assumes all your subsequent questions are about New York … but it doesn’t understand linguistic references to previous queries, not even simple ones.
But Is Siri Useful?
But what about the practical aspect? Is Siri useful as a virtual assistant? I suppose I might use it to find restaurants or movies, or to check flight status. And just the other day, in the midst of a conversation in the car with the kids, I wanted to know Hitler’s birth year, and I asked Wikipedia on my iPhone — it would have been nicer to ask Siri instead.
So, yeah, for a few specific functionalities, where Siri’s language engine and database integration are well-tuned — yeah, it may be genuinely useful.
But my impression is the useful functionality is really VERY narrow and brittle. If you go even slightly beyond what the application has been specifically tweaked for, the results seem to be useless and annoying.
As a single example, consider the following snippet from my first conversation with Siri, given in full at the end of this post:
Ben: What is Kate Braverman’s latest book?
Siri: OK, here are some businesses named “Kate” a few miles from here
This is really an unnecessary gaffe’, but it’s not exceptional; Siri, in its current version, does that sort of thing quite frequently. It makes this mistake because the query is about …