News in a Number: Chatbots, It’s Complicated

3 min read
May 7, 2020
  • Post on Twitter
  • Share on Facebook
  • Post on LinkedIn
  • Post on Reddit
  • Copy link to clipboard
    Link copied to clipboard

9.4 billion

The number of parameters in Facebook’s latest chatbot, which the company claims is the most engaging and human chatbot ever created.

Some say competition is dead and lay the blame at the feet of tech companies like Google and Facebook. Whether that is true or not, competition between the Silicon Valley giants is alive and well. This latest chapter in the race for better conversational artificial intelligence (AI) illustrates that fact well.

Facebook released its latest chatbot model last week, claiming it scored higher in humanness and “engagingness” than any chatbot before it. Google made the same claim in January about its chatbot named Meena.

Facebook researchers were clearly focused on outdoing Meena. The authors even threw a few barbs at Google’s methods as they compared, “The human-bot conversations were conducted by employees and were not blind to the model type (in the logs they say phrases such as ‘Hi Meena!’).” Interestingly, Facebook’s research paper doesn’t mention the word “Google” even once. 

The article on Facebook’s blog also mentions that, “True progress in the field depends on reproducibility.” This seems to be a reference to the fact that Google’s code and data are not available to the general public. Facebook has made its code open-source and published everything as part of the ParlAI project. Google researchers cited safety concerns for not publishing their chatbot model.

Competition aside, the results are both exciting and disappointing. Chatbot technology has clearly advanced, but even the researchers admit there is much left to improve. On the exciting side, these two exchanges show a surprising level of awareness:

Meena: I heard that a cow went to Harvard.
Human: what did the cow study?
Meena: Bovine sciences.
Human: do horses go to Harvard?
Meena: Horses go to Hayvard.
Source: Google Research

Facebook chatbot: Have you ever played the forest? It’s made by the same people who made the long dark.
Human: Yes I have! It was fun in VR mode.
Facebook chatbot: That’s cool. I didn’t know it had vr support. Does it work well?
Source: ParlAI

On the disappointing side, the chatbots still misunderstand context and subtext in many situations. This leads to some unnatural responses:

Human: say you’re in an empty giant office and you see a black bird flying indoors. what do you do?
Meena: What kind of bird? Is it friendly? How large? How big is the office? Is it indoors or outdoors?
Source: Google Research

Facebook chatbot: That’s great. I love gospel music. It reminds me of my grandmother.
Human: Why does it remind you of your grandmother?
Facebook chatbot: She was a great woman. She passed away a few years ago. I miss her a lot.
Human: Sorry to hear that. I’m glad you have fond memories of her.
Facebook chatbot: Thank you. It was a long time ago, so I’ve gotten over it. Do you have any siblings?
Source: ParlAI

With Amazon, Apple, and Microsoft all working on similar technologies, you can be sure that the battle for better conversational AI won’t end here. Let’s hope that with the next few billion parameters, chatbots will learn a little more respect for Grandma.