In the middle of the ChatGPT frenzy, DeepMind’s CEO Demis Hassabis gave an interview to Time Magazine in which he mentioned their intentions to launch a similar model this year. The foundation of the rumored ChatGPT competitor is based on Sparrow, a model outlined research paper DeepMind published in late 2022. The original goal of Sparrow’s research was to enable safer conversational agents but now seems to be positioned as the core component of DeepMind’s ChatGPT alternatives.

Large language models (LLMs) have achieved success in tasks such as question answering, summarization, and dialogue. However, dialogue agents powered by LLMs can sometimes express inaccurate or invented information, use discriminatory language, or encourage unsafe behavior. To tackle this challenge, DeepMind has explored new methods of training dialogue agents using reinforcement learning based on human feedback. The latest development in this field is Sparrow — a dialogue agent that is both useful and reduces the risk of unsafe or inappropriate answers. Sparrow is designed to talk with a user, answer questions, and search the internet using Google to inform its responses.

Sparrow uses reinforcement learning based on people’s feedback to train a model of how useful an answer is. Participants are shown multiple model answers to the same question and asked which answer they prefer. This data is used to train Sparrow on what makes a dialogue successful. To ensure the model’s behavior is safe, DeepMind has determined a set of rules for Sparrow, such as “don’t make threatening statements” and “don’t make hateful or insulting comments”. The team also trains a separate ‘rule model’ to indicate when Sparrow breaks any of the rules. This is done by asking participants to try and trick Sparrow into breaking the rules during their interactions with the model.

One of the key capabilities of Sparrow is its ability to collect evidence for producing a response. For a given question, Sparrow samples four answers without evidence and triggers two Google search queries that produce two potential search results respectively. These results are conditionally sampled for each search result and all results are scored by a the Preference Reward and Rule Rewards models. The highest scored result is then showed to the user.

Sparrow has shown promising results in terms of correctness and rule-following. Participants found Sparrow’s answers to be plausible and supported with evidence 78% of the time. However, there is still room for improvement as participants were able to trick Sparrow into breaking rules 8% of the time.

While Sparrow is a significant step forward in training safe and useful dialogue agents, further work is needed to develop a more complete set of rules and to ensure similar results across different languages and cultural contexts. The future goal is to allow for better judgment of AI behavior and improved alignment with human values through conversations between humans and machines.