Using NLU to derive evaluation results from qualitative in-product feedback

Partner:Alexander Fulcher, Girl Effect, Non-Profit

Overview

Project Description

Girl Effect builds products for girls in developing markets. We work in Rwanda, India, Ethiopia, South Africa and Tanzania. Our product portfolio contains magazines, television shows, music videos, websites, chatbots and social media channels, all designed to drive uptake of knowledge and change in attitude around key subject areas defined in girl effect’s theory of change.

We try to make products which put our users at the front and centre of the experience. In our digital products, this means allowing girls to share their feedback and guide their own user journeys, rather than be passive recipients of content. As a result of this, we have collected a lot of unstructured qualitative data from our users. In Big Sis, our chatbot product in South Africa, we are now using this data to enable girls to write their sex and relationships questions which we run against a dictionary we have created based on existing training data and provide relevant content suggestions.

At various stages in the user journey, users are able to provide free-form feedback as to the quality of their experience using Big Sis. We have seen a broad range of comments demonstrating uptake of knowledge as well as changes in attitude and behaviour in the real world. It is difficult, however, to turn these anecdotal observations into a more robust view of the change occurring as a result of engaging with the product.

Currently, Big Sis is exploring a number of different evaluation techniques. We will be running an online research community where we will take preselected target users through key activities and measure the impact of the product based on pre and post surveys. Meanwhile, within the app, we have created a control/ exposed evaluation where we are able to compare the results of an evaluation quiz taken by girls either before or after they have consumed content. We would like, however, to make the most out of the girls’ desires to express their experience in their own words. The challenge is to be able to interpret the key words and phrases which represent the process of change and create something we will be able to measure. We must also take into account the language that girls use in South Africa, and more broadly in all the countries Girl Effect operates in, is not ‘standard’- rather it uses text speak and local vernacular and is prone to misspellings.

Once we have developed this approach in one language, we would then like to be able to apply our model to a different language- for our equivalent product, Bol Behen, which is largely the same user experience as Big Sis but has been localised into Hinglish, the English-Hindi patois used by young Indians online.

Expected Deliverable

We would like to see a model/ a number of models developed using real user data which which we can connect to our chatbot via API - to store new records, classify them according to the model, and potentially then to send back information about the category to inform the user experience within the chatbot.

What would a successful semester look like to you?

Proof of a model based on user data that is able to demonstrate the signs of change according to Girl Effect’s theory of change.

Data

Models

Conclusion