Pivoting after User Research

Background
My ROLe
  • Lead product designer with a team made up of a product manager, data analysts, engineers and customer service agent.
  • Research - analysed existing business problems and conducted user research. Formed product strategy together with lead product manager.
  • Design and develop - applied research results and designed several experimental solutions.
  • Evaluate and analyse - evaluated A/B testing results to inform next steps.
CONtext
This was one of the projects I had whilst working at N26 Bank Products team.
PURpose
One of our business goals is to increase the adoption rate and contribution of overdrafts. We believed that by delivering contextual promotion to our customers, we will gain more overdraft users and therefore increase the revenue to the business. However, after a few rounds of user interviews and data analysis, we realised that this would not fulfil our customers' and business' needs. Instead, facilitating loan repayment would create a win-win situation for both customers and business.
Objectives
  1. Understand who our overdraft users are, their motivation, pain points and their context for using the product.
  2. Promote User-Centred Design approach to the team and stakeholders.
  3. Meet business objectives - increase net contribution (increase revenue or decrease cost) of bank products. Increase monthly active bank products users.
Approach
  1. Data analysis & qualitative research to deep dive into the problem.
  2. User testing to prove our concept.
  3. Solution ideation to define scope of release.
  4. A/B test to understand the results and future improvement.
Summary
Challenge
  1. As a business, we want to increase the contribution of bank products to the business (€70/customer).
  2. Overdraft is a legacy product at N26, any small change will require a fair amount of development effort. Our solutions need to consider using current possible resources and create impact to our customers.
  3. During our user research, we found out that instead of pushing users to borrow money, helping our customers to track and control their finance could help vulnerable users to repay their loans and therefore reduce costs to the business. The question is, how could I persuade the team and stakeholders to change our direction.
SOlutions
  1. Data analysis - we are losing money from current loan users and we expect this trend to continue with new users.
  2. User interviews - users were not aware that they were in negative balance. They want to have more control of their finances.
  3. Concept testing - users felt anxious and that they were "pushed by the bank" by being notified about the product.
  4. Re-discovery - ideation workshop with the team to better support our overdraft users.
  5. A/B testing -  validating our hypotheses and identifying impactful solutions.
Project detail
Define
Quantitative research
  1. We have increasing debt in 2020 because compared to the past, overdraft users have a larger negative overdraft balance
    • 6% increase in users with active overdrafts since June 2020
    • 30% increase in average outstanding overdraft amount per user
  2. We examined different overdraft user behaviour* , we found the following trends related to users who use overdrafts properly (i.e. never had arrears):
    • On average the balance in their current account has been decreasing since June 2020 by 47%
    • These users spend more days with a negative balance, having reached its highest value in October 2020 to 13.7 days
  3. We had a hypothesis that users who have a failed transaction might be lacking cash which could provide us with a chance to promote overdrafts. However, the data showed  less than 15% of failed transactions were due to insufficient account balance.
*users who used overdraft and repay properly, users who used overdraft but did not repay on-time, users who activated but never in overdraft
*users who used overdraft and repay properly, users who used overdraft but did not repay on-time, users who activated but never in overdraft
The data indicated that we need to help our users be more aware of their finances or the business will face more short term losses (i.e users cannot repay debts) and long-term customer lifetime value losses (close account, negative word of mouth).
Qualitative research
Since we had never talked to our users regarding overdrafts, we would like to learn who are the customers using overdrafts, get insights from common use cases and explore with which events and moments we could introduce contextual triggers.
  1. Participants
    • 2 users - do not have overdraft activated.
    • 4 users - used overdraft at least once in last 4 weeks.
  2. Research questions
    We believed that activating an overdraft is a reactionary; we need to be more active to promote the feature for non-overdraft users.
    • How do our customers discover overdrafts?
    • When and how would customers like to have an overdraft available to them?
    We also see 75% of customers enabling the feature, and then never using it – either, they don’t really need it or they forget it is available to them. Does it make sense for us spend time recruiting more idle users?
    • Do customers know they have enabled an overdraft?
    If we were to push overdrafts to new users, how often and when should we do it?
    • Why/When are customers going into overdraft?
    Exploratory research - how can we improve overdraft experience?
    • What was their experience with N26 overdraft?
    • What are the other financing products they have used?
  3. Results
    We were able to depict the overdraft user journey from undergoing negative balance to repaying their loans, their thoughts, emotions.
  1. After synthesising our interviews, we found out -
    • If users do not need to borrow small amounts of money in the short term, they will not activate their overdraft.
    • Users do not need a notification to be "pushed by the bank" to enable overdraft. They would rather discover by themselves.
    • Users expect to be informed about important charges so that they can be in control of their finances. Users have also devised outside app hacks to help them stay informed on their usage or remind them to pay back.
"I forgot to top-up my account since N26 is not my salary account. I didn't even know I am using an Overdraft"
"I don't know when will I be charged the interest. I would have paid back if I knew"
"I want to feel like my bank is on my side - help me to understand when I will be debt-free"
Discovery workshop
Challenge

Evidence showed that recruiting more overdraft customers might hurt our business if it results in more users struggling to pay back their loan. Therefore, after the research, we decided to pivot the project to - HMW help our users be more aware of their finances?

Methods - Discovery Workshop
  • My Role - Facilitator & Moderator
  • Participants - Product Manager and Lead Engineer
  • Format & Tool - Online workshop with Figma
We gathered together to define business problems and user needs again from our interview insights
The reverse ideation
After the discovery workshop, we had the mutual understanding of our problems, I then used "The Reverse" ideation method to explore more solutions with the team.

"The reverse" technic helped us to think without the current limitation. It empowered the team to be brave and creative.
A/B Test - Experimenting with our Assumptions

Considering current development resources and tech architecture, we decided to experiment with different "info cards" on our home feed and push notifications for 1 month.

The experiment ran between 2021-04-29 and 2021-06-04.
Hypothesis
Target Group
Sample split
Success Measure
Users were not aware of their financial situation - A notification to start using overdraft will increase the chance to top-up (especially non-salary account users)
Users that start using overdraft
Random sample
- 50% test gets info card
- 50% control no info card
Test samples show higher top up rates within 7 days of info card
Users want to be in control of their debt - A notification when users are using 50% of their overdraft limit
Users crossing 50% of current limit used
Random sample
- 50% test gets info card
- 50% control no info card
Higher % of users in one sample
- Top up in the next 7 days since seeing info card
Results and next steps

Hypothesis 1

  • Users who start using overdraft: top up rate has increased 5% from the control group
  • We would continue notifying users when they did not intend to fall into negative balance. This would help us to maintain a positive relationship with our customers and make them aware of their finances
  • We will keep monitoring if it could also reduce write-off amount from this experiment

Hypothesis 2

  • Users were not motivated to top up their account (no impact to arrears or write-off)-
  • We will not continue this experiment but consider sending the notification when it's closer to their overdraft limit (e.g. 90%)
Lessons Learned
  1. Defining right problems definitely saved us from launching an unfavourable product. I cannot imagine how much more revenue we would lose and how many users would be annoyed if we started promoting overdraft to more users.
  2. Being ethical is a requirement. It is not just something nice to have, especially for a bank. When we realised our users might be using overdrafts without even noticing, we knew we had to do something for them. Although we haven't seen quantitative results, we heard from other studies that users appreciated that the bank is finally on their side.
Next project

Map Search

mengchlin@gmail.com
LINKedinDRibbbleCV