New Course Reviews

Hundreds of new courses are published every month on Udemy, but few were generating enough revenue to be considered “successful.” As part of our larger initiative, our team designed a program to leverage our most active students to write reviews for new courses.

 

My Role:
Product designer - discovery, research, design

Team:
Soumya Santhanakrishnan, Product Manager
Michael Blau, Analytics
Janel Faucher, UX Research

Problem

 

Udemy relies on successful new courses to help keep its catalog fresh and up to date. However, we started noticing that fewer and fewer new courses were meeting financial expectations and popularity. We soon discovered that Udemy’s policies, algorithms, and user experience weren’t effectively working together to support them.

The whole system was broken.

Research

We did extensive research to understand what was happening from instructors, students, and experts within the company.

Key Insights:

  1. No differentiation between brand new courses and highly popular bestsellers.

  2. Instructors must initial momentum to their course within the first 90 days.

  3. Many instructors are reluctant about sharing their course to their network.

  4. Our algorithms weren’t working in line with our policies that helped new instructors.

 

New courses were often sandwiched between highly successful and popular courses

“I wanted nobody to see it, but someone to buy it”

-G.R., Udemy instructor

Ideation

We led a multi-day design sprint with participants from various departments to brainstorm different ideas and initiatives. We generated multiple hypotheses, and evaluated them based on feasibility, impact and effort.

 

Solution

Create a “Trusted Reviewer Program”, a program where our most engaged students enroll into new courses for free in exchange for writing a review.

 

Big Picture

Reviews are a critical component in our algorithms. By producing higher quality reviews, we can build more trust in our students, courses and platform.

The core experience works like this:

 
 
 

New courses homepage

Discovery landing page highlights all new courses that have been published within 90 days, with more emphasis given to courses that are showing signs of quality.

 
 

Free course enrollment

Students can enroll in a new course for free, but must complete a review before being able to enroll in another one. Students also must finish at least 30 minutes of a course before being able to leave a review.

 

Giving detailed feedback

Modified review flow encourages students to leave more detailed private feedback, since students were reluctant to write negative comments publicly.

 

Creating the System

Our first step was to see how this new system fit into the existing ecosystem of Udemy to avoid repeating our earliest first mistakes. These were also used for internal education

 

This project led to important changes in algorithms, adjusting our marketing initiatives, and has played an important role in course differentiation and pricing models. The Trusted Reviewer Program is currently in development, to be released in late 2020. I also learned a ton along the way.

 

Make a decision, but be open to starting over

In the presence of a million good ideas, it was easy to spiral into a state of complete inertia. Taking a stand and running with an idea while being ready to let it go at any moment ended up being the best strategy to push our momentum forward.

Look at the whole system

One of our biggest lessons as a company was understanding how much our systems were out of alignment. Taking time to step back and understand how our new program would fit into the greater system. This allowed us to have early discussions with stakeholders to minimize future misalignment.

Engage stakeholders early (and check in often!)

Partnering with our stakeholders and continually checking in with them helped advance more efforts that could ultimately help our instructors.