Background

At Last.fm, users track every song they listen to - we call these ‘scrobbles’. Every week, month, and year, Last.fm publishes a Listening Report to each user, summarising their listening for that period.

‘In-progress reports’ were conceived as a way for users to preview their stats for a Listening Report before it was published. The hope was that by providing them with data mid-week, users would be incentivised to scrobble more, and ‘sculpt’ their report ahead of time.

A design concept had already been developed by another designer, but was untested. When I was brought on to the project, I discussed the brief with senior stakeholders, who were lukewarm - there was a feeling the early designs were overcomplicated, and hard to distinguish from the existing Listening Reports. I hoped to re-evaluate the feature, and streamline the functionality to produce a simplified but effective outcome.

Project aims

I used my meeting with Head of Product and Engineering Leads to nail down what Last.fm hoped the feature would achieve. I was able to summarise these into four key takeaways -

Increase scrobble volume and cadence

Last.fm wants to leverage stats and feelings of self-improvment and self-competition to drive users to scrobble more every day. More scrobbles means more valuable data, increased user retention, and a higher likelihood of subscription conversion.

More data for users = more site visits = more paid (Pro) subscriptions
Last.fm wants users to keep visiting Last.fm daily, while exposing additional, exclusive content for Pro subscribers. Exclusive Pro functionality tied to daily visits should increase subscription retention.

Clearly defined, unique functionality
It is important that the release is content- and feature-distinct from both the Library and Listening Reports, so there is no ambiguity as to its purpose.

Streamlined release plan
There was an expectation that this feature was ‘small', and as such, was only given a single (2 week) sprint turnaround for both design and implementation.

Concept and copy

Based on what I learned, specifically that this was primarily a subscription driver and incentive to scrobble more, I stripped the feature back to a much more simple concept - a progress bar, and some persuasive text to incentivise scrobbling.

The free user variant was even more stripped-back, incentivising them to upgrade to see the rest of the content.

I considered the headline copy to be a key aspect of the feature, and as such worked with our Head of Marketing (and copywriter extraodinaire) to come up with punchy, persuasive text for different scenarios (you’re behind, you’re catching up, you’re ahead, etc).

Testing and release

Given the short time-frame, I was only able to perform one round of user testing (via UserTesting.com’s remote platform). Feedback was mostly neutral, and, struggling with the fact that these were not existing Last.fm users, of limited value.

To make up for this, I reached out to some of Last.fm’s site moderators for feedback. They were happy to help, but the focus was mostly on the layout, colours and position on the page, rather than any discussion of the text content.

After making a few tweaks to accommodate this feedback, Last.fm decided to go ahead with the launch. The reaction was not what I’d hoped.

(Near) universally negative reception

It didn’t take long for users to notice the feature and let us know how they felt. Feedback started streaming in through our social channels, on Reddit, Discord and Twitter, and I was faced with every designer’s worst nightmare - people hated it.

Here’s a selection of some of the less explicit responses I could find -

“I find this annoying tbh, it's not a competition for me”
”Not a fan. I find the website works best if you just listen to whatever you want and don't care about the numbers”
”THIS IS SO ANNOYING, it’s just available for pro users and it’s so annoying on my profile. SO MANY F***ING ADS”
”A f***ing useless 'feature' that doesn’t convey anything we have’t seen before”

Disaster!

As the dust settled, and a couple of days had passed, I revisited our passive feedback channels to properly diagnose what went wrong, and come up with some solutions. In this time, the sentiment had calmed a little, and amongst the feedback were a few glimmers of hope.

Sentiment improves (a little) -

“For me, an easy improvement would be presenting it in a more neutral, comparison kind of way.”
”The statistic is interesting, but the presentation of it turns it into some sort of competition, which is not what most people use last.fm for.”
”This is just gonna make me boost my scrobbles even more.”

I really needed to hear this more level-headed feedback, as I was close to shuttering the feature entirely. These types of responses helped me to understand that the concept wasn’t entirely without merit, and could be redeemed.

The turnaround - turning problems into solutions

I set to work analysing the feedback in depth, and developing improvements. I collated the user response to the initial release, and summarised it thusly -

Scrobbling is NOT a competition
Users disliked the aggressive persuasion text, as it made them feel pressured into listening to music in a specific way.

Profiles are precious
Users are more resistant to changes to their Profiles than other parts of the site/app. The simplicity of the feature didn’t justify its dominant visual impact and position on the page.

Free users don’t like Pro adverts
This one seems a little obvious - but without a tangible benefit to free users, they users didn’t respond well to the new Pro subscription upsell on their Profile.

People liked the core concept
The ability to track their scrobble progress throughout the week/month/year, and compare it to previous periods, was actually quite gracefully received. Some users explicitly told us it’d increase their scrobbling.

Given the above, I redeveloped the feature, making the following changes to tackle the percieved issues -

Remove persuasion text, and deprioritise text content in general. Prioritise the data.

Add weekly scrobble count to the free user offering, to add more value to the feature and sweeten the Pro subscription upsell. Simplify the upsell copy.

Change the layout so the feature was less dominant on the Profile page. This includes making the bars thinner, and pushing the content further down the page.

Releasing the update

We pushed out this release super quickly; given the initial feedback, it was a high priority to show our users that we had listened, and were capable of responding immediately to valid feedback.

This time around, the response couldn’t have been more different. Users were much happier with the feature, and they vocally appreciated that we listened and were able to respond to their complaints. There were still some who considered the feature irredeemable (or pointless), but the general sentiment was that we had ‘made good’, listening to their complaints and taking relevant actions to address them.

Some feedback from the release -

“Good. Less pressure now.”
”Yes! I was so glad to see this!”
”THANK YOU to whoever made this happen - I appreciate it.”
”I MUST SCROBBLE”
”yay they finally changed the annoying ‘falling behind’ message”

Finally, a week behind schedule and with our tails tucked firmly between our legs, we could start monitoring the feature’s impact on our users’ scrobbling, and our Pro subscriptions, knowing the feature was in a place our users (and Last.fm) were happy with.

Post-mortem – Impact and learnings

Over the next few weeks, we monitored our analytics to see if the feature was performing as expected.

Subscription conversion
We saw a notable bump in subscribers for the release month, though this was likely due to the associated social posts and emails that are sent out whenever a new feature was released. Longer-term analysis would show whether the feature itself was driving subscriptions.

Scrobbles-per-user
We saw an almost overnight increase in scrobbles-per-user-per-week, amplified especially amongst our existing subscribers. This was particularly exciting to see, as the feautre’s core concept of ‘chasing last week’s scrobbles’ was clearly having the intended effect.

Site visits & page views
Unfortunately we didn’t see much (if any) change to our page views. We had hoped that users would want to ‘check in’ on their progress throughout the week, but there was little evidence this was taking place.

Overall, this project was a concise example of two things -

1. A lack of thorough user testing led to a poor reception on release. While we were able to fix things quickly, it was still embarrassing, both for myself and Last.fm as a whole. The project was a clear demonstration to senior stakeholders of the value of early and robust user testing. Since then, Last.fm has afforded much more time and effort to explore the validity of hypotheses, and test, test, test before laucnh! That includes testing copywriting, which was my own biggest takeaway from this project.

2. Listening to user feedback and responding quickly is greatly rewarding. Despite the messy launch, I was thrilled with how we reacted once the initial sentiment became clear. Not only did we end up with a product release that more or less met its goals in terms of KPIs and performance, but a lot of the positive feedback was directed at us at Last.fm; specifically with regards to how we were able to listen to our users and take their complaints seriously. This kind of reputational benefit is hard to quantify, but I’m very proud of how we were able to turn a product ‘failure’ into a success story.