User Testing

How Milanote iteratively improves their product by testing everything with real users

Published November 2, 2017 ⚡ Updated on October 2, 2023 by Markus Pirker

The notes app Milanote is used by creative professionals working at big players like Apple or Facebook. We got together with Ollie, Co-Founder and CEO at Milanote and talked with him about his obsession over design details, the constant struggle of keeping the product simple and how he makes sure everyone on the team has some regular contact with real users.

Tell us a bit about your background

I’ve always worked somewhere in between design and technology. I originally studied to be an architect, but dropped out after a year and switched to computer science. After I graduated, I was a developer for 10 years before I switched back to design and started a UX agency with a few other people. Milanote was spun out of that agency in 2016.

What made you decide to start working on Milanote?

It was really the classic case of solving your own problem. We were running big design projects for our clients and we needed somewhere to store and analyse our research.

At the time the wall of our studio were constantly covered with post-it notes, sketches, ideas, notes etc. We couldn’t find an online tool that did the same thing, so we built our own.

How long did it take to put together Milanote?

We’d been playing around with the idea off and on for years (mostly by talking to potential users about it) but we started working on it full time in Feb 2016, and launched one year later.

As a Co-Founder & CEO, what do you enjoy most?

The thing I enjoy the most is the craftsmanship that goes into making great software. My favourite thing to do is sit next to someone and obsess over the details of a piece of design or code.

I like to think of myself as a high-functioning perfectionist, so I find watching Milanote get closer and closer to perfection really satisfying. “God is in the details” is one of my favourite quotes.

I also love seeing how people use Milanote, either by talking to them, looking at stats, or seeing their boards. I really just love great art, literature, design and all other kinds of creative work, so seeing that sort of thing take shape in a tool we created feels great.

When it comes to testing, how often are you testing Milanote with real people?

We did a lot of in person research when we were getting started which helped us figure out the early direction of the product. We tested paper sketches, prototypes, early visual designs and everything in between.

Our most recent round of user testing on the actual product was only 3 weeks ago, so while we do less than we did in the very beginning we still think it’s really important.

We also get a huge amount of feedback and ideas from people using the product (mostly through Intercom) which is a big influence on what we decide to work on.

In your experience, what’s the biggest advantage of watching people really use your product, instead of just analyzing quantitative data?

Actually seeing people experience problems with your product makes them a lot more real, and I think it makes a huge difference in terms of how motivated you are to fix things. Reading an abstract task in Github like “error with login process” is really different to actually seeing someone encounter that problem and struggle to get past it. That’s one reason why I try to make sure everyone on the team has some regular contact with real users.

I also think that if you’ve done a lot of user testing, you pick up on lots of subtle things that none will ever explicitly report.

The things they click on that don’t work, little details in the interface that make them hesitate, where they look for things on the screen—really just all the little non-verbal behaviours which tell you so much about whether something is easy to use or not. Those little things make a huge difference how a product feels to use.

Can you remember your last „Oh shit“ moment while watching someone using Milanote?

We’re in the process of redesigning our onboarding, so we’ve been running in person user testing on that. We’ve also had UserBrain running in the background to test the same thing which has been a great way to watch people’s first experience with Milanote.

We have a special “test” server set up with our new design on it, and our UserBrain task points people to that server to sign up and talk us through their first session:

This has helped us spot all sorts of issues—for example we found that clicking on a particular part of the interface during our new onboarding flow made it impossible to continue creating an account.

Recently we had the brilliant idea of simplifying the Milanote interface so that text notes, images and links were all combined into a single “card” that would adapt based on the type of content you pasted into it.

It only took us about two user testing sessions to realise that new users found it completely confusing, so we abandoned it. But that’s just how design works—failure is the secret to success etc.

What is your biggest challenge at Milanote, when it comes to ensuring a good user experience while adding features?

We’re trying really hard to keep Milanote simple, so we’re always really conscious of adding too many features. We say “no” to features a lot more than we say “yes”.

It can be easy to accumulate “user experience debt” on a product because it’s much easier to justify spending time on cool new features rather than improving things you’ve built in the past.

I think the key to avoiding that is dedicating specific people to enhancements, fixes and bugs. It’s a lot easier to say “⅓ of our developers will spend all of their time on fixes and enhancements” rather than say “Everyone will spend ⅓ of their time” on the same thing.

How do you make sure that the issues uncovered during usability tests actually change anything for the better?

I’ve found the key to iteratively improving a piece of software is just to write everything down. Everything! We have a huge backlog of issues, most of which we’ll probably never fix. But by getting into the habit of writing everything down, estimating it and prioritising every week, you give yourself a much better chance of working on the right stuff.

For us, the key is to separate the collection and estimation from the prioritisation. If you say “nah, let’s not do that” too soon, you end up killing some great ideas too soon.

How do you feel you have grown from experience?

I try to put myself in situations where I can learn. I really like doing things I’ve never done before. I try to talk to people who know more about things than me as much as possible. I try to think of everything I do as an iterative prototype—whether it’s a piece of design, a piece of writing, a company, a product, whatever. I think prototyping can be a sort of life philosophy—that way nothing’s ever “done”, you can always improve it.

I also like to think of my career in its entirety, rather than just think about what I’m doing now. I’ve been designing digital products for about 15 years, which by my calculations means I’m about 35% of the way to retirement. When you think about things in decades like that, you realise how much time you have to learn new things and become a different person.

What insight has that experience given you that you can share with people who are just starting their journey into UI, UX design?

One mistake I see people making at the start of their career is labelling themselves as a particular type of person. As soon as you start saying “I’m a designer” or “I’m a developer”, you’re putting boundaries around the things you’re allowing yourself to learn or be interested in. If you avoid these labels and start to think of your career as something that’s going to last 45 years or so, investing time in adjacent or completely separate disciplines starts to make more sense.

For example, take the classic argument: should designers learn to code? Well theoretically a designer could take a break, do a whole PHd in computer science, and still have about 90% of their career left. That’s kind of a silly example, but it shows just how much time you have to learn.

What are you reading/watching/listening to educate and inspire yourself every day?

I read a lot, listen to lots of podcasts, spend lots of time on all the obvious websites, but personally I feel like I learn the most about the world from reading fiction. Nobody in tech seems to read fiction, which I think is a bit of a tragedy!

I just finished reading The True History of the Kelly Gang by Peter Carey—amazing novel, highly recommended.

Back to homepage