I got to hear Marty Cagan speak at Etsy’s Code as Craft series in Brooklyn last week. It was a great talk, covering all aspects of building innovative products and a great product-focused culture (something I care about a lot, and unfortunately don’t get much of a chance to practice). The talk is available for streaming here:

Marty Cagan is the Founder of the Silicon Valley Product Group, where he gets to work with many of the leading technology teams in the world. Prior to this, Marty was the original SVP of Product and Design for eBay. He began his career working for 10 years as a software developer at HP Labs, and then moved on to join a young Netscape Communications as their VP Platform and Tools (alongside a few other bright lights, like Marc Andreessen and Ben Horowitz). Marty is the author of INSPIRED: How To Create Products Customers Love, which digs deeper into the ideas he touched on in his talk.

Highlights

Marty’s ten observations for building a continuously innovating workplace were:

1. Vision and Passion - define a 2-3 year aspirational product vision of what you are trying to achieve
2. Know What You Can’t Know - recognise that the standard product development approach is crap, that half or more of your ideas won’t work, and it’ll take 2-4 iterations to get something that makes money. The Continuous Discovery model helps separate the good ideas from the bad ones in a fast, iterative approach.
3. Know What Your Customers Can’t Know - accept that people don’t know what they want until they see it, so asking them directly doesn’t always work. Instead, use qualitative and quantitative observation to elicit ideas.
4. Product Discovery - this should be your core competency: getting to validated ideas with data showing that they’re going to work. You can discover ideas from your Engineering team or your customers to then run experiments. These experiments are the start of your product.
5. Dedicated Teams - the best companies run dedicated product teams comprising Product, UX and Engineers. These are not project teams but stay together for at least 1-2 years.
6. Outcome Over Output - use KPIs and goals to give teams outcomes to work towards. Don’t measure outputs like lines of code.
7. The Role of Design - Design is hugely important and is more than just the look and feel (or UI). Recognise all the different areas and incorporate them.
8. True Collaboration - You need Engineers, Designers and Product Managers truly collaborating in a colocated space.
9. The Need for Speed - Measure how long it takes to get an idea through the product development cycle and improve your techniques for evaluating ideas.
10. Product Culture - A product culture is not a company’s culture (and vice versa). A product culture respects data over opinions, and represents true collaboration between product and engineering.

Continuous Innovation

Innovation is the purpose of the various types of people in a product organisation:

• Product Management
• UX
• Engineering

Continuous Innovation is about delivering significant optimisations, which is about consistently adding value, more than just incremental changes.

Marty felt that these companies represented great examples of innovative companies:

• Uber - they’re innovating across a whole range of transport markets
• Amazon - they’ve done it dozens of times across any number of industries. They not a one & done place.

Ask yourself: If you just kept your existing product going, with no new work or features (even simpler: if you just kept the site up), would you still grow? If the answer’s yes, then you’re not innovating, you’re just experiencing organic growth.

A bit of Marty’s background

Marty previously worked as a developer at HP Labs. Here he got to see a research lab at a great time for HP (it was the age of the desktop, after all). But one interesting thing he saw was the lab had some critical KPIS, including measuring the % of revenue from products created in the last 2-3 years of the lab (it had to be at least 50%). After a particularly bad waterfall project, Marty asked himself, “Who decided we should build this product?”. He came to the conclusion that other people had made the decision not the technology team. The further realisation was there’s not much point investing effort when you don’t know that you’re going to succeed.

Marty then worked at Netscape alongside Ben Horowitz. Marty strongly recommended to us Ben’s new book, The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers. They worked together at a fertile time for Netscape which spawned JavaScript, SSL and a bunch of other technologies. Marty was then introduced to Pierre Omidyar, co-founder of eBay. Marty set about building a product/UX team at eBay, which was his first time building an app, rather than a platform/infrastructure piece.

As a side note, Marty considers Etsy as what eBay could have been!

So what are the keys to innovation?

Marty had a list of 10 keys to continuous innovation. They’re not his prescribed list, just things that he’s observed in top-performing product teams.

1. Vision and Passion

Way too many teams just start building. They just start pumping out code. Funnily enough, a lot of teams in Capital-A Agile start a bunch of sprints cranking out code.

Instead, you should spend some time defining a product vision.

A product vision is a 2-3 year aspirational view of what you are trying to achieve. You want to be painting a picture of where you see things going. This helps recruit talent for your team - those people join because they find your vision compelling. Our job as product people is to get the team excited about that vision.

A good example of the power of the vision is Workiva. They’re a company built around SEC filings, not exactly the most thrilling thing ever. But they have actually fundamentally changed the lives of their customers, those accountants and business people stuck doing those filings in big enterprises. That’s a powerful vision. They now have a higher Net Promoter Score than Apple!

2. Know What You Can’t Know

Most people in the room had read The Lean Startup, and Marty pointed out that the lesson that it teaches us is that waste is bad. So let’s not waste things! Blindily building what we fundamentally can’t know about is just wasteful.

Marty then showed us a diagram he’d drawn of the typical product development process in most companies:

1
2
3
4
5
6
7
8

Idea
   `-> "Biz Case"
            `-> Roadmap
                    `-> "Requirements"
                             `-> Design
                                    `-> Build
                                           `-> Test (QA)
                                                   `-> Deploy

In this view, once an Idea is found, a Business Case is used to establish the profit and the cost of the Idea. But the sad thing about most Business Cases is that they will actually make nothing. How can you actually know if the Idea is any good if you haven’t deployed anything?

You don’t know! You don’t know how much it will actually make or how much it will cost. They’re just guesses and assumptions.

In fact, when it comes to costings, a lot of businesses will do finger in the air ‘estimation’, where they corner some poor engineer and badger them to get t-shirt size estimates (Small, Medium, Large, X-Large, etc), which are totally arbitrary and fake.

Then, businesses build a Roadmap, which Marty considers the absolute worst thing. Because now that the organisation has the Roadmap, they depend on it to guide their products and call out others on their performance based on the Roadmap. The team then blindly starts a Requirement process, writing user stories, or in even worse places, writing requirement documents. There’s a Design phase where some UI work is drawn up based on the stories.

Then finally, they eventually start a Build phase. This is where a lot of so-called “Agile” teams believe they’re being Agile, holding several iterations here. Some worse teams even tack on an iteration for Test (QA), which they call ‘Hardening’. Eventually, the Idea is Deployed. And now our product validation can finally begin!

However, if we recognise that we don’t really know what we’re doing, then this process is the most wasteful process ever.

Marty had two key observations to share here:

1. Half (at least) of your ideas will never work with customers.

• Sometimes the idea is just bad and doesn’t work with customers.
• Sometimes your customers do want to use the idea but it’s hard to understand, so they disregard it.
• Sometimes your development has made it hard to use, so they disregard it.

Marty thinks that the best product teams start with the assumption that 75% of their ideas are crap.

2. For the other half of your ideas that do work with customers, it will take 2-4 iterations before we get to Time to Money.

In the product development process, Time to Money would be the point where we’ve got the right, validated product, and we can sell it, and we’re making money. But, if it takes 3 months for a full iteration from having the idea to deploying it, with 4 iterations required to really get to Time to Money, that’s a whole year before you’re at the Money stage!

Marty’s comment:

The best teams embrace this.

Marty had an alternative approach to introduce to us, the Continuous Discovery model:

1
2
3
4
5
6
7
8
9

Ideas
    `-->
          Discovery
      ------------------------>
         |   |   |   |   |
         |   |   |   |   |
         `-> `-> `-> `-> `->
        -------------------------->
            Delivery

This process involves Product, Design and Engineering collaborating.

Discovery helps us separate the good ideas from the bad ones.

Delivery is about building production quality products, involving considerations of test automation, scalability, globalisation, security, etc.

For Delivery, Marty thinks teams should be releasing at least every week. Teams should be aiming to move to the Continuous Delivery model, like Etsy has. In this model, you should be doing at a minimum 5 iterations a week. You should be aiming for at least 15 iterations a week.

Remember, we can’t know until we validate our ideas with customers. The faster we do that, the better.

3. Know What Your Customers Can’t Know

In B2B-focused products, the typical customer approach is that you have a bunch of sales organisations who go away and sell your product. They come back with a cheque for a sale and a laundry list of features.

In Consumer-focused products, your problem isn’t your customers, it’s that those feature ideas are filtered through your executives. Your executives don’t know what’s really possible with the technology that you have (or that you might potentially have).

Really good stuff is made possible by technology. Engineers know that. Executives often don’t.

If you’re a Product Owner or a Product Manager, and your role says you should be gathering requirements, that’s the antithesis of Product.

As an aside, Marty remembered how companies used to do product development in NY: they had Capital-I Information Technology (IT) teams, which was a service organisation for the Capital-B Business. Marty also thinks it’s a big smell if you start hearing product teams talking about the Business: it means they are no longer thinking about their customers. Thankfully, Marty thinks NYC startup are currently operating on par with Silicon Valley startups in their product thinking.

Ultimately, Marty believes none of us know what we want until we see it. Apple is one of the few organisations to get this, and consequently built the most prototypes of any organisation that Marty had ever seen (Marty recognised this could be a consequence of them being a hardware company, but it’s still an interesting note).

So how do you go about finding out what your customers actually want?

When Marty works with a product team, he ramps up the customer iteraction way, way up to at least once every week at a minimum. Ideally he’s aiming for multiple times a week.

The two main methods of observations are:

• qualitatively: an in-person observation
• quantitatively: a data-based observation

The best teams do both qualitative and quantitative observation.

4. Product Discovery

This is the main competency of a product organisation. This is our day job!

The end-goal of this process are validated ideas where you have data that they’re going to work.

Marty thinks that the best source of innovation in your company isn’t your product manager, it’s your engineering team. They see the possibility of the technology every day. They’re optimistic about a technology that solves a problem. They are inspired by ideas from looking at the data.

Other sources of innovation:

• watching our customers struggle with our products.
• watching our customers struggle with our competitors products.

Referring back to The Lean Startup, Marty reiterated something I’ve talked about before: that the Minimum Viable Product (MVP) is not a Product. It’s an experiment. If you asked your marketing team about what they thought about your MVP, they’d call it a pile of crap that destroys the brand.

Some better terms to use when referring to these are MVP tests or a MVP experiments. They aim to get to the term that Marc Andreessen coined: Product/Market Fit.

The MVP experiment here is the start of something, not your final product. They’re prototypes. Your actual product is the point when you have something that you can deliver, and that you can sell.

Prototypes

Marty alluded to 2 main types of Prototypes:

1. Simulations (User Prototypes) - a friend of his called this the “fake it before you make it” approach
2. Live Data Prototypes - 2 examples: connecting to real data sources, or sending real traffic to your prototype. This is used when we need real data and real traffic to validate our idea.

Our aim in building a prototype is to build enough to measure the minimum required to validate our idea. Marty’s rule of thumb is that your prototype should be somewhere in the range of 20-30% of the real quality of a production version. You’re also aiming to send only a fraction of your traffic to the prototype.

Marty summarised Discovery as: Find out the good ideas from the bad.

5. Dedicated Teams

Marty told us that the best companies run dedicated product teams. These teams are together for an extended period (1-2 years) and handle everything to do with that product: new features, bug fixes, optimizations, everything. In Lean terminology, this is a durable product team. The team needs to comprise Product, UX and Engineers - a truly cross-functional team.

Marty clarified that this is not a project team. Project teams don’t stay together for long enough and don’t go through enough iterations to truly move between Discovery and Delivery. It’s very much an old-world product development approach to use a project team.

If it’s possible, you should colocate the team. At Yahoo, Marissa Mayer removed the idea of working remotely or from home (even as she was 8 months pregnant)! Very few teams were colocated, and a lot of teams had yearly roadmaps to work towards (even worse)!

How do you manage these dedicated teams? Don’t give them a yearly roadmap. Give them KPIs and goals, and let them experiment with ideas to reach those KPIs. This leads us to the notion of outcome-driven development.

6. Outcome over Output

Marty reminded us that in a product-focused organisation, pushing code itself isn’t interesting. Outcomes are very interesting.

As we were sitting in the Etsy Labs, Marty dreamed up a hypothetical scenario that might play out for a continuously innovating organisation like Etsy. Let’s say that the Etsy Seller Team wants to boost seller numbers in Brazil. Rather than having a bunch of feature requirements handed over to that team, a good KPI would be that they want ½ a million sellers in Brazil. It’s then up to the team to go figure it out to reach that KPI. It is up to the team to solve it (for instance, they should get on a plane to Brazil to embed themselves with their customers).

7. The Role of Design

Most companies just don’t understand design. They think that it’s look and feel. Or worse, that it’s just the UI: the buttons and the colours.

Design is not just what it looks like and feels like. Design is how it works.

Design is a multi-disciplinary area, and companies need to realise that this comprises:

• User research, both qualitative and quantitative research
• Information Architecture
• User Experience
• UI Design
• Interaction Design
• and much more

A bad smell is that “design” is handled by Product Managers who still make wireframes (in absence of any formal training) and then blindly hand this over to a graphic designer, who just pretties it up. This creates a bad situation, and undesirable products. Our customers just end up with no desire whatsoever to use our product.

8. True Collaboration

Marty feels there are 3 critical competencies to have in the product organisation:

Engineers - they need to participate in what to build, not just how. It’s best to involve every engineer in ideation (and the best teams do). At a bare minimum, the Lead Engineer should be involved in ideation. Engineers should be participating in discovery and watching customers. This should really be happening for 20 minutes a day minimum.

Designers - they’re working on discovery and prototyping.

Product Managers - the Product Manager role is pretty unclear. They definitely need to understand the customers deeply. They also need to have a deep knowledge of the data: how the tool/app is actually being used. They need to be incorporating and analysing any number of different types of analytics: data analytics, customer analytics, compute analytics (processing, etc).

The Product Manager should understand each of the stakeholders involved in your product. A litmus test for defining a stakeholder is if that person could throw a wrench in the works, then they’re a stakeholder. What you’re trying to understand here is not their requirements, but the constraints that they operate under, eg. a Finance stakeholder cares about the cost and money it takes, a Legal stakeholder operates under the constraint of law and regulation. You need to convice those stakeholders that you understand them and that you will only propose solutions within those constraints.

Some other important activities that the Product Manager should do is understand and be aware of industry trends, and sourcing reference customers (sometimes harder than it sounds).

Overall, once you have the 3 critical competencies in your dedicated team, colocation makes working together easier: then you’re not in document sending hell.

9. The Need for Speed

Look at your own product development process: if the workflow from Roadmap to Deploy takes weeks, then you are super slow. You should be able to take on 5 significant ideas or approaches a week at least. An analogy is baseball’s number of at bats.

A typical reaction for most companies (sadly too true) is just to pressure your engineers to do more (or just some vague command to “Get faster!”). Don’t do pressure your engineers. Speed really comes form the techniques of evaluating things faster.

10. Product Culture

When talking about a Product Culture, Marty doesn’t mean a company’s culture. A lot of companies have a great company culture but also have a horrible product culture that hasn’t innovated for years. Conversely, there are a number of companies with a great product culture that have a horrible company culture. Marty didn’t point fingers, but I think we have a few ideas.

A product culture really respects data over opinions. If your data proves something different to your belief, then that’s where you go. Admittedly, sometimes you will still do things in spite of the data, but there have to be valid reasons and circumstances for why.

A product culture really represents true collaboration between product and engineering (sounds great, doesn’t it!). This culture means that we’d never release something without the measurements in place so that we can learn from it. This environment pushes for rapidly testing and learning, and consensus not dictatorship.

Questions & Answers

After wrapping his talk, Marty had some time for questions.

If you’re just getting started and don’t have customers (or you’re in an industry with a limited or finite number of customers), how do you get reference customers to test against?

Marty reminded us that most of the time when we’re experimenting with our ideas, we’re actually not trying to be statistically significant. For B2B environments, the process of sourcing customers can be typically very hard. In both a B2B environment and a customer-focused environment, we’re aiming for a qualitative assessment. We’ve got to go face-to-face. One of Marty’s friends had a great quote along the lines that if you go face-to-face with a customer, you likely won’t get all the pieces from that session. But every face-to-face sessions gives you a bit more, until you get to that headslap moment: “why didn’t we see that before!” In a B2B environment, Marty uses the Customer Development Program technique to help this along.

Essentially, we’re trying to get to product/market fit and we’re aiming to find 6 live reference customers where we meet all their needs.

How do you execute so many tests?

One of Marty’s friends, Tom Chi, running design at Google Glass was running 15 iterations a week (which we all agreed was insane)! In an environment working on both hardware and software like that, you have to get very creative. So they used things like clay to work on hardware models, rather than build in plastic every time. Every prototype was a hack, and they tried all sorts of ideas, like a Minority Report style interface. They had all sorts of tests for every aspect of the Glass: weight, size, shape. But the key thing was that every prototype tested at least one idea.

So how does this actually play out?

For example, if you wanted to try out a new payment mechanism, you could build a Simulation Prototype to put in front of users. Then you observe if the users are lost, or if we’ve missed something. Once we have a version, then we build a Live Data Prototype.

How do you choose what ideas to fund?

If you have a new idea that you want to play out, and there is already a team in place with KPIs, they might already be solving it. You could also expand the KPIs to deal with the new idea.

If it’s a new thing (eg. Uber wants to move into pizza delivery), then you should spin up a new dedicated team, but just the discovery team is funded, comprising a Product person, Designer and a Lead Engineer.

How do you deal with companies that were innovating at one point and no longer are?

I asked this question around how companies that once were very innovative places degrade to places that no longer innovate. For instance, startups that iterated like crazy at the start to get to product/market fit as fast as they could, but then settle on a product and just scale it out.

Marty said that a lot of these kind of companies fall into the trap of believing the propaganda out there: that it’s a startup or you’re scaling. But you can’t stop innovating at any point: it’s right there in the title: “Continuous Innovation”. You have to do more than the basic optimisations.

Some of this is really about educating the leadership of this. Skunkworks projects and hackathons are great for this. I asked Marty if he thought that going dark (where you basically stop communicating to allow the team to work in isolation) was an effective strategy. Marty said to be careful here, if you go dark for too long, then there are other problems that arise. Overall, Marty believes you can’t wait for permission, you’ve just got to do some stuff.

Marty recognised that there are different approaches for product discovery at a startup versus a more established company. One of the nice things about being a startup is that there’s nothing to protect in your tests: there’s no brand, no revenue, no customers! Your only mission is to go find something that gets traction. On the flip side, there’s no money and it’s a race to get to Product/Market Fit before the cash that you do have runs out.

Once you do get to Product/Market Fit, you can’t just stop the Discovery process now that we have traffic and we have real customers. A lot of companies waste that traffic, and are just sitting there waiting for a competitor to come along and disrupt them. Companies still need to be testing, but in a way that protects brands, revenue, employees (there are techniques to do this in a safe way that does protect these things).

Everyone should be doing A/B testing but you should especially be doing A/B tests for highly risky stuff so that you’re trying new ideas. Mostly, A/B tests are done blind, but sometimes you cam break this rule and explicitly invite people into your test (like a beta). This will skew your numbers a bit upwards (it’ll skew left on the adoption curve) but you can get better information.

How do you do this discovery process with a remote team?

An example was posed of 37signals: they have been successful at product innovation with a fully remote team (they even wrote a book about it!) - does this approach work with such a team? Marty countered with the explanation that 37signals is actually mostly designers, and they get to pick what they want to work on. So, a pretty atypical organisation.

Marty told us it is possible to work remotely, but it’s just not the best way. Sometimes you’re constrained and have to work remotely. For instance, MySQL wanted the best resources for their team anywhere in the world, so that meant having to work remotely.

In general, you want:

• Plan A - colocation
• Plan B - Product Manager, Designer and Lead Engineer are colocated together. Rest of Engineering team is remote. The Lead Engineer is the liason to the remote Engineering team.
• Plan C - Product Manager and Designer are colocated together. Engineering team is remote.

Where do ideas come from?

You very rarely find a company that doesn’t have ideas. They’re typically stored up in a backlog somewhere. The hard part is the conversion of ideas from the backlog into the discovery process.

If you can do 5 ideas a week, then you can work through a lot of that backlog.

When it comes to assessing ideas that you do have, ask yourself:

• Does this idea take you in the direction of your vision?
• Does this idea take you in the positive direction of your KPIs?

If you answer yes to both of those questions, then an idea is promising.

Most of all, listen to your engineering team. They’re closest to the technology (and are most aware of new possibilities in other tech). But really, recognise that anyone is smart and will have ideas. You should also watch your customers using your products.

Remember, our job is to quickly separate good from bad. If you can assess 20 ideas in a quarter, then your goal should be to get to a 100 ideas in a quarter.

What does it take to do this?

You need to spend a lot of time ideating, prototyping and building.

We can use our backlog as a measurement tool:

• If the backlog is empty, then Product is failing to come up with new ideas
• If the backlog is weeks or months long, then Engineering is failing to build out ideas

Another way of viewing the Continuous Discovery model is as a Dual Track Agile approach. Every idea on the backlog is fully baked (designed, validated and ready to go), so that Engineering can just pick it up.

We’re really looking to assess changes to throughput.

How do you balance between continuous discovery and continuous delivery?

A lot of optimisations are just the small, easy ones (like tweaking a colour for higher conversions). Just pump these ones out.

What should we be focusing on instead? Validating the high risk ones. You should use your judgement based on your KPIs for the best things to do.

The reality is that it will depend on the team and that point in time. For some teams, the right thing to do based on the data will be just to do more small optimisations, or focus on bug fixes. For other teams, the data might say that Engineering is slowing down (it just takes too long to build things), so the focus should be on eliminating technical debt.

How do I find time for this process with all the meetings I have to do?

After the laughter from this question settled down, Marty set out how the people on the teams should be allocating their time:

• PMs/Designers should typically spend all day in Discovery, with 1 hour a day set aside for working with Engineers in Delivery.
• Engineering should spend all day in Delivery (development, test automation, deployments), with 1 hour a day set aside for working on Discovery.

The exception to this is where a prototype requirements development, eg. a Live Data Prototype. In Scrum, this would be a “spike”.

Close

Marty closed his talk by reminding us of two big points:

It’s very important to keep iterating (multiple times).

Never consider a discovery phase a failure, it’s just what you learnt (a saying from Tom Chi).