INTRODUCTION
This article is inspired by my own observations, a somewhat prophetic warning in the book The Lean Startup, and the confusion that artificial intelligence has brought to the development cycle. The article has two messages that are mutually independent but functionally related. And yet
LEAN STARTUP RECOMMENDATION
Let’s imagine that we are a breakthrough company that has achieved market fit with its product. We are aware that this is just the beginning and that our product must grow through iterations. Of course, we don’t do this blindly; instead, we have a lot of analytics built into the product. We also regularly conduct interviews, surveys, and user testing. We measure churn rate, daily active users (DAU), retention rate, lifecycle value, etc. After an initial jump, the metrics show that we have more active and paying users month after month, but the growth graphs are quite flat.
Due to the not-so-impressive results, the team slowly begins to wonder whether the upgrades actually contribute to the mentioned growth or if it is a consequence of other factors. These could be, for example, marketing efforts, viral adoption, rising prices of competing products, or simply inertia. So, where is it most worthwhile to invest resources?
The solution proposed by The Lean Startup is for the development team to treat each iteration as an experiment. Consequently, this means that it is necessary to pre-define the metrics that new functionalities are expected to improve (and by how much!). If the planned metrics do not change after the release, it is time to redirect our efforts to another functionality or activity related to the product (pivot). If, however, the planned metric improves, this is an indicator that we are on the right track.
To prevent our indecision from lasting too long, it is better to use cohort analysis than summary graphs for analytics. The graph below is drawn from the same data as the one above, but it gives us a more realistic picture of progress. On the cohort graph, for each month, we can see what proportion of the total growth is represented by a certain type of user.
Although it seemed to us on the summary graph that the number of paying users was still somehow growing, the cohort graph tells us that new paying users in the last three months represented only one percent or less of all new users. Obviously, the functionalities we launched were not what would truly attract users. On the other hand, in the June release, we did something right (4% new paying users). It would be better to continue in this direction than to develop subsequent functionalities from a list. Although test users were satisfied with them, it seems these were not the functionalities that would be profitable for us or would convince new users of our solution. This information was not evident from the summary graph, but the cohort graph clearly highlighted it.
The recommendation, therefore, is to formulate a hypothesis before release, specifying which indicators we expect to change (and by how much). Based on the analytics, we then decide whether to pivot or persevere. Companies that adhere to this recommendation already outperform most of their Agile competitors.
Now, we will apply this insight to the world of AI-powered product development.
WHEN PROGRAMMING IS NO LONGER THE BOTTLENECK...
Although we are all quite tired of discussions about AI, the idiotic prophecies of unprofitable AI giants (and their IPO hallucinations), the fact is that AI has changed the way new products are developed.
The essence of Agile development is rapid and inexpensive validation of ideas with the client. We do this with prototypes and sometimes MVPs. MVPs are more time-consuming than prototypes, but they can generally be immediately integrated into production.
Paradoksalno AI ni odpravil ozkega grla razvoja, ampak ga je samo prestavil drugam. Včasih je razvojna ekipa čakala na prototip. Danes prototipi čakajo na uporabnike. Vsak posameznik v razvojnem timu lahko zdaj dnevno kreira množico kvalitetnih prototipov ali MVPjev. Če želimo povratne informacije, je vse te prototipe potrebno testirati z uporabniki (kvantitativno in kvalitativno). Ozko grlo razvojnega procesa se je torej premaknilo z zamudnega razvoja prototipov in MVPjev na njihovo testiranje. Tega se s pravimi tehnikami da pohitriti, a pri tem ne smemo pozabiti kaj je naš cilj (produktni cilj, KPIji…). V nasprotnem se ne bomo mogli odločiti med na primer tremi konkurenčnimi prototipi iste funkcionalnosti.
There are two proposals for the effective utilization of this avalanche of prototypes and MVPs.
1. Define the desired outcome before you start testing
This is the Lean Startup recommendation we explored in the previous chapter. If we have ten high-quality prototypes or MVPs in front of us, we need to choose between them. The best way to make that choice is to rely on the KPIs we defined before the experiment. In doing so, we establish a hypothesis that the experiment will either validate or invalidate.
2. Decide when to stop
High-quality alternative prototypes and MVPs for existing functionalities will continue to emerge. To avoid neglecting other functionalities and product growth, at some point it will be time to shift focus. This can also be attributed to the 20/80 rule.
Since the influx of new proposals will not stop, it is advisable to pre-define criteria for when a certain feature will be considered sufficiently developed.
A few examples of such criteria:
- When 40% of users have used this feature at least once per session, we will consider the functionality satisfactorily developed and will focus on others.
- When 90% of users who started the registration process complete it, we will consider the registration process satisfactory and will focus on other and new functionalities.
- When the NPS for the core functionality improves by 10 points, we will focus on integrations with other products.
- If the usage of this functionality does not increase from 2% to 10% by next Friday, we will no longer invest time in it.
CONCLUSION
With the accelerating pace of development, some already known product concepts are becoming more critical than a few years ago. With AI, generating ideas and prototypes is becoming almost free. However, the cost of validation with real users still increases linearly with the amount of testing.
The next step in optimizing the development cycle will be eliminating the bottleneck in user testing. This topic goes beyond the scope of this article, but initial attempts are already here. We are talking about:
- entire synthetic users,
- AI as a pre-filter for real user testing,
- multi-agent simulations where AI agents offer your product to other agents with different functions,
- user digital twin built on existing user data
AI user testing will not displace classic user testing, but it will optimize it. Its limitation will remain the identification of:
- completely new behaviors,
- unexpected reactions,
- new needs,
- “black swan” users,
- cultural changes.
