In a previous article, I described how, as a result of AI adoption, the bottleneck in software development has shifted from waiting for user story implementation (programming, code review) to user testing.
With the use of increasingly advanced AI tools, we have now been surprised by a new problem in larger Agile environments that would have been difficult to predict a year ago. This is the problem of team productivity comparability. The surprise is all the greater because we all more or less understood that in mature Agile environments, we had left the concept of productivity (and especially its measurement) on the scrap heap of history. Let’s look at where things have gone wrong.
CONTEXT OF THE CONFLICT
Story points (SP) allow a team to relatively estimate the complexity of work without pretending that complex development work can be accurately estimated in hours. However, their usefulness is based on a certain degree of stability: yesterday’s relationships between the complexity of different types of work must be at least approximately applicable tomorrow as well.
Procedurally, story points serve two purposes:
- Team uses them to estimate how many user stories it can commit to in a sprint with a realistic expectation of completing them
- Delivery (in story points) helps the Product Owner plan the release date and related activities (marketing, investors, sales…)
AI has introduced additional variables into this equation. In a scaled Agile environment where multiple teams work on the same product, there can be large differences in AI adoption maturity between them. Some use it only to assist with programming, others also for writing unit tests, still others for regression testing or code review. Consequently, the unit of effort required to develop a given functionality has become even more relative than was intended when it was created.
THE CONFLICT
Imagine we have two development teams:
- Team A uses AI intensively (requirements, UX/UI prototypes, programming, code review, testing…). Their velocity per sprint is 30 SP.
- Team B uses AI more conservatively or not at all. Their velocity per sprint is 40 SP.*
Note: development teams may have different definitions of the base SP, but this does not affect planning. Nor is it an indicator of team productivity.
Before I describe the conflict, one more note about rewards. We reward development teams “level-up“. This means that in a single-team project, we do not reward individual contributions, but the team as a whole. This shared responsibility encourages information sharing within the team and mutual collaboration among members toward a common goal (reward). In a multi-team environment, we reward the group of teams working on the same product as a whole. We do not measure or reward the performance of individual teams.
EXAMPLE
We are at a multi-team backlog refinement or sprint planning event. Teams are estimating the complexity of user stories in the product backlog. Sooner or later, Team A will start wondering why Team B estimated a user story at 21 SP (half of their capacity for the upcoming sprint), when they themselves would have estimated it at only 8 SP, which is less than a third of their capacity for the upcoming sprint. Is Team B avoiding work? Are they not trying hard enough? Are they not “team players”?
These questions are entirely legitimate, yet at the same time the seed of inevitable conflict. We are talking about an objective difference in the productivity of Teams A and B. This difference is not masked even by the different base SP size used by the teams. AI has caused a difference in team productivity for which no term yet exists in Agile.
Traditionally, in similar cases, we have resorted to the assertion that we do not measure the quantitative productivity of Agile teams, but rather the business impact of the solution for the customer. Measuring productivity immediately brought to mind companies that rewarded their teams based on lines of code written. I don’t think this practice needs special comment.
In the case of Teams A and B, differentiating between them based on business value created does not withstand contact with reality. The customer at the sprint review does not evaluate the work of Teams A and B separately, but evaluates the integrated product increment. The customer does not care which functionality was developed by Team A and which by Team B. What matters is only that the increment solves their business problem.
Imagine that Team A created the UX workflow and UI, while Team B created the API, integration, and data model. The new integrated functionality created 100k of business value for the customer. How much of this value was created by Team A and how much by B?
Team A is of course aware that they are rewarded “level-up” and that it is in their interest to collaborate with the slower Team B, but their reservations will remain. If this disparity is not addressed by the organization, it will sooner or later lead to resentment and hinder collaboration between the teams. Have we reached the point where we will start measuring the productivity of individual teams again?
HOW CAN AN ORGANIZATION HELP
Let’s assume we have confirmed that Team A is indeed more productive due to more extensive use of AI. In this case, the company must first stop deluding itself that Team B will be able to operate in the “old” way indefinitely. The sociological conflict arising from different team productivity will not disappear on its own, but will deepen until the entire organization suffers as a result. Team B must therefore change its way of working. The challenge for the company is how to do this as tactfully as possible.
In the resulting conflict, we can identify three gaps:
- Measurement gap: the organization does not have an appropriate metric to legitimately represent the difference in productivity. SP are not comparable between teams, and business value is shared.
- Capability gap: Team A knows how to do something significantly more efficiently with AI than Team B.
- Fairness gap: Team A observes that it must carry a larger share of the collective burden because of B.
The goal is to help Team B increase productivity without pressure or causing feelings of guilt.
MEASUREMENT GAP
The organization does not actually need a new productivity metric. It needs a mechanism that will show whether a better way of working exists and then spread it throughout the entire system.
The subject of comparison should not be the teams’ output, but their way of working. Our finding might be, for example:
“It appears that Team A is using a way of working that B does not yet know. What is this way of working and can we extend it to other teams?”
If, for example, it turns out that AI-assisted regression testing is a good practice, it is not turned into a KPI, but offered for use to other teams. Of course, the company must provide mechanisms and conditions for spreading good practices.
These can be, for example:
- pair programming where a developer from Team A joins a developer from B and helps them use good AI practices,
- a shared library of useful AI prompts,
- Lean Coffee,
- Communities of Interest / Practice,
- Scouts,
- Travelers,
- “Slack time”. More on this later in the article.
You can read more about the described tools in this article. One of the best collections of appropriate techniques is gathered under the name Liberating Structures.
Over time, we expect that the identified good practice will become a standard part of the development system.
Self-organization gives teams autonomy in choosing their way of working, but not the right to permanently ignore a demonstrably more effective way of working.
CAPABILITY GAP
Consider how we would view a team today that a priori refuses to use CI/CD, version control, or unit tests. At some point, a certain practice became part of the development standard. The same applies to good practices related to AI.
On this basis, we can define an AI capability loop that will help us formalize good practices:
- Observation: a larger difference has been observed between teams in estimating the complexity of a comparable type of work.
- Verification: we determine what is the reason for the difference. This is not necessarily AI usage. It could be, for example, a different problem domain, legacy code, technical debt, competencies, dependencies, etc.
- Knowledge sharing: the more efficient team presents its workflow to other teams. I wrote about tools for this in the Measurement Gap section.
- Experimentation: other teams test the new workflow in their environment.
- Standardization: if the improvement proves itself with other teams as well, it becomes part of standard engineering practice.
This is also important because Team A’s advantage today is not necessarily its advantage tomorrow. Next month, Team B might discover a significantly better AI testing workflow and become a source of knowledge for A.
With the described loop, we have established a foundation for organizational learning. In Lean terminology, this would be called Yokoten.
FAIRNESS GAP
It would be a mistake if the organization immediately converted every saving created by AI into a demand for greater output. This would remove precisely the capacity we need for improving and spreading AI knowledge throughout the organization.
Agile does not expect 100% utilization of a developer’s time. Moreover, so-called slack time is even desirable. Developers can use it for additional training, personal projects, communities of practice, Lean Coffee, or any of a multitude of group activities. The desire to maintain slack time is a consequence of the realization that continuous cycles of work optimization sooner or later lead to a so-called death march situation, which is unsustainable for developers.
A team that saves time with new approaches must receive part of that time in the form of “slack time”.
In this context, the greatest danger of AI in a scaled Agile environment is not that some teams will become more productive than others. The danger is that more productive teams will experiment faster due to additional capacity and thus become increasingly productive, while less productive teams will fall further and further behind due to lack of time for learning. The organization’s task is therefore not to equalize velocity, but to prevent the gap through systematic knowledge sharing and maintaining capacity for learning.
Therefore, slack time must not only be a reward for already achieved productivity. The organization must temporarily create capacity for learning even for teams that are lagging behind. If they are 100% utilized, this capacity will not exist. Consequently, we expect faster teams to devote part of their slack time to sharing knowledge with slower teams. After all, due to level-up rewarding, this is also in their interest.
CONCLUSION
There is no universal recipe for resolving inter-team conflicts resulting from different AI usage (and thus different productivity). However, I present my suggestions:
- We use good practices for collaboration and knowledge dissemination within the organization. Here I would emphasize the use of Travelers.
- We ensure that despite gradually increasing productivity, we maintain “slack time”, which is the driver of future improvements.
- We prevent competitiveness that would push slower teams into a defensive position.
- We plan tasks in advance so that (temporarily) slower teams do not create dependencies for faster teams.
- We use interfaces and simulations (mocks, stubs) so that faster teams can continue development without waiting for slower teams.
- We define clear API contracts that enable more independent team work.
- We tolerate a certain gap in team productivity. The foundational knowledge maintained by teams with lower AI usage must remain in the organization. When it is needed, it will still be available internally.
P.S. A bit more about story points
If the reader has by any chance concluded that the use of story points is pointless after all that has been said, I would like to address this dilemma as well.
Despite AI assistance, implementing individual functionalities still requires a certain effort. It is necessary to write prompts, connect agents, set constraints, debug, conduct security reviews. With some practice, this effort can also be estimated and converted into story points, which then serve the two functions mentioned at the beginning.
In terms of content, however, the benefit of estimating the complexity of user stories is much greater than just the Fibonacci value that is its direct result. The main benefit of estimation is the conversation during this process. The team uses, for example, planning poker where members with certain concerns or specific knowledge have the opportunity to explain why their estimate differs significantly from the estimates of other members.
The information obtained in this conversation leads to the process of:
- identifying and eliminating risks,
- spreading knowledge and information within the team,
- identifying effective AI prompts,
- consensus on communicating acceptance criteria to the AI system,
- buy-in from participants who have reached consensus on the complexity of the user story,
- identifying individual tasks needed to implement the user story,
- identifying interdependencies of tasks and dependencies with other user stories,
- UX optimization
The estimate in story points is therefore not just a number that helps with sprint and release planning, but an entire process of team alignment that enables easier and more effective work.
