Every software product blossoms from a bright idea. Ideas are like little gems floating in the ocean: sparkling on the surface, yet difficult to catch.
Pushing an idea into a fully working feature in production is the central story in every product team’s life. We are familiar with the software development lifecycle and all its variations. In simplest terms, it starts with requirements gathering, followed by design, development, testing, and finally release. It sounds straightforward. On picture, the arrows and circles are simple to understand; one step leads to the next, either in short or long iterations.
But this picture is inaccurate. Turning an idea into usable software is indeed iterative. Yet, each iteration is not a linear process. Some claim that software product development is like a factory assembly line. Hence, we took inspiration from product manufacturing value streams, such as car production, and adopted it into our software engineering process.
But we missed one major detail: the key equipment in software product development is the collection of human brains working either simultaneously or asynchronously to build something new – it’s nothing like any machine in your typical factory.
How our brains form ideas
It’s no surprise that forming novel ideas requires a complex interplay between our memory, which houses our knowledge and prior experiences, and our brain’s control system, which pushes us to “think outside the box”. Research shows that several parts of the brain are activated during creative thinking; it’s typically tested by asking people to think about other unconventional uses of a basic object such as a cup1. These studies scientifically validate what we already know: that our memory will instruct us to default to what has already been proven to work, while our ingenuity will challenge us to imagine a better way. Connecting these two is the recipe for innovation – the ultimate goal of technology.
Software product development is in essence building the future; we’re creating an experience that will allow us to do certain things in a way that has not exactly been done before. We are constantly forming ideas; not using molds and templates to mass produce a steering wheel. It’s a creative process that involves consistently churning informed choices and acting on it, rather than repeating tasks and following a formula in a fixed cadence.
The flaw in the flow
The reality is software product development is still a project; it’s a venture that relies on finite resources with a deadline. That’s why we often require teams to organize their activities to get from point A to B with the least amount of time and money.
So, we follow a straight line within a loop: ideas must be written as requirements; requirements must be converted to specifications; specifications must be implemented as code; code should be tested, built and released. Then, we repeat. It’s a logical and supposedly cost-effective system.
If it was that simple, how come many product teams are still failing to keep up? Millions of dollars are wasted on unrealized ideas that never make it to production.
I don’t claim to have the absolute answer to that. But I do know from experience that there is one flaw in the flow: it’s failing to accommodate the nature of the creative process that is intrinsic to software product development.
Creative ideas are notoriously spontaneous
“It just popped in my head” is an expression that we often hear, even in software development. How many times have a colleague told us that they thought about a bug fix while doing a random house chore? How about a user who only came up with a new requirement half-way through the sprint?
Instead of saying: “we should have thought of that earlier” or “why didn’t you say so”- blaming rework to incomplete acceptance criteria, design spec, test case or line of code – we should accept that we couldn’t possibly imagine all these information ahead of time; that some of these just “came to mind”. Sure, we need to try our best to think ahead and be as creative as we can be during ideation, design and planning sessions. But even the best of us will miss an important point or two.
Perhaps, it’s time to change our ways; embrace this sporadic strike of brilliance into our day-to-day flow.
But isn’t this what Agile is already about? To a certain extent, yes – but, not entirely.
We’re still scheduling ideations and planning at the start of a sprint and committing to delivering exactly what we have agreed on by the end of 2-3 weeks. However, what happens in between these days is crucial to creativity; it’s these days that good ideas begin to emerge from the corners of our brains, stimulated by each new information we unravel from working together or alone in deep focus.
The problem is that we keep setting aside these brilliant ideas because we’re wired to stick to the plan.
So, should we drop ideation and planning altogether? Of course not. But we should embed ideation at any point in the process; limit the planning to simply listing high-level features and specifying qualified, research-backed reasons WHY they are valuable. From there, we can pick the most high-valued items one at a time; get started and don’t stop until that perceived value is realized.
Trying to identify all the acceptance criteria and estimating the “points” or “size” during a four-hour meeting every two weeks could introduce the flaw in the flow: it’s a creativity blocker.
Whereas, if a team is given a clear picture of the value that they will help realize and is allowed to use the rest of the allotted time to be continuously creative in their pursuit – fluidly moving back & forth between ideating, refining, designing, coding, testing and releasing in an arbitrary, at times parallel, and yet harmonized sequence – like a paintbrush from the palette to a canvas, following no defined stroke and beat – then perhaps the team can still deliver a masterpiece, just in time.
Idea-driven development
I’d like to call this process “idea-driven development”.
In this flow, the team is hyper-focused on a central idea for a feature and doing everything it takes to get that feature to production to achieve the perceived value.
Each feature cycle can range between 3 days to 6 weeks, depending on the complexity and the timeliness of the central idea. High-valued features are most likely the most in-demand; it is important that the team and the stakeholders can mutually agree on when it will be delivered without sacrificing the team’s freedom and creativity that is anyways needed to make a great feature that users will buy.
During the feature cycle, the traditional development tasks and activities such as designing user experience, identifying test-cases, coding, modeling data and behaviors, drawing diagrams, uncovering acceptance criteria, refinements, observing behavior on production environment, etc., can be done simultaneously, in any order the team see fit, as long as they are collaborating and ensuring that each step will bring new learnings, refining the central idea until they get closer and closer to harvesting the expected value on production.
Where do ideas come from?
Ideas can originate from any of the above activities, not only during ideation and refinement sessions, but also while coding, testing and observing a select number of “beta” users on production.
An idea can be anything: a change in the user experience, an additional field, a code pattern, a better external library to use, a tweak in the architecture etc. It’s easy to recognize; It’s typically an answer to these two types of questions:
- Usage question: “How will the user interact with feature xyz?
- Implementation question: “If we implement abc, would the user be satisfied with feature xyz?”
Any answer to these two questions is an idea. The team can then evaluate each idea and choose to either disregard or act on it. An action can be anything from drawing a diagram, testing a hypothesis, updating a UX mockup, changing a line of code, provisioning a new infra – you get the point. The output of any action are the actual building blocks of the feature.
To illustrate this flow:
Let’s look at a concrete example:
Let’s say that your team is building an app for tracking a user’s stock investments across all markets. One of the highest-valued features is that every day the user can see their total gain or loss across their entire portfolio.
First, during planning, the team can simply note the following:
Feature: Display daily gain/loss percentage
Why is this valued by the user? Because it helps them evaluate their investment risks so they can make informed decisions about their life e.g: should they send their kids to an ivy league school or not?
Then, the feature cycle begins and the ongoing rounds of ideation, design, development, testing and observing kick off. For example:
During the initial collaborative session, the team can work on forming ideas and agreeing on actions:
Questions and answers (ideas):
- How will the user view the total gain/lost percentage of all investments?
- show a gain/lost tile from the dashboard page and the landing page
- show green + when it’s gain and red – when it’s lost
- If we recalculate the gain/lost percentage and cache the aggregate each time a new investment record is added in the database, will it make it faster? (will the user be satisfied with the feature?)
- Probably yes, but it will add an overhead to sync the materialized computation as opposed to doing in-memory calculation
After this session, the team can work on the actions to execute their ideas, either still collaboratively or alone in deep focus:
Actions:
- Update dashboard page and investment page mockup
- write unit tests for computing gain/lost
- implement calculation and caching as db change-triggered stored procedure
- test the performance
- write api specs that ui can consume
- implement GET api
- implement UI
- deploy to prod with feature flag and test with beta users
- release to all users
At any point in this cycle, team can call for collaboration sessions where new ideas should be noted and discussed, and new actions can be added or disregarded
The cycle is done when most users are happy with the new feature and can make decisions based on their investment portfolio status.
Systematizing creativity
It may sound like a sacrilege to put a strict structure around creativity, but even artists follow their own sets of principles and practices to achieve a timely result. Creativity is only a means to an end. Exchanging ideas is just the thread in the flow. The most important thing is still the finished product – and that requires a certain amount of discipline and a delivery system.
An idea-driven development is a more pragmatic approach to solving a problem. To a certain extent, it’s heuristics-based, rather than algorithmic. Everyone in the team must contribute; they are expected to combine all their expertise and ingenuity to arrive at good ideas and have the tenacity to see it through. The most critical risk is that they get stuck in an idea limbo.
Here are some ways to avoid chaos during the creative feature cycle:
1. Timebox a feature cycle
To avoid getting stuck, the team should still be responsible for estimating the length of the cycle, without being constraint to a specific cadence e.g. every 2 weeks. The team should be given the freedom to propose the cycle length, whether it’s 3 days or 3 weeks, and negotiate with the stakeholders as to align with what’s feasible for the business.
2. Collaborate as needed
Collaboration activities come in many forms. It can be a formal meeting, a chat, a quick call, an email, a note, or a demo. All these activities are necessary to feed ideas, get clarifications or unblock an action. Apart from a few meetings that must be planned due to people’s availability, collaboration sessions should be on-demand, as opposed to scheduled. The team decides if there’s a reason to collaborate and should have a concrete agenda and outcome. They can also choose which type of collaboration is most effective. Not everything should be a meeting, but some things are easier to fix during a call.
3. Block deep focus time
It’s very important for everyone in the team to block deep focus time during the cycle, to turn ideas into actions and real results. It’s also critical to learning and hence, to discovering new ideas that can help make the feature better. Each individual member should be responsible for setting a goal and expected outcome from their deep focus time. Otherwise, they also risk falling into a thinking limbo, or worst, procrastinate.
4. Keep a feature notebook
Taking inspiration from famous artists and geniuses, a team’s most important tool in an idea-driven development is a shared notebook that contain the evolution of the central idea into experimentation results, design choices, lessons, feedback, more ideas and more choices that went into building the feature, and ultimately the product.
Like an artist’s notebook, it will be filled with historical artifacts of the team’s ingenuity and give clarity on the why and how the feature was implemented.
Is this flow for your team?
If there is one thing we have learned from all the software development lifecycle frameworks, it’s that there’s no one flow that rule them all. Idea-driven development requires a certain maturity. The team must also invest in cultivating a high-functional culture where teammates have a good relationship with each other, can communicate effectively and are capable of self-managing.
At the end of the day, it doesn’t matter which framework or process flow you go for. If the team catches the gems in the ocean and uses them to build a brighter future, we’ve done our jobs.
Resources: