Agile Is Not Incomplete or No Documentation — Agile Means “Right” Documentation.
Agile methods have been growing in popularity since the first formal adaptations in late 1990s to early 2000s. Agile has its own fan clubs and is fragmented in to practice areas, from simple, single team based Agile Scrum to complex models like SAFe Agile. Besides the more popular Agile methods, there are relatively lesser known yet equally effective methods like Crystal, DSDM and Feature Driven methodology. Regardless of the method, all Agile methodologies share much of the same philosophy and features – regular feedback loops, speed, flexibility, and above all, regular communication and interactions between stakeholders, whose roles have been clearly defined and limited in all Agile methodologies.
As the focus has grown on speed, flexibility and continuous delivery, many Agile practitioners have come to regard Agile framework as highly flexible, evolving structure without rigid guidelines, rules, or methods. Agile teams have at times contended that documentation assumes lesser priority in Agile methods while some consider maintaining documentation a matter of convenience. Along with varying viewpoints, people have come up with their own adaptations of Agile practices and guidelines, merrily choosing one while ignoring other.
Why is documentation important?
All software systems are complex products developed and delivered through a series of steps and / or processes. Irrespective of the method used or the skillset involved, each software begins with an idea and translates into a product, prototype, document or piece of code to be used for the next project. Whether a product, document, prototype, or working software, the artifact created in one step becomes the input to the next step.
The Agile Manifesto states as one of its core principals, “Working software over comprehensive documentation”, hence many people assume that Agile means little to no documentation. Nothing can be farther from the truth. In reality however, it is often a fallacy; and nothing more than a common misconception of those inexperienced with agile. Thinking that a project can be delivered more quickly and easily by avoiding documentation is often akin to not doing enough testing in order to save time.
Image Source: Mastering Business Analysis
Documentation serves an important purpose. In particular, project teams want to capture high-level information in documentation but not details, which is largely fine as long as documentation requirements are met. You will still need to capture important information permanently, and sometimes regulations require certain levels of documentation (yet another requirement)
- Documentation is integral -: Documentation is as much a part of the system as the source code. In addition to working software, Agile teams need to minimally deliver user manuals, support documentation, operations documentation, and system overview documentation.
- Initial stages and recording agreements -: Documentation plays very important part in recording understanding of everyone on the project team with regard to the overall scope of work and their individual wok areas and the points of integration. Documentation pertaining to requirements and recording definition of done, recording understanding of solution and design documents are integral part of project documentation. Without this documentation, most of the project work can come at risk and re-work can become the least of project team’s headaches.
- Inputs to next work effort -: Building high-quality working software which meets the needs of the stakeholders is important, yet its equally important to ensure that the software can be maintained and enhanced throughout its useful life. Documentation helps regular operation and support functions and without adequate documentation there are higher chances of wasted work efforts and re-work.
- Not all projects are created equal -: Projects vary in degree of complexity and time taken to complete. Complex projects demand more concise, well-detailed documentation than simpler projects. Similarly, working on new systems or new requirements means more documentation is needed as there is little pre-existing knowledge or resources to fall back upon. Working on complex and unknown systems means more documentation as interactions will need to be understood by the team.
- Maintain transparency and communicate better -: documentation helps teams communicate better as well as maintain transparency in to the team’s activities. It’s a fool’s world to think that project teams exist and act in isolation. In varying degrees, whatever happens on a project team, impacts other teams around.
- Ideas and tasks cannot just be all in people’s heads -: stuff that’s not documented is sometimes lost as soon as the project is over, and definitely lost over an extended period of time. Documentation however survives as long as it is considered relevant and maintained.
Image Source Scott W. Ambler
Unlike waterfall project management, where the focus is on clearly defined documentation, and documentation is a project phase unto itself, Agile views documentation effort as only what is bare minimum essential to the project.
Agile encourages “just enough” documentation as may be required for the project to meet the expectations of customers and stakeholders. Among other things, the level of documentation needed for a particular project depends on newness of system and requirements, maturity of project teams, complexity and scale of the project, future needs, etc.
The documentation on an Agile project can be in the form of white boards or brain maps; or sticky notes or writing on flip-up chart papers. Pictures, rather than lengthy documents can be the flavor of the day. Similarly, videos can be created of project team members explaining requirements or detailing solution or just how any part of the solution would work. Agile does not believe or recommend in documentation just because some stakeholder feels that each and every detail of the project must be captured in writing to retain knowledge for future. The aim of Agile is release increments better and faster. “Just enough” documentation fits in the overall picture of efficiency.
While some information will always need to be captured in written form or documented clearly, there are techniques that may be used to reduce amount of writing, but will still give the customers and stakeholders what they want. One way to reduce the amount of writing is to create diagrams through brainstorming using a whiteboard and capturing those as images saved by individually or as part of a document.
Image 3: Source Internet
Agile – Documentation and changing requirements
The standard waterfall requirements documentation doesn’t hold much water in Agile world. First and foremost, for any medium to large project, obtaining stable requirements upfront is impossible. The business needs and customer preferences themselves change over time, and in many cases business and customers aren’t even aware of what they want till they see a final working version of the product. In many cases customers realize they specified wrong requirements only when they see the final version of the product. The other problem with requirements identification is understanding the intent, or the unsaid needs of the customer. Perfectly well written, and word-by-word noted requirements turn out to be well short of the customers’ expectations because the team did not understand the unspoken requirements.
Image 4: Source Internet
The product backlog, where the requirement came from, and how it will be accepted by the customer (acceptance criteria or definition of done) are all various documentation that are essential and central to the success of the project. It cannot be optional, it cannot happen “whenever convenient.” It is a part of the development and acceptance process.
Documentation in and of itself doesn’t hold much value unless it is paired with an effective communication method. The focus in Agile world is, and should be, rightly on conversations first, and documentation later. In the above cases where traditional documentation of requirements will fail miserably, communicating frequently and holding conversations about product features regularly so that the customer and project team can build the product together holds the key to product acceptance. The simple idea is to come to an agreement point at the end of discussion about each feature or story point (where features are broken in to story points) and documenting the same. Once again, remember! Conversation and agreement first, documentation later. The agreement between customer and business and project team centers on what is the acceptance criteria (definition, roles, acceptance tests), which is later documented and forms part of Agile project documentation. This final output, after the discussions are done and the dust is settled, is the key towards a successful project as without this final step, of documenting what’s agreed, months of work and planning can flow down the drain.
Acceptance criteria & acceptance test form the core of Agile requirements. These documented agreements – Acceptance criteria and acceptance tests – cut the ambiguity around the requirement as defined by the customer, thereby improving the chances that dev team/s build the right thing, that stakeholders will be happy with. Further, it affords developers some flexibility in how they implement or develop the requirements, as long as the final product meets the criteria/tests documented in previous steps.
Image 1: Credit SkepticalAgile
In addition, acceptance tests and acceptance criteria bring about an important change within the customer organization. In many cases there are more than one or two stakeholders. Entire customer teams from cross-functional areas can participate in requirement discussions and its very common to find divergent viewpoints among the same team. Discussing and documenting acceptance criteria lends a seriousness to the entire exercise and forces the entire customer team to come to shared understanding, which simply would not exist without documentation. Divergence among stakeholders is the single biggest pain point while working with medium to large projects and Agile documentation can very well nip the problem in the bud.
At implementation level, these criteria and tests become the common language between project team, developers and the stakeholders. Acceptance criteria and tests established earlier become the basis for discussing changes in requirements, as well as how progress is tracked. Of course the acceptance criteria and thereby the defined acceptance tests can themselves change if requirements change or even otherwise following the same original course of conversation and communication followed by agreement and documentation. Furthermore, these acceptance tests can be automated, using Acceptance Test Driven Development (ATDD) tools commonly available today.
Post the requirements’ documentation, another sticking point with many is the topic of Technical documentation. Technical documentation is meant to be a description of the software as it exists including all the changes to the original version thus far as on the date of recording. The purpose of technical documentation is to allow continuity, i.e. to allow a system to be maintained by future developers, long after the original team has left or as long as the system is deemed to be useful. In reality, however, technical documentation, such as UML diagrams, may not be very helpful in solving day-to-day development problems, and are often found to be out-of-sync with the actual behavior of the system.
At the core of Agile technical documentation is documentation through tests, in what is normally referred to as Test Driven Development. When the development team starts work on a story or feature, the testers and developers come together first to define and automate the acceptance tests. This helps on more than one front. First, this helps the team maintain their focus as they implement the acceptance criteria agreed earlier, by including only those factors that contribute towards meeting acceptance criteria and rejecting those that don’t. Second, this helps the team gain a common purpose and work towards the same goal in the most time efficient manner as the inconsistencies are resolved upfront. Thirdly, this acts as a final level of check to identify any tasks which can derail the entire project. Test driven development’s core feature is to only include the tests which are appreciated by the stakeholders, and which amply demonstrate the product features and acceptance of those features, rather than including mundane or repetitive tests.
Lastly, project teams’ maturity, whether it’s a new team or has worked on several projects in the past together, is an important consideration on the amount of documentation needed. It is extremely important to document, starting from requirements to acceptance tests, however the level of documentation needs to be managed at project level.
Image Source Dilbert
While it is true to a large extent that Agile methods advocate doing away with rigid guidelines, documentation and maintaining artifacts is not a rigid requirement to begin with. Lets keep in mind, agile is first of all a set of values, not a settled and strict, rigid framework. All Agile practitioners should be confident on how to adjust these values to their environment. Remember that every company has its own pace, characteristics and values.
All Agile documentation must follow the same rule – it is not that agile eschews documentation, so much as it eschews up-front documentation.