Agile software requirements engineering: How to efficiently communicate, document and stay adaptable

In November 2020, our fellowship team started work on supporting the development of a more user centric military to civilian digital transition experience for Canadian Armed Forces members as part of our government partner Canadian Armed Forces Transition Group (CAF TG). To learn more about the fellowship, our team and the project, please visit our public Wiki.

Need For A Common Language

You might have heard of the Tower of Babel as an analogy for software development projects: when people speak a common language, they might succeed in very challenging projects. However, if everyone speaks a different language and can’t communicate, the project will surely fail.

In the Tower of Babel story, people were successfully building a tower reaching to the heavens until they started speaking different languages. This story illustrates why requirements engineering and clear communication are important for successful software projects.

The software development effort involves various stakeholders from different backgrounds and expertise. Each of these stakeholders is comfortable with describing things in the language of their expertise and might be unfamiliar with the language of the expertise of others. We need to find a common language for effective communication between stakeholders with different backgrounds to increase the chances of having a successful software project and avoid project failure due to communication problems.

One of the main challenges that we were asked to focus on by our government partner was “How can we communicate what we need to our developers more clearly and validate what is delivered?” Agile Software Requirements Engineering was in our toolset to help.

Finding the Common Language Through Behavior Driven Development (BDD)

While software projects can fail for many reasons, they mainly fail due to communication problems. For example, imagine a project where a product manager fails to clearly describe use cases of a proposed functionality. In this case, the developer can misunderstand the scope of a feature and the product team might not be able to correctly verify if a feature is actually done. [1]

Through the use of Behavior Driven Development (BDD), we can use simple, structured language that all stakeholders understand while developing the software requirements. This way, we can also enable the path to conveniently automate validation for product requirements.

A BDD story consists of a narrative and one or more scenarios that are provided as acceptance criteria. Because it is written in a structured language, the acceptance criteria requirements can be directly used for writing automated tests by using various testing frameworks.

Finding the Requirements Through Requirements Engineering (RE)

While using BDD as our common language to specify the requirements can solve communication problems, we still need a way to identify and manage the requirements. The subfield of software engineering that looks into how to find and specify requirements is called Requirements Engineering (RE). The ultimate goal of RE is to facilitate a common view and a shared understanding of the requirements amongst all parties involved in the development effort. [3] BDD can be used in RE to achieve the ultimate goal.

Agile Requirements Engineering

Requirements Engineering (RE) intends to ensure that the team is building the right product. For this reason, it should be part of any agile process in some form to reduce chances of project failure especially due to lack of planning. Unfortunately, due to its use in past traditional practices in waterfall-like processes, there is a misconception that RE is only for pre-development requirements documentation and it can not be part of an agile process. However, requirements engineering can be done in any form as long as it leads to common understanding and building of the product that the users need.

Based on the project’s need, agile RE can start during pre-development planning or it can start at the same time as the development. Key concepts to keep in mind for an agile RE are:

• Stay flexible regarding the process and what is not known.

• The requirements will change, they are not finalized.

• The requirements are the responsibility of the whole team.

Since Agile Software Requirements Engineering is a relatively recent software engineering topic, it is not completely explored and understood. [4] In the appendix below, we survey recent literature to understand industry best practices and learn where we can improve the process.

What About Documentation?

A common misconception is that there is no documentation in agile. While the Agile Manifesto values, “working software over comprehensive documentation," this doesn’t mean no documentation. [5] Instead, there should be just enough documentation to avoid the creation of unnecessary artifacts. Agile does not mean avoiding documentation but documentation that is done efficiently, collaboratively, and adaptably.

All the artifacts that are created outside of a traditional formal documentation (including code, development tickets, automated tests, wikis, etc.) can be considered as a form of documentation. If a traditional formal documentation is necessary for the success of the project, that should also be created.

We also need to consider organizational needs around documentation. For instance, in our case, formal documentation was required for the approval and initiation of a project for organizational compliance reasons. The initial formal documentation is usually done in a way that targets business users — this usually needs to be processed further for a developer to work from. In cases like ours, my suggestion is to create the required formal documentation directly targeting the developers but in a way all stakeholders can understand. That way we can avoid the creation of extra documents.

After all of the considerations, we decided to produce a hybrid version of the software requirements specifications (SRS) document using an agile approach, while still documenting and standardizing the requirements to support service level agreements and development initiation. SRS is the standard traditional formal document targeting developers which describes the software that needs to be developed.

In my next blog post, I'll be sharing a step-by-step walkthrough of our process to create agile software requirements.

Appendix

Since Agile Software Requirements Engineering is a relatively recent topic, we looked into the following literature to see what has been done and get new ideas regarding best practices for an agile requirements engineering process.

Paper 1 - Requirements Engineering in an Agile Software Development Environment

Description: Reports on a case study of project requirement engineering processes and documentation of a military software development project, which seeks to identify the potential benefits of using agile-based requirements engineering processes.

Points Taken: Create links between stories to avoid scope creep. Add story controls to point to the regulations. Track the changes.

Paper 2 - Agile Requirements Engineering Practices: An Empirical Study

Description: A qualitative study of 16 organizations, targeting to answer the following two questions: What RE practices do agile developers follow? What benefits and challenges do these practices present?

Point Taken: Active dialogue results with better software.

Paper 3 - Integration of Agile Practices: An approach to improve the quality of software specifications

Description: The research proposes an exploratory study in the industry to investigate how requirements engineering is used in agile projects.

Point Taken: "Continuous collaboration with the customer" is a major challenge to the projects.

Paper 4 - Non-Functional Requirements Elicitation Guideline for Agile Methods

Description: The study presents elicitation guidelines for Non-Functional Requirements (NFRs) in agile methods.

Points Taken: The NFRs are often ignored as a bad practice. Relevant NFRs for Government and Military are "accuracy, confidentiality, performance, privacy, provability, reusability, security, standardizability, usability, verifiability and viability".

Paper 5 - An Approach to Support the Requirements Specification in Agile Software Development

Description: Based on the findings of investigations carried out in the literature and industry, the research developed an approach that is a set of practices to specify requirements in agile projects targeted to development teams called Requirements Specification for Developer (RSD).

Point Taken: This paper encouraged us to come up with our own approach based on our project’s needs. We tried the proposed RSD approach but found it too complex for our needs.

References

[1] Anastasov, Marko "How to build rock-solid Ruby on Rails apps with BDD", https://www.freecodecamp.org/news/how-to-build-rock-solid-ruby-on-rails-apps-with-bdd-735de9319cc6/ (current August 30, 2021)

[2] Nur Eren, Melike "Mastering Given-When-Then", https://www.adesso.com.tr/en/insights/adesso-blog/mastering-given-when-then.jsp (current August 30, 2021)

[3] Van der Zee, Sven "Requirements Engineering and Agile", https://re-magazine.ireb.org/articles/requirements-engineering (current August 30, 2021)

[4] Curcio, Karina, Tiago Navarro, Andreia Malucelli, and Sheila Reinehr. "Requirements engineering: A systematic mapping study in agile software development." Journal of Systems and Software 139 (2018): 32-50.

[5] Fowler, M. and Highsmith, J., 2001. The agile manifesto. Software development, 9(8), pp.28-35.

You're here to help residents, and we're here to help you. Interested in how to bring digital tools and skills into your department? No matter where you are on your digital government journey, learn more about how Code for Canada can support your work.