The most common questions asked of Testers and Quality Assurance professionals are “Has this been tested enough?“ and “Can we put this live now (no pressure!)”. And the awkward response given is always “err, well it’s not that simple…”.
At NHS Choices we have transformed this response over the years into “it is if the product owner is happy”.
A Look Back
This started three years ago with Transformation of the Test Team, both structurally and culturally. We moved into a matrix management approach, akin to the Chapter model professed by Spotify. This meant sucking up of egos and typical line management hierarchies and releasing day-to-day management of individual testers to delivery squads. This has since been cemented by the adoption of Assignment and Career Managers in the new Operating Model, but at the time this meant throwing the approach of delivering software into a testing department out of the window.
NHS Choices was becoming Agile with the adoption of Scrum techniques and its evolution to Scrum-Ban and so it wasn’t appropriate for Testers to sit in an ivory tower waiting for release candidate products to be thrown over the fence – only to be thrown back again at the first sight of a major defect. Instead testers became part of the delivery team, responsible both for the product and its delivery. Their role was to be quality champion with input not just into verification of changes, but also delivery processes, story refinement and customer satisfaction. This meant adoption of significantly different skills, moving away from pure testing techniques to include stakeholder management, business analysis and team collaboration.
The result though was what every business strives for, lower cost of quality. Delivery Teams were able to identify defects earlier. When a defect is found earlier it is cheaper to fix, there is less development rework and the customer can be satisfied quicker. One of the key tools delivery teams were able to adopt was Story Refinement using the INVEST criteria. Testers joined with Product Owners/BAs and Developers as part of ”Three Amigos” sessions where stories (requirements) were captured and reviewed. Involving ‘technical’ people at this early stage in delivery meant that potential defects, both with the product and the story, could be found and fixed earlier. This led to more conversations about how something ”may be a good idea but it’s not technically possible” and then on to discussions about alternative ways to make customers happy.
This collaborative approach continues throughout the delivery pipeline, Testers acting as a voice of the user while applying a testing specialism. However, this still didn’t answer the challenge of ”Has this been tested enough?“ Testers within delivery teams were now wearing three or more hats but still only available for the working week, how could we possibly be doing enough testing??
Current Day – Managing Risk
We had to innovate how we measured quality, we adopted the current trend of Risk Based Testing. Managing risk is an important factor in customer satisfaction, as adopted in ISO9001:2015, and with product changes we needed a full understanding of how delivery teams could control this. Previously a tester would never have said ”this has been 100% tested” or ”this release is good to go”; there is always one more test that could be run (no matter how expensive) or one more approach that could be considered. We extended this idea into a ”Testing recommendation for release”. This is not a sign off, rather providing information for the product owner and the delivery team on which their approval decision can be given to the change board for go live. A Product Owner will have multiple things to consider when approving a release including priority of the project, effectiveness of the delivery process and deadlines. In hindsight basing a Go Live decision just on test results seems a bit narrow minded.
Testers use customer analytics, product design and a bit of gut feel to plan their testing. Testing is targeted at the highest risk areas of the product change first and by doing this we can pseudo-apply the pareto principle and diminishing marginal benefit. We accept that software will have unresolved defects, but we recognise that the impact of those defects is so small or so unlikely to affect customers that we are willing to accept them and the cost of supporting them in production. This is applied at the story level so that product owners accept each story, agreeing that the acceptance criteria have been met. These stories then build up to make the MVP and not all stories will be accepted for going live. This fits nicely with the adopted agile approach of iterative delivery which allows defects to be resolved in subsequent releases.
The Future – Not Another Ivory Tower
So the Tester becomes the voice of quality in the delivery team and provides a recommendation for release, but it is the product owner who is responsible for the decision to go live.
All that is great, but we don’t rest on the laurels of the new delivery process. We can’t fall into the trap of believing that just because our new method is better than the last that everything will always be wonderful. Instead, we are agile, we keep tweaking our approach and learning lessons. We are implementing Test Automation to provide more capacity for exploratory testing and we are sharing best practices with our industry colleagues and learning from the challenges they have faced. We have not reached our destination, we are learning to enjoy the expedition.