Verizon’s globally distributed internal IT department and its captive development center in India traditionally used waterfall techniques for building large IT systems that provision and activate its core network for voice and data. A typical product development cycle would run from 3–6 months. It has a change control board to approve any changes or enhancements after the requirements document has been signed off on. The development teams were distributed 70% in India and 30% in the U.S.
Initially during the BAAIS Video implementation, Verizon followed its time tested waterfall model. This caused some unanticipated fallouts as noted below:
- Since video was a new domain for Verizon, the requirements for the project were changing frequently, as the LOB (Line of Business) clients were learning simultaneously.
- Long delivery cycles of four months, with no earlier access to the system for that particular phase, provided no opportunity for feedback from the clients. If the requirements were misunderstood or miscommunicated, they were realized only towards the end of the release phase, leading to huge delays.
- A gated model with required sign-offs meant longer requirements and functional design cycles up to 3–4 weeks, as every detail had to be captured.
- The detailed design, development and unit testing were split between onsite and offshore developers, and both teams would work on the same phase with interdependencies. This led to constant miscommunication during hand-offs and hand backs between onsite and offshore development teams.
- System integration testing (SIT) was done by a testing team eight weeks into development and would last for four weeks. There were always integration problems with code for up to two weeks into the SIT phase without a stable build to start testing. The developers ended up having to fix integration issues, even when the detailed documents were created to avoid such issues.
- This created high bug counts and unstable releases leading into the user acceptance testing. If the implemented features were not as expected, it led to unhappy clients and often resulted in a delay in release.
- Late requirement changes could not be accommodated and were especially difficult to communicate to the offshore team.
To address these challenges, we started making multiple changes in our development process:
We started documenting the test cases for user acceptance testing during the high level design stage. This approach brought the testing teams to the requirement meetings, where they helped to formulate different test scenarios. We also started involving representatives from offshore teams so that they receive the information first hand. This helped to greatly reduce mis-communications and gave the offshore team a heightened sense of responsibility to deliver the project.
To address the issue of code integration between offshore and onshore teams, we introduced a development server to perform automated code builds and a regression test every day after the first two weeks of development. Since the test cases were developed ahead of time, we could automate those tests for regression using simulated interfacing systems. Developers were required to check-in code often. The daily code build ensured that the compilation dependencies were resolved, and the regression test suite would catch any bugs that would break the expected functionality. This helped avoid any huge functionality gaps or code integration failures to creep in.
Once we had these tools in place to improve the quality of the development and delivery, we started to turn our attention towards the constant requirement changes from the clients. We started by creating a demo server, where the working version of the product was deployed, and which the clients had access to. This demo server was updated with the latest working version every four weeks. The possibility of changing the requirements before it was too late and the regular visibility into the working product led to a reduction of the detailed requirements design phase to less than one week from four weeks.
Since the overall project team size was less than 20, it was possible to conduct a short online meeting every day or two to discuss new plans, changes in current plans, issues between interfaces, etc. Each meeting would be led by lead developers on a rotation basis, and the meeting was automatically logged (for the managers to look at) since it was online and not a call. This way the team learned to be more self-organizing and more responsible.
Above changes made it possible to deliver a large project involving a globally distributed team without sacrificing the quality or the productivity. Both onshore and offshore teams became equal partners in the project with both approaching clients with equal visibility. Though Verizon is not a very process oriented company, just a due diligence effort to improve process led to agile/iterative techniques without a conscious effort. Though not all difficulties due to global collaboration were eliminated, most of them were definitely mitigated, resulting in a better product, a better (time) managed product life cycle and happier customers.
It had been stated that offshoring decreases the productivity by 0.7% for every 1% increase in offshore, and reduces quality in terms of the number of bugs increased by 1.5%. In my opinion, with this kind of methodology, though productivity may fall initially, it does improve over time with better coordination and by finding a non-physical way to reduce the proximity between the global teams. Like Martin Fowler quoted in Using an Agile Software Process with Offshore Development, "…separating teams by functionality and not activity boosted their morale and increased their sense of responsibility towards the final delivery of the product." This was well executed at Verizon in the example I gave above.
In conclusion, the iterative or agile methodology is an efficient way of managing globally distributed software projects. Agile methodology simplifies the communication and coordination problems rather than increasing them, and is a better approach than resorting to the waterfall method for solving technical and organizational challenges while distributing work globally. It also accommodates changing customer requirements, and there are a lot of success stories out there, as demonstrated by the articles and my example. Companies setting up offshore units just need to invest a little time implementing their own successful global delivery model using agile methodology.
No comments:
Post a Comment