Wednesday, March 9, 2011

Is Agile Architecture an Oxymoron?

Much has been written by many great minds on agile development processes and how (or indeed whether) architecture fits into such processes. People like:
  • Scott Ambler (Architecture Envisioning: An Agile Best Practice). "Some people will tell you that you don’t need to do any initial architecture modeling at all.  However, my experience is that doing some initial architectural modeling in an agile manner offers several benefits".
  • Mike Cohn (Succeeding with Agile). "On an architecturally complicated or risky project, the architect will need to work closely with the product owner to educate the product owner about the architectural implications of items on the product backlog".
  • Walker Royce (Top 10 Management Principles of Agile Software Delivery) : "Reduce uncertainties by addressing architecturally significant decisions first".
  • Andrew Johnston (Role of the Agile Architect): "In an agile development the architect has the main responsibility to consider change and complexity while the other developers focus on the next delivery".
  • Martin Fowler (Is Design Dead?):  "XP does give us a new way to think about effective design because it has made evolutionary design a plausible strategy again".
    I've been contemplating how the discipline of architecture, as well as the role of the architect, fits with the agile approach to developing systems whilst reviewing the systems development lifecycle (SDLC) for a number of clients of late. In my experience most people have an "either-or" approach when it comes to SDLC's. They either do waterfall or do agile and have some criteria which they use to decide between the two approaches on a project by project basis. Unfortunately these selection criteria are often biased toward the prejudices of the person writing the criteria and will push their favourite approach.

    Rather than treating agile and waterfall as mutually exclusive I would prefer to adopt a layered approach to defining SDLC's as shown here.

    The three layers have the following characteristics:
    1. Basic Process. Assume all projects adopt the simplest approach to delivery as possible (but no simpler). For most software product development projects this will amount to an agile approach like Scrum which uses iterations (sprints) and a simplified set of roles, namely: Product Manager, Scrum Master and Team where Team is made up of people adopting multiple roles (architect, programmer, tester etc). On such projects decide up front which artefacts you want to adopt. These don't need to be heavy-weight documents but could be contained in tools or as sketches. Here the team member performing the architect role needs to manage an "emergent architecture". The role of architect maybe shared or not dedicated to a single individual.
    2. Complex Process. At the next level of complexity where multiple-products need to be built as part of a single program of work and all these have dependencies some level of governance usually needs to be in place that ensures everything comes together at the right time and is of a consistent level of quality. At a micro-level this can be done using a scrum-of-scrums approach where a twice or thrice weekly scrum that brings together all the Scrum Masters happens. Here the architect role is cross-product as it needs to ensure all products fit together (including maybe third party products and packages). This may still be a shared role but is more likely to be a dedicated individual. This may involve some program level checkpoints that at least ensure all iterations have created a shippable product that is ready to integrate or deploy. The architecture is not just emergent any more but may also needs to be "envisioned" up-front so everyone understands where their product fits with others.
    3. Complex Integration Process. The final level of complexity is caused when not only multiple products are being developed but also existing systems need to be incorporated which have complex (or poorly understood) interfaces. Here the role of the architect is not only cross-product but cross-system as she has to deal with the complexity of multiple systems, some of which may need to be changed by people other than the core development team. Here the level of ceremony as well as the number of artefacts to control and manage the decisions around the complexity will increase. The architecture is certainly not emergent any more but also needs to be "envisioned" up-front so everyone understands where their product fits and also what interfaces they need to work with. This is something Scott Ambler suggests happens during "iteration zero". 
    Each of these layers is built on a common architectural as well as process "language" so everyone has a common understanding of terms used in the project.

    I'm much in agreement with Mike Cohn's comment on his blog recently where, in reflecting on the tenth anniversary of the Agile Manifesto he says: "The next change I’d like to see (and predict will occur) over the next ten years also occurred in the OO world: We stop talking about agile" and goes on to say "Rather than “agile software development” it is just “software development”—and of course it’s agile".

    I would like to see an agile or lean based approach as the de-facto standard that only adds additional artefacts or project checkpoints as needed rather than thinking that every time we need to either adopt an agile approach or a waterfall approach.


    1. I generally agree with what you have blogged above. There is no one-size-fit-all approach even within the context of one project in any of your 3 layers above and I would argue that for the "Complex Integration Process" layer, the architecture would need to be even more emergent because the more complex the problem is - countless modules/entities/nodes each connected to one another resulting in a whole greater than the sum of its parts - the more important it is we do not make any static architectural decisions that do not allow the architecture to emerge as time passes and complexity continues to grow.
      I just came across an interesting post, arguing in similar directions -

    2. Thanks for this. You are right, I don't think I'm advocating an either/or between emergent and envisioned but rather a combination of both as the problem becomes more complex.

    3. Great post Pete. May I relate this to organizational maturity as well? While the organization grows, the complexity of its its portfolio and landscape increases. However, stange but true as you point out - all three layers are still evident within the organization - due to the nature of evolution/growth of a company.

    4. Hi Peter,
      thanks for this great post! I also don't see that agile development should be mutual exclusive to any other approach (e.g. waterfall). I'm much convinced that the most important thing is to understand the values and principles of the Agile Manifesto (they are quite fee/clear) and hoping that this will be common sense for software development (as mentioned by Mike Cohn). In software development, as in other disciplines, we can only be really successful with wisdom. It means we have to do the RIGHT THINGS and software architecture is certainly something we should take care. Sometimes Evangelists concentrate their efforts in doing just the THINGS RIGHT, they are focused on the process (e.g. how meetings should be done, what is correct/allowed or not, etc...), this propagates the mutual exclusive thinking. I'm waiting to see you again in DB Frankfurt for another architecture training session!

    5. Subhav, Yes I see this as applying to organisational maturity. There is another corollary between process and organisational maturity too which is that as they both grow they tend to attract more resource which has the effect of making them more complex which means more resource is required to manage them and a viscous cycle soon ensues...

    6. Tiago, I love your comparison of doing the right things as opposed to doing things right. Sure you can solve a problem that is technically correct (i.e. it's doing the thing right) but are we truly doing the right thing in solving those problems. As you may know the SEI draws a distinction between "verification" which helps answer the question “am I building the system right?” and "validation" which answer the question “am I building the right system?” For me it is more important that the architect can answer the latter than the former.