An architecture can emerge.

Recently I had the joy of stumbling over an interesting discussion regarding architectural emergence. It started with a blog post by Igor. A nice and intriguing read that's basically saying that, although every system has an architecture and therefore emerging architectures are possible, they will be of limited complexity due to lack of evolutionary fitness. I like the idea of evolution and that Igor admits emergent architectures are possible. The original post lists a main key constraint to the quality of emergent architectures, though: „Developers do not anticipate upcoming requirements“. Furthermore, it is stated that “natural selection doesn’t think at all”.

Coming from this “reset to zero” (with local changes) idea the author argues that emergent architectures tend to be in higher technical debt than designed ones. Technical debt is not inherently a problem but can influence the system adaptability negatively in a way that it limits possible complexity.

I tend to disagree.

Firstly, I believe there is no lack of vision within development teams. Usually developers know what kind of product (from a high level perspective) is expected as a “goal”.

Secondly, I do not think that evolution is purely random, this argument would leave out evolutionary processes such as co-operative behavior and parent–offspring communication.  In fact, the fittest species are the ones who communicate most efficiently (Hi internet!). Hence I think that, though technical debt might be temporarily higher, communication will usually lead to quick changes. Nowadays we have the possibility to change technology layers or tiers by the means of refactoring and abstraction. In the EJB example of the original post, I don’t see why an upfront design decision to use EJB 2.1 with a change request to EJB 3 would be different from the team realizing that EJB 3 is the way to go and incrementally changing implementation (e.g. by assigning a team to refactoring). The more changes, the more YAGNI.

To some extent, these issues have been addressed by Alex. He is adding Dual Inheritance (Alex knows the terminology much better so forgive me any formal mistakes, please) theory to the formula, adding a bit of pessimism towards the outcome of the mentioned communication. In real life, debt would most likely not be detected and eliminated; it would be covered and not even sanctioned. This is a pretty good point, yet for some reason Alex limits it to myopic agile processes, which weakens it a bit.

Nevertheless, I still disagree.

Agile itself does not mean iterations are industrial or in any way myopic. The mentioned bias towards success is actually a bias towards “not failing”. Innovation can only take place if graceful failure is embraced and considered success, whereas procrastination is considered true failure. In Agile development, not failing can mean multiple things. In most of the cases, a developer (being an engineer) will realize that a technology might lead to not reaching the Sprint or Product goal and suggest adjustments accordingly. In my experience it’s not the developers being risk-averse but the management who would not accept not meeting milestones even if this would mean a better end product would be shipped.

Agile requires feedback: Between the users, product management, operations (read DevOps), project leads and the development team. This is also discussed by Alex in his following post. I very much agree with Alex that lack of feedback is one of the key reasons projects fail. I do, however, disagree that direct feedback stays myopic and does not change the ecosystem or culture around it (which somehow implies that no feedback is no problem). To stay with the metaphor of Evolution, this would ignore changes during the development of an organism, during childhood or via environmental changes.

My arguably optimistic assumption is that every developer wants to have success and deliver a good product. Given that he or she receives sufficient feedback, the developer will strive towards a “good architecture”. Because they want good feedback, interesting projects, reference letters, svn praise and all that in a flat world. In the democratic process which is happening between the developers, and between the teams (it’s good practice to have multiple teams, rather than having “one development team/stream for each system”), questions will arise and possible problems addressed. If you have, however, one architect or project manager designing and making decisions, he or she might indeed act as Alex pointed out: not acceptingchange eventually leading the project to grief, frustration and death. In the worst case, by turning this into a hindsight bias “problem of the developers” and trying an even tougher policy next time. 

In building architecture, an architect is someone who has an overall idea, mainly communicating, motivating and facilitating process change with the end goal in mind. No architect in the real world will not consider experts and listen to them every single day unless the keys are handed over. The beauty of evolution is that it has found to many ways to overcome its own randomness. As adaptability is actual fitness, evolution has found ways to incorporate feedback probably better than we humans do in everday life. A good architect is a feedback-broker who fosters evolutionary processes. Because he or she knows one day the actual world (as in society change, earthquakes or subprime crisis) will hit him hard.

I believe that software architects are still necessary. Instead of having "master architect" attitude, they should have a very similar role to building architects. Sticking to a major plan in one person’s head with the argument that emergence will lead to worse results is the dark side of choice management; it’s giving up adaptability in favor of predictability of failure. Accepting emergence, though, is the flipside. Channeling ideas coming from emergence into architectural decisions, embracing change coming from democracy within the development team and keeping all options open will eventually lead to even higher adaptability, more complex systems and – just by the way – a better life as evolved human beings.