Scoring Documentation Completion

This blog demonstrates a quick and easy way to score documentation completeness and improving quality which is more accurate than guesstimating,

The Goal

To be able to with a reasonable degree of accuracy estimate how complete documentation is, and to not need to take more that 30 minutes per document to do that. I need a balance between speed and detail.

Its not always the job of an EA to actually do these assessments but as an EA its good to get an understanding of documentation that’s based on an assessment rather than a guess at how complete something is, so I often teach this approach.

Some of the benefits of this approach

In following a structured approach we get

  • Better understanding of expectations. When the document authors understand the things we want to see the chances of getting them increases
  • Easy to visualize status – because we are quantifiable we can easily report status in a number of ways visually.
  • Better trace ability of progress – because again, being quantifiable means we can show improvement over time.
  • More realistic assessment of completion. Normally when i ask someone how complete something is they say 70 or 80% but when you start to look its often not so – people want to naturally please other people and tend to often give higher estimates, or sometimes do not think of the detail involved. This mechanism forces people to look at the work in more detail than if you were to “guesstimate”
  • Raises other quality issues. In doing an assessment like below you are reading and aligning to a set of criteria, which are the things that are important to you. Normally when If run a quick assessment i will come up with half dozen or so comments along side the review.

The End Result

I am looking to get an overall percentage of completion – performing an assessment of something i normally end up with a table which will look a little like this:

Figure 1 – A resultant score of several architectures.

Bear in mind the table is only an example. If I was creating scoring for high level design I may have more detail. You can see across the top I have listed some different architecture documentation. In this example its possible that a project may be composed of several teams working to deliver different things to a customer.

Going down on the left is the breakdown of the things we want to see in the document (Lets call them documentation areas). Each area is given a score of 0 to 5 based on some simple criteria which I will show in a while.

Another important thing to note is this is an estimation of completeness as a percentage and not necessarily work. Some of the things I give points for take longer than others. Reading through an assessment can help estimate the remaining work though.

An Overview Of The Process

The process i use is normally fairly simple. I normally pick a few people that will be assessors and give them an hour of training time with me. We run through a live document together and I explain what each document area means – and this description is normally supported with a template. So for each document we are assessing we normally have one template filled out – and we may go back several times over the course of a project to run the assessments again.

The assessment will happen between the assessor and the document owner. They run through each criteria and agree scores together and make notes to agree improvement areas. It can be done without the document author but i find that the communication between an author and and assessor in a meeting is more personal than just sending a filled in assessment. It also gets better commitment for improvement and enables the assessor to explain each criteria and answer questions.This can also be done as a peer to peer exercise.

I make sure that all people referred to in an assessment (approvers, technical validators) have actually reviewed the work and are aware of its state throughout the process.

Creating An Assessment Template

Its really easy to create a template that can be used for assessing the documentation. It can be done in any format – as a confluence page, as a word document, or as a google form, or Microsoft form for even easier collation.

Minimum Header Information

We always need to know as a minimum:

  • What is assessed – the document name normally – with a version number. I normally keep a copy of the actual document assessed with an assessment, rather than a URL to the current version of a document.
  • Who the assessor was – Just the name of the person who is running the assessment.
  • The document / design author – The person that is actually doing documentation work.
  • When the review was performed – just a date
  • A link back to any previous versions of the same assessment – If you are automating this process for a project in google forms for example the link back to a previous version might not be explicitly stated – you could calculate it if you know what was assessed and when. so if we did three separate reviews of the same thing it would have three different rows of data in your google forms.

The Criteria information

Normally I will expect to see this information each one of the different criteria

  • The criteria – The criteria we are assessing.
  • A short description of the criteria – normally a paragraph just to clarify things
  • The agreed score – using the scoring mechanism below
  • The scored mnemonics – I explain what each letter means in the scoring section.
  • The name of the person that approves the work – has to be a person with approval authority – not normally a project manager. Projects are transitory and end. The architecture normally gets delivered to someone in operations, and they should approve it.
  • The name of the person that technically validated things – normally a technical person who is not the author – its good to get a technically competent person from another team if possible.
  • Notes – any helpful information captured during a review.

In a very large project with multiple architects involved in an assessment I might also add the need for the Architect name, and the individual date You can see part of an example in figure 2:

Figure 2 – A partial example of a template

Scoring Documentation Areas

The Score

I apply a very simple scoring system; the order I show here is the order I assess on. I don’t normally assess something is fully complete until its partially complete. I don’t assess language before something is fully complete, and so on.

You don’t allow people to be their own technical validator or approver.

Figure 3 – The scoring

Short here is a mnemonic letter I use to show easily in assessments what I gave points for. So if something is at least partially addressed a point is given – an extra point is given once its fully addressed.

The Math

The math for calculating percentages in figure 1 is simple, but i will just mention it

  • There are 9 different things to assess and a maximum score of 5 points for each area – giving 45 points total as a possible score.
  • Row 14 is just a sum of the proceeding rows.
  • Row 15 is the percentage calculation – so we are calculating what percentage of 45 is in row 14. For column B that would be =(100*B14)/45

Summing It Up

This has been a quick introduction to how I approach doing quick assessments of documentation status when in large complex projects. Its not perfect but it is fairly flexible and quick and easy to implement. Its saved me many hours of having to do more in depth reviews before teams are ready to do so.

I hope this helps someone out there!

Documenting An Architecture Model

There are many considerations to make when documenting an architecture model. This blog speaks of some things to think about when documenting elements and relationships.

When I am speaking about documentation I am primarily talking about the documentation fields that are available with different modelling tools for documenting elements and relationships.

On Monday I posted about designing architecture through document templates. Keeping on a theme of documenting things, I wanted to talk about some of the things to consider when documenting within a model. Although I primarily talk about Archimate here, similar considerations must be taken when documenting any kind of relational model.

Elements & Relationships

ISO 42010 talks a lot about elements and relationships, and as I have said before, Archimate extends the concept.  The image above shows two business objects connected by a relationship.  The shapes in ArchiMate are elements – the connecting lines or arrows between them are relationships. 

We have said before that our modelling tools are creating architecture models and not pictures and that is important to understand when it comes to documenting elements. Each element is a building block that may be in many different  views which means when we document an element the focus of the documentation is the element itself and not its connection to other objects. Consider the diagram below:

Part of a very simple Service Realization View

We have an application component MySQL. Within this view we could document the element by saying:

MySQL provides back end services for our Apache application.”

In a word document that would be perfectly fine. In a model this would be bad practice for a few reasons – I have some points to make here:

  • Firstly, If we were to use the MySQL element in another view, such as a realization view for a PHP application service, although the statement above would still be true in the first view it would not be contextually relevant to the PHP realization view. 
  • Secondly – We have already expressed the documentation in a relationship on the view – the arrow between MySQL and Apache – It says MySQL serves Apache.  This relationship documentation would actually be a better place to put our documentation text although, as we said, its not needed.
  • Thirdly – If we find it relevant, to telling the story of the entire picture, we can actually place the documentation as part of the view – although it would be better for the view documentation to provide an overview of what the whole picture is about for the audience, rather than focusing on the minutia that is in each and every documentation element. We can also document each individual folder in a model, as well as applying documentation to the model package itself.

Deciding when you need more elements.

If you consider the example with MySQL again, we could provide a description of what MySQL is such as:

MySQL Server is a relational database management system developed by MySQL AB. As a database server, it is a software product with the primary function of storing and retrieving data as requested by other software applications.

…this is pretty good text because it explains what the element is and it could be reused across multiple views – however there’s another thing to consider. The relationship between MySQL AB and MySQL.  When writing something like that I ask myself is there any value in separating the vendor from the application component. I would say yes, its handy from a vendor management perspective – so on the view I showed above you can clearly see the vendor, and it only takes a few seconds for me to generate a view which would show all the applications that Microsoft are related to.

I would probably allow the definition above to be used in many cases – because even though it talks of the relationship between MySQL and MySQL AB, I think its OK because that relationship between product and vendor is very unlikely to change. If we were talking about a relationship that was more dynamic in nature I would probably redraft the definition.

How much should we document?

I would consider documenting all elements, and most relationships when fully modelling something, as well as documenting the view as well, to tell the story of the view and a little of the views intentions to its stakeholders.  The thing to remember here is not everyone knows what you do and what is obvious to you may not always be obvious to all readers of your view, or why relationships exist.  A colleague of mine who works with architecture in the defense industry was telling me he asks that all elements and relationships are documented as a rule of thumb for exactly that reason.

Its important that we get our balance right – we do not document for the sake of it but the view is providing the audience a story and all the elements and relationships within it form part of that story (I talk about this a little in my blog Improving Archimate Model Quality). Because Archimate provides some correspondence itself I will normally read out all the relationships on a model to myself. For example – “Owen is assigned to the role of Enterprise Architect”. I then ask myself does that say enough. Do I need to tell the audience why? If the answer is no, then I probably will not put any documentation in that relationship – because the relationship type (assignment) says enough by itself.

Asking For Help

I like to peer review views that I create, and get another architect to tell me the story of what my view says – a fully documented view I will also ask questions on just to see if its intuitive enough to enable the architect to answer.

Within my architecture domain I encourage all my architects to spar this with me, and in some cases i will give a view to a non architect too, just to see if they can figure things out.

Effective communication is key – if you are finding it hard to find the balance, sharing in a peer review can be a good way forward.

Designing Architecture through Document Templates

Document templates have some fantastic benefits – but I tend to avoid relying on them heavily when documenting architecture. This blog discusses why.

A question I often get asked – Can you show me a good High Level Design (HLD), or where can I find a good template for a high level design? The reason is normally that those resources want a starting point for doing their own architecture work; and in some cases its because those architects do not have so much experience and want a helping hand. 

Although I speak of Word or Excel, these are just the tools I use at work – we could easily be talking about Open Office applications for example, and facing the same challenges.

Using Word & Excel Documents In Architecture.

There are some definite issues in using Word and Excel for the creation of architecture – and I will run through some of these here, and ways to address them; Although I tend to use models to document rather than documents.

Change Control & Collaboration

Architecture changes by its very nature – as do the inputs to it – such as requirements. Maintaining such a inputs in Word documents is simply not practical when the stored in these formats. The biggest danger here is when you send files via email, multiple versions of the same thing suddenly appear. It is far better to store and version things on a collaborative platform such as Microsoft Teams or SharePoint.

Multiple Contributors

It’s true we have version tracking, and the ability to comment in tools such as Word. When building architectures we do not only manage change at the document level. Individual requirements, for example, need to be under change control. Its better to create requirements on a collaborative platform such as Jira, so we can see when they are created or changed, by who, and we can also lock down releases of requirements so we can be sure everyone is working efficiently together. You could of course refer to Jira and specific releases within your Word Document, and manually control release management, or control it by using some features in SharePoint for example.

Information is hard to manipulate and process

If I have 5 solutions and as an architect I want to understand the common requirements for those solutions, maybe as a part of an initiative to improve offerings – I cannot easily do it if each solution is a document. To do it I would need to find the right files, and I would need to read each one individually. This is not optimal and could take a few hours as opposed to some of the alternative ways of managing this – in an architecture model – we can literally extract such information in a matter  of minutes, because we have a fully relational model.. A collaborative platform such as confluence offers a compromise – it would still require the information to be collated and read manually but at least having things on a common platform makes searching and indexing easier. Its far easier to use an architecture model with documented elements.

Information is point optimized and not often updated.

With Word Documentation the document author should be maintaining documentation that’s used – but it doesn’t often happen. The problem is if someone makes a decision, they tend to document that in the documents they own or know, and not in others. If a product manager changes position, strictly speaking all our Business Continuity Plans, GDPR documentation and other deliverable’s should change with it (if the resource is named by name). I have lost count of the times I have seen failure in this. Business change has to be reflected in documentation.

This isn’t efficient but its the price of using such static manually created information sources such as Word or Open Office documents. Collaborative platforms such as confluence help here – because the work of correcting the shared knowledge base is also shared. A static document can refer to a dynamic system. You need to be careful though because if you do this its very easy to get to a position that is misrepresenting the truth.

If i had a document called “Project Pegasus 2018 End of Year Status” which links to a jira project where statuses of tasks are tracked i can very easily accidentally link to the current task status for 2019.

Word is good when you want to take a snapshot of the current state of affairs, or where you want to share that information with a third party.  For that reason tools such as BiZZdesign’s Enterprise Studio actually have inbuilt mechanisms for publishing documents and the distribution of architecture.

A fully relational model such as a shared repository is much easier to control than when you store information in a document. If you change an element in one place, it automatically changes in others.

So What About High and Low Level Design Document Templates

There are a few reasons I shy away from word templates in architecture design.  This is from ISO42010 – Lets look at it:

ISO 42010 Overview

Stakeholders (The people that get value from architecture – have concerns (issues but also requirements in this context).  Concerns are address by having a view of architecture which is governed by a viewpoint. When we are dealing with complicated architectures we should define solutions that meets the concerns of our stakeholders.

The standard also states that concerns need to be addressed in one or more viewpoints.

Put simply, an architecture solution should be making sure that everyone including the customer, operations, project services, and all the other stakeholders have a solution that meets their needs. Needs change – and depending on those needs, our point of view of the architecture needs to change too because needs of our stakeholders determine the things that need to be emphasized. The architecture design and the structure of work can be very different depending on the story we need to tell – even though in some cases many of the stakeholder concerns are the same.

Learning From Document Templates

In my experience learning by template examples doesn’t work. You can see results of the architecture process but not the architecture process itself. As an example; A template may have a section on process risks. You could choose from your experience to fill in that section – and you may capture a few important things there.  But that set of risks will be much less complete than if you ran through a BPMN risk analysis process. You cannot teach an architect architecture by document examples alone. Stakeholder concerns influence the approach and structure a document may need, and the method is invisible.

Different stakeholders and concerns may lead to radically different documents layouts. In architecture we don’t blindly follow pre-existing structure. We create our own optimal structures.

Copying A Template Example To Fulfill Customer Needs At The End Of A Project

Every time I get asked to do this, I die a little inside. You should do Architecture before, during and after a project . Your Stakeholder needs change or are refined. You need those changes to be included in the architecture. The architecture provides both details of the journey, and decisions made.

Filling out Word templates at the end of a project just to tick a box has very little value. Some people deliver documents then forget about them. Document based architecture needs to be part of a process to provide value. There would be significantly more value if we modeled the architecture design. We can of course use views from a model in our word documents. This is a typical way to communicate with external stakeholders.

The frustration of mindless documentation

Personally, When I am doing design documentation for someone else, Its normally a frustrating exercise. It involves interviewing people, and trying to find answers to questions that have been long forgotten. People make decisions for good reason. If they are not tracked finding reasons for them is hard work.It is not efficient to take this approach . When documentation is after the fact it may check a box for the project. Because its a document rather than part of a model that information may just disappear. If you are building a document template for a solution build in a section to keep track of decisions.

Summing it up…

Templates are good when you need to capture information in a standard way, but that is not the objective of architecture. Architecture is about building new structures, not adhering to existing ones. We have a myriad of processes, frameworks, languages and tools to support that process. 

The reality is that Word remains a good mechanism for outputting architecture to stakeholders, but in a world that changes and expects more lean agile ways of working, we need to use more agile tools to support this.

For documenting architecture methodology, technologies such as word are not optimal, but can work if you have strict process and discipline. You also need to be aware this will cause a major overhead and complexity when trying to optimize architectures later.

Using architecture models, and keeping documentation within them can work a lot better – but a good compromise is to build a model as your primary source of information and then output to work for specific stakeholders as needed.