Creating High Level Designs

High level design is a subject of much discussion in pretty much every organization I have worked with. The question “How Much Is Enough?” is often put to me. This blog addresses this question.

If you ever have to bring a High Level Design (HLD) to me for review – this blog is a good read because it sets a level of both design and review expectations. I promised a colleague I would publish this blog, so it was written in a bit of a rush and I may go back and update it later.

HLD Overview
Figure 1 shows the key areas in a HLD.

What is a High Level Design?

I normally break architecture down into 3 basic levels

  • Conceptual Architecture – Is a single view that sets the scope of the architecture at the highest level of abstraction
  • High Level Design – Is something in the middle. It gives us the answers to all the fundamental questions, whilst normally staying at a level of abstraction from actual technology implementations.
  • Low Level Design – Is architecture at its lowest level – which can include actual machine and network configurations, and exactly how things are implemented

Prerequisites

Its not practical to build designs without putting them into the context of the business. Architecture is about building a solution which solves the needs of our stakeholder. With that in mind there is some related documentation we should have.

A Business Case

The business case is basically setting the scene for what we expect from the architecture. If an architecture isn’t providing a benefit to our business we shouldn’t be doing it.

Bear in mind, this doesn’t necessarily mean that the benefit is profit. Whatever the benefit is, it needs to be stated, it needs to be clearly defined in a way that is measurable and not subject to opinion.

“We need to make a lot of profit” may be a true statement – but “we will make 1 million Euros in the first year” – is a lot more clear.

The business case is establishing the goals, and business requirements. It basically needs to lay out what it is thats being proposed, and the value we get from it.

Basic Business Case Validation

When assessing business cases I am normally looking to see also the reason behind the numbers. If a product states it will make a million Euros per year i want to see the rationale behind that number, and to see that such numbers are realistic. The most realistic cases are based on existing numbers.

If Microsoft reports an 64% growth in Azure sales, it doesn’t mean its partners see the same growth. Growth and profit are influenced by many factors including how fast you can get an idea from idea to realization, and how well your solution can be scaled.

For basic validation of business cases I normally think as TELOS related questions.

A good business case demonstrates the thinking behind the numbers. If it states we need to hit a revenue target then these goals are part of our architecture motivation.

The key takeway here:

If you don’t have a clearly defined business case, then you cannot know that any technical design or service design will be fit for purpose.

A Conceptual Architecture

Conceptual architectures can come in many forms – I am normally looking for a single view in ArchiMate that shows the concept of the business case architecture. I normally am focusing on scope.

A conceptual architecture for a product normally shows the product and the services within it (Like a product view), as well as showing the key application components, or technology layer components relating to the product, as well as the teams that support it, and other related products & dependencies. In product service architectures i am normally showing Business Application and Technology layers, but not showing data or information flows. This is because of how we use conceptual architectures as part of business validation – adding relationships and information flows makes it messy.

In a non product related conceptual we make sure the view we provide hooks into the current existing architectures (services or products normally.

With any architecture I am looking to ensure It has considered the architecture aspects.

Requirements

I recently wrote about Architecture Concerns and will not go into too much detail. For any architecture, having a defined set of requirements to validate is essential. This can take many forms (such as User Stories). This could also be done in a set of ArchiMate views.

Basic Requirements Validation

I want to see the following when I validate a HLD:

  • A set of requirements – which are clearly defined and unambiguous.
  • Documented evidence of requirements management – Showing how requirements have been managed, and pointing to where in the architecture requirements have been realised.
  • Considerations from all the stakeholders perspectives. Lets break that out separately.
  • Consideration of incoming and outgoing requirements – we need things from other teams – for those things we need agreements with respective teams. On the flip side we may also have requirements towards us. If our solution uses monitoring for example, the monitoring team may require a specific agent to be installed on something.
  • TELOS – As mentioned in the business case phase a more detailed look into TELOS is warranted as part of the high level design to ensure that our requirements consider Technical, Economic, Legal, Operational, and Scheduling needs.

Stakeholders

When we talk Stakeholders its very easy to be preoccupied with just the customer or the end user but in reality many teams or people either provide something to your architecture, or require something from it.

ISO 42010 has a good list of potential stakeholders of a system which includes users of the system, designers of the system, owners of the system – the list goes on. As a minimum need to know that an architecture will meet these needs:

  • Customer – this includes different roles within customer organization – for example the end user cares if they get a easy to use continuous service – where as the person that is buying the service from you wants to be able to show their management how well the service is used, or the value it brings.
  • Internal Business – around revenue and profit for example. or if its needed for strategic reasons we need to keep tract of that kind of things. Sales teams may also have concerns with architecture.
  • Internal Ops Team – They need to be able to operate the solution the architect gives them in an efficient way – there are normally Staffing, Training, Capacity, monitoring, and procedural related requirements for example.
  • Partner – If we are using partners a business interface needs to be established with them – a set of requirements together with an SLA is a fantastic way to do this.

Validating the Architecture Model

One thing to note is the old adage “If its not documented, it doesn’t exist”. Its important for all information around your architecture to be stored together. a High Level Design can be produced completely in an Archimate model, or maybe part of a document (I discuss this in Designing Architecture Through Document Templates.

Our High Level Design is telling a story. Effectively all those prerequites we have described are setting a challenge. Its the job of architecture to describe a solution that addresses all the needs of a challenge.

Regardless of if you chose to use a document or an ArchiMate model, or in Word, the story needs to be threaded together in a single solution. There may be multiple parts – such as several prerequisite documents, but they need to be referenced together in one place.

Within any solution or service design there are normally many meetings, and its essential we track decisions that are made in a single space that is available for all. Where i currently work i normally track all decisions relating to a specific architecture on a single page so its easy to follow. Its ok to reference external sources for things liike requiremetns or supporting information, but when this is done you have to consider the following

  • Ensure there is version control so we know what version of a particular document or source we are referring to on our design. normally when i am doing a review meeting i will take a copy of related documents and store them together so that there is a timeless copy that cannot be interfered with
  • You have to know your stakeholders and ensure they all have access. I have lost count of the times information has been locked up so i cannot see related information.

I also want to ensure as far as possible that i do not duplicate information – its better to refer to master information sources for things when required.

If we are creating an entire high level design in ArchiMate you have to remember to document the model. We are not using our architecture tools to produce diagrams, we are using them to produce views of architecture. Our tools enable us to document the architecture view, to explain what we are showing, as well as the elements and relationships. Remember to address the audience.

Covering Architecture Aspects

Remember just as with our conceptual an architecture has to address architecture aspects. This doesn’t mean i have to address them all in a single place in the architecture – but I do have to ensure that we consider all aspects and not only technology. For example you may address processes and roles in many different views.

A minimum set of views

For those of you that do not know where to start I present to you a minimum set of views to consider. We do not document or create views just for the sake of having them. There may be reasons why you wish to use other views, or deviate from this structure. Thats Ok.

Lets take a look at some of the views I would expect to see. in an archimate model:

We will walk through this bit by bit. An ABB is an Architecture Building Block. Normally I prefer to deal with architecture at a service level but there are times where thats not practical. In those cases related architecture may be together in a grouping box which is named something to explain what it is. For example I might have everything relating to application monitoring described (including its interfaces, triggers and processes) in a grouping box named “Application Monitoring”. You would use a grouping box where you might want to represent a group of elements that are cross layer. Some of our application monitoring processes may be business related or technology related for example. Where possible I prefer to use services or functions to ABB’s.

Conceptual Architecture or Layered Views

The order of things I am showing can happen in two different ways. where I work currently normally at the point architecture begins a business person has already got a conceptual idea in their head and want to start there. Our product management teams are used to working in a specific way – I normally have to start by asking what is it they are trying to do, and then when I understand what – I am wanting to understand why.

Conceptual architectures are normally a good starting point when you are in this situation, as are layered views. I normally take a service focused approach to building architecture. Take a look at my Services In Archimate blog – it shows some layered views.

I normally start with a conceptual view and then refine and define services from there. Every Service or function I have defined at the upper level needs further detailing:

  • It should appear in a requirements realisation view
  • It should appear in a Service realisation view
  • It should appear in a implementation and migration view.

I do not necessarily have to see all services defined in their own service realization views, although I often will. As a rule of thumb if i have more than 30 elements in a view i am asking myself if I need to split it into sub views. Again remembering each view tells a story it may make sense to group together several services in the same service realization view if several services are realized in the same way – or if theres another logical reason to do so.

Requirements and Rationale

Working with requirements realization views, my key element of focus is the requirement. Working with motivations views I like to ensure i have the drivers, goals, and requirements clearly defined. other elements just supporting the story of how a need (driver) leads to a set of requirements that will achieve a goal.

In my Planning Work With Archimate blog i talk a bit about creating motivation views for stakeholder concerns, and I also show a level of motivational modelling in my User Stories blog. Those mechansims are good for capturing different concerns from different stakeholders, but the important thing to note here is that we need to align the motivation model with the actual business case.

I already described the order i do things in normally because its the quickest way to deliver some value in my particular case. However, an excellent approach is to start with a motivational view, to create some requirements realization views from that where i figure exactly which services and processes I need to put in place to meet the requirements.

All requirements need to be shown to be managed within an architecture design. If we are working with ArchiMate we do not necessarily need to see every requirement on a view – its ok to group together requirements on the view to make it more manageable:

Figure 3 – Requirements realization

The above (Figure 3) is OK- if in the documentation for those requirements you link back to a source where those requirements are all managed together – so we can easily see the working status of each underlying requirement.

The important thing here is that requirements are mapped to their respective areas of architecture. Figure 3 showed a very high level view – I would expect in such a case we would also need other requirements realization views to show more detail because although figure 3 gives me a level of confidence that we are managing requirements, it doesn’t really tell me how.

As an example. Consider customer requirements/user stories for our Web Hosting product:

Figure 4 – Example User Stories

The point I am trying to make is different requirements may be realized in very different ways. There are different ways we could address each of the needs of our stakeholders, and we need to show in the high level design that we have considered them. If we have 100 requirements for an application by all means group them together, and put them in a single element, but only if it makes sense to do that, and you can show in other documentation how each requirement is handled separately.

Sometimes the choices we make do not appear to make sense. For example, if we had chosen to have a manual business process for billing rather than an automated application process. There might be a good reason – like not having resources to do the automation, or maybe the billing is only once a year and a 2 minute job in another system and doesn’t justify the expense. That’s a business decision. Such a decision quite often gets made in a meeting, maybe minuted and dropped into an email and forgotten about. It is vitally important that decisions are tracked in a work log and not just in a meeting; I have seen thousands of hours burned because of a decision not being tracked and then trying to track back with the relevant people how and what was decided. Sometimes people leave positions in an organization and sometimes you never find out why. Its wasted time that can be easily avoided with a little discipline.

Take a look at my blog on Requirements Realization Views if you are interested in going into more detail.

Business, Data, Application, Technology

Borrowing from TOGAF – Business, Data, Application, Technology, we need to ensure we describe these layers of information. We could do this in a detailed layer view – but I normally break things down into several views.

The Service Realization

Every Service has a service realization view (Several can be part of a single view). This may be small if a service is provided by a third party – because then you are only defining the business interface – but regardless we need a complete picture.

The service realization view is showing how the business layer connects into the application layer. In our web service hosting example, our business service might be realized by two different application components – Apache & Internet Information Server (IIS).The view shows how this service is provided.

The Business Process View

In my blog on Aligning Archimate to BPMN – I introduced the idea of creating process overviews, such as this one:

Process Overview
Figure 5 – A Process Overview

In a high level design I don’t expect to see fully defined processes. For the business processes I only want to understand:

  • What the process is
  • How the process connects to the rest of the architecture
  • What events trigger the process
  • What the possible outcomes of the process are
  • Which roles or people provide the process

Normally this can be done a lot quicker than actually fully defining a process.

The Technology Usage View

Within the service realization I have identified a number of different application layer items (Apache & IIS) which should exist within a technology usage view. The technology usage view is connecting the application layer to the technology layer. I might have a simple view which shows how IIS is served by a technology device – or the scenarios could be more complex – for example we may be using virtualization.

Application Usage or Cooperation Views

These views are useful for seeing how application elements connect together, and how they tie into application data elements and flow of information, which is a good baseline for work with GDPR for example. If you have data elements in the Information Structure Views its good to show them here if it makes sense to. When looking at HLD’s I normally like to see both of these views – however sometimes I find application cooperation and service realization views have a little bit of overlap when people model these views.

The Technology View

The technology views show how our technology layer fits together. Following our example if we have technology devices for Apache and IIS I would want to understand how they fit together and are connected via a network.

At this point I would remind people we are still looking to build high level designs. I am not normally going down to the point i have IP addresses. Quite often with services we offer them with a variety of different service levels, and dependent on the service level we may have different technology views. For example, our best service level might necessitate us to build a technology architecture that has complete redundancy and multi site. Our lowest service level may be a single server in a data center. Model the common scenarios.

Information Structure Views

These views show basic information structures/taxomomies relating to the architecture. We create structures that make sense in the context of what we are producing. For example, if we are implementing a new service in service now then we need to determine a CMDB structural template, or maybe even a service now structure. We can lay out the relationships between the data elements and can even document the relationships to explain why they exist.

For more complex structures we could of course use something like a UML class diagram – but most of the time the structures I use are simpler. Figure 1 showed a simple information structure in the business layer, but of course we can create information structure views in the Business, Application and technology layers, or structures to show how they are related. Take a look at figure 6

Figure 6 – An information Structure View

I chose to use a Grouping box rather than create individual compositions to each data object. Depending on how i was going to use this architecture model I might have done it the other way, but we should visualize in a way that makes sense to our stakeholders.

There is a balance to be had. In a high level design we do not want to go to too much detail – Figure 6 has value because in our fictional example it shows our audience the configurations that we will need to support and helps set scope. We can see clearly that we would need to define two virtual machine templates and a standard network configuration for implementing our basic Apache – and we might chose to reuse these elements. The view gives context to some data objects that would need to be defined in the implementation views.

For more complex structures we could of course use something like a UML class diagram – but most of the time the structures I use are simpler. Figure 1 showed a simple information structure in the business layer, but of course we can create information structure views in the Business, Application and technology layers, or structures to show how they are related.

Relating to information structure is information flow. Its important to document how information flows throughout your architecture. I talk about how you might approach this in my Modelling Information Flow In Archimate blog.

Project & Lifecycle

I talk about modelling projects in my blog Planning Work In Archimate. That blog covers the basic minimums I would expect to see in order to understand a project and its road map. It doesn’t cover more complex concepts such as using plateau’s to represent different states of architecture. I will save that for a future blog.

Using Vendor Or Other Reference Architectures

I am fine with the idea of reusing with pride; several vendors have standard reference architectures. Just because a vendor has a standard reference architecture or a standard set of best practices, it doesn’t mean our job is done.

We still have requirements that need to be realized and we still need to cover the architecture aspects when using a vendor architecture. We also probably need to establish a proper interface and set of requirements for the vendor.

Summing It Up…

Theres a lot of things that go into any architecture design and this is only one approach to creating one. The reality is, that depending on your requirements the amount of emphasis you put into different areas of architecture radically change – for example if we are using Amazon Web Services as a provider of our technology platform we might not need to fully define that platform – although it may still need a small amount of architecture – just for us to state we use the technology level services from Amazon, and maybe what the interfaces into those services are. There could also be related requirements realizations needed of course.

I’ve done my best in this blog to address a fairly complex subject in a relatively short blog. The approach in the blog isn’t perfect, but its one I have developed and used over time to meet the requirements placed upon me by various stakeholders.

Implementing a High Level Design, as described here goes a long way towards creating a solid design that mitigates risks and significantly reduces cost to business.