Apply deliberately different business logic in more than one place to mitigate change risk without increasing change cost or timescales.
Image by sipa from Pixabay
Key thoughts on this topic
Businesses are always adapting - launching new products, opening to new customer types, and opening new stores. These events spur the creation of new data which needs to be processed correctly or be excluded from downstream systems, in possibly dozens of places in your organisation. When this isn’t done, issues occur.
Conventional wisdom would have you drive towards controlling data selection logic for a system in one place, however, if done with intent, having this selective code in two places rather than one can mitigate these change risks while not increasing the cost of change.
The Problem
Businesses are complicated and messy. From the outside, we see shiny branding, plush buildings, and fancy websites. Inside, these same companies, particularly those that have been around for decades, we find a mishmash of systems in various stages of transformation, running on multiple operating systems and built on design principles that have changed considerably over time.
This should not be a surprise; after all, any company which existed before around the 1960s had processes and procedures built without even the concept of the modern computer, let alone the internet, websites, and mobile phones.
Organisations of all types need to keep pace with these changes in technology and those of customer expectations, the regulatory environment, and the wider world. On top of that, many companies have not grown entirely organically over time and are the product of company mergers and multiple re-organisations – in short, they are messy, they are organic, and their systems reflect this. These businesses are constantly shifting and reinventing themselves inside and out generating an astounding amount of change which is difficult to manage and brings with it risk. The risk that a new product or service is missed from a data load or, sometimes worse, that data is included in when a system isn’t ready to accept it. Either scenario can bring issues for the business and for customers.
Here are just a few examples to bring this to life.
A store is opened in a new territory and the income is included in the main feed on which tax calculations are based resulting in an overpayment of tax.
A new outsourcer for a service is signed up and the data is sent out to the existing partner resulting in a data breach.
A new product is launched, and the product isn’t included in the system feed for customer direct debits (or refunds) resulting in loss of income for the business or a public relations issue in the papers.
A new customer classification is introduced to deliver a higher level of service but because the data output to our servicing partner doesn’t include the new category, these customers receive no service rather than an improved one.
The examples above are all examples of where something new has been introduced and because of a failure to manage change effectively, between multiple departments and/or partners, all of whom handle data for the organisation in some way, an issue arises.
The issues above can be split into two categories that surface the underlying approaches used in the system design.
· Issues where new products need to be included but are not
· Issues where new products need to be excluded but were not
If either of these issues occurs, we have a problem. Which one is worse depends very much on the scenario.
To Include or to Exclude?
When we take data from system A and load it into system B, we rarely want to take everything – we almost always want a subset. For example:
Only produce annual statements for a particular customer category or product type.
Customers based in a different territory may have different tax treatments which are handled by a different system.
There may be regulatory boundaries meaning that certain data may not cross into parts of our organisation or be moved overseas.
There are two approaches to how we take this subset, each with implications for how new data is introduced.
By Inclusion (Selecting In)
Here, we choose the values that we want in the new system and pick data where it fits with what we want. In SQL this would look something like this:
SELECT
Name
, address
, postcode
FROM
Customer_addresses
WHERE
Country = “England”
By Exclusion (Selecting Out)
Here, instead of selecting the data that fits our model, we exclude data that doesn’t fit. In SQL this would look like this:
SELECT
Name
, address
, postcode
FROM
Customer_addresses
WHERE
Country <> “Scotland”
If we have data only for England and Scotland in our data set, then both these will deliver the same set of records. It should be clear however that they respond differently when we expand the services we offer to France.
When selecting by Inclusion, we still only get England. When selecting by Exclusion, we now get England and France in our data.
Whether the first or second case is better, will depend entirely on what systems A and B are and what the new data represents. It may also be the case that adding “France” to our data now is better if we include but later when we add “USA” to our data, it is better to exclude by default, so I make no specific recommendation here about the merits of one vs another.
There may be applications where one or other of these scenarios is always preferable and would be considered a safe default. This is an example of considering how to make a system fail-safe but is not the reason I bring these two different methods to your attention so for more on designing failsafe systems, read my article on that topic here.
Where to apply the logic
We have another choice to make – where should this logic be implemented? Do we extract all the records from the source system A into the destination system B and apply the logic there or do we restrict the data transfer? While data principles would recommend restricting at source to minimise data transfer, there are good reasons to have more data than is strictly necessary sent to system B so let’s explore this aspect briefly as we do have a choice.
Apply the logic as we Extract from system A
Smaller, and therefore more efficient data extracts and loads
Improved security - less data moving is generally a very good thing
If we want to expand the service provided by system B we need to speak to the owners of system A to coordinate our change meaning change can be more costly.
Apply the logic as we load it into system B
Larger data feeds and longer load times
Greater potential for data breaches as data that doesn’t need to be moved is extracted and moved or copied around.
Potentially easier to make changes to the provision of service in system B as we don’t need to coordinate changes with teams owning the source system.
The reality is, despite design principles and data governance, that over time, we likely end up with some logic in both places. Remember that business is messy and at some stage, a new product will have been introduced, we may have opened up the feed from A to include all products to allow system B the ability to include/exclude at will (thus reducing change cost in the future) and a decision to stop part of the service in System B at short notice may have resulted in an additional exclusion or two on the data load.
This two-system view is also a zoomed-in view of our organisations system landscape and the situation probably looks more like this…
When considering where this logic should sit, we would naturally try to set all the logic in one place or the other. This sounds instinctively correct as it provides the ability to audit the code simply as we don’t need to consider how two pieces of code interact, and it makes for a simpler understanding of who we need to speak to in order to make any change.
This is where my thoughts diverge from convention and I encourage building the logic on both places with a very clear intent and approach.
The Gateway
Let’s take a step back and revisit the requirement with a view to the risk at hand and how we control for it, by design. Our design should (assuming no prior knowledge of new data values):
Allow us to readily identify that new values exist
Not allow them to be processed in system B until we have had a chance to evaluate how they should be catered for
Minimise the cost of change (ideally allow us to make a change in one place only) when a new product or service is introduced
Keep data extracts and loads small to keep data safe and fit with data security principles
Here is the proposed gateway design:
System A:
Extract the data, explicitly excluding values you know you don’t want in system B
Use Exclusion type rules ONLY
This minimises the data extract and removes unwanted data
System B:
Load the entire contents of the data transfer from system A into a loading area
Pull the data from that loading area you specifically want out of the loading area and into your system for processing using inclusion rules ONLY.
By doing this, we create a two-gate system. When the two halves of the gate are correctly synchronised, they do the same job. If we make no change to this set-up and introduce new data with unhandled values, this new data gets past the first gate as it is not being explicitly excluded and is blocked by the second as it is not explicitly included.
Any data with new/unhandled values then do not get processed in our destination system and are clearly evidenced by their existence in our loading area – our ‘new items trap’.
The system is then completed with simple alerting control over the new items trap to notify us that something remains in here.
We now look at the contents of the new items trap, discuss with the appropriate business areas what the correct treatment should be, and make a change – just one change.
If the decision is to include them in our destination system processes, then the logic in system A, which by default does not exclude them is already correct and we change the inclusion logic in system B.
If the decision is to exclude them, then our system B logic which doesn’t look to include these values is already correct and we need only change the logic in system A to exclude them from the feed.
Once the two logics are back in sync, we can empty our trap – either by pulling the contents into system B or by discarding them as no longer required and our control is reset.
Expanding the concept.
Implementation tip
This kind of setup takes some thinking and careful implementation work. It’s easy to talk about this using the example given where we’re selecting from a single table and a single value such as country. Writing and applying SQL logic to only exclude or only include is not always that easy and as such it may be that this approach is taken on those key data attributes where change is reasonably expected to happen rather than on everything in the relevant extract logic.
Protect against implementation failures
When systems are changed, sometimes logic can be bypassed or wiped out in a failed implementation. With this set-up, if either the logic in extracting the data or the logic in loading the data fail completely and are bypassed, no erroneous data will be loaded and processed in system B as the two sets of logic duplicate each other in functionality providing an additional set of safety and providing time for the changes to be rolled back and saving major headaches in trying to clean-up data that should not be there.
Provide dual-key/two-to-sign changes around critical data exports
Transferring data outside of your organisation? You could use the principles here to have both sides of this gate in the data extract mechanism. There is no need for them to necessarily be in different physical systems. You could implement the initial logic to extract the data using the exclusion approach into a pool within system A, then produce the output file to be sent to the third party using the inclusion rules. You could extend the control around this extract further by ensuring that each side of this logic be ‘owned’ by a different change approver or team to bring in the dual-key approach so neither one could allow new data out (and either party could prevent data output).
First Published 20/08/2021
All views expressed in this article are solely those of the author
© Alex Cowhig 2021
Comments