Article · 2025-09-11
Why does order matter in large operations?
Hans Åkerman
I was at a financial client with a number of database servers. They ran Linux and PostgreSQL. There were not many production servers – perhaps around sixteen clustered and non-clustered instances.
They were looked after by different administrators and had very specific requirements for handling financial information (for example clearing, warehousing, reporting, administration and regulation).
They had also grown organically over time. And the problem was that every server looked different.
There was no installation documentation, and no operations documentation either.
Nor did they use any way of managing (idempotent) changes in their systems. Everything was done by hand.
I made a table of some basic things that had been set up – everything from kernel tuning (sysctl) and paths and mounts for database data to what the backup scripts looked like and how the databases were parameterised. There you could see that, for example, backup scripts and paths differed.
After this mapping, work began on documenting how a system should be set up (an installation guide). That meant every new server at least had more order than what existed in the production environment at the time.
Why order? Why documentation?
If we have sixteen different values on a server, that is a lot to "remember". If there is just one value that is the same, it becomes a stronger and simpler symbol for communication.
Everything starts from living documentation, for example in a wiki, or better still in a configuration management tool (Puppet, Ansible, Terraform and so on). That also gives several people one place to consume and change information, instead of it being installed in disorder on the servers.
This may be a banal text – but clarity in larger operations can be forgotten when staff carry a lot of information in their heads and do not communicate. A clear picture of the configuration becomes a clearer language. (Imagine everyone having their own local, subjective name for the same meeting room – it would be hard to remember everyone's name for the room, and it is more information to persist, both in documentation and in human memory.)
Troubleshooting becomes easier, and onboarding new engineers faster.
So we want to bring the degree of entropy down and get greater cardinality. Less noise. A clearer signal. Few points for mechanical change and for distributing information.
Note
There are cases where a configuration tool can cause trouble in a larger information monoculture – where systems become more fragile. That is when you can change large parts of the infrastructure and something goes wrong: we have not tested the change first, and we have not rolled it out to subsets of production.
Translated from the Swedish original, "Varför är ordning viktigt i stora driftsystem?", 2025-09-11.