Better Living Through Data Process Automation

Hidden From View, but Not Out of Sight

Flip a switch, electricity flows, and a light comes on overhead. Almost all of the modern conveniences that we enjoy rely on electricity to make them functional, and the reliability and near-instantaneous responses our electric appliances provide make it easy to take for granted – until the power goes out, of course. Then we’re left trying to figure out how to get work done, how to make dinner, how to wash the clothes, and more – and doing it all in the dark!

And of course, we can’t forget that the immense power of electricity also comes with a substantial risk: It’s dangerous to work with, and a short-circuit could result in property damage that takes a long time to fix.

Data process automation is the equivalent in today’s data systems. When properly set up, automation can hum along so easily and reliably that we can forget it’s there. Powering data movement, quality checking, model testing, outbound marketing, and more, data process automation can reduce the cost and complexity of performing some critical everyday tasks. But when it breaks down (or if it was never set up properly in the first place), it can wreak havoc: critical data sets deleted or overwritten; marketing messaging pushed to the wrong audiences or at the wrong times; infinite loops that consume processing time or threaten your API usage limits.

Just as washing the dishes or making dinner can be done without electricity, all these tasks can be done without automation – it’s just a lot harder, and your risk of making a mistake is greater.

So how can organizations minimize the risk of issues with their data process automation?

1. Clearly define the goals of an automated process. Be specific and narrow. “Retrieve the data from a provider, load it into a local database, and check for duplicates” represents three goals, not one. The three goals may integrate closely with one another, but they are separate goals and very likely will require different automation approaches. Treating each one individually allows your team to build your automated processes up brick by brick.
2. Be creative about imagining failure states. Processes can fail in myriad ways, but as humans, we tend to be overly optimistic about things – especially processes we’re designing or relying on. Spend time brainstorming the different ways things could go wrong. Write every idea down! It’s okay to prioritize based on the likelihood of it happening or the resulting risk of ruin, but “squirrel chews through power line” is a perfectly acceptable failure state to bring up. Being realistic about how things can fail will help you to build robust automation.
3. Develop contingency plans for when things go wrong. Did an API call fail? How long do you want to wait to try again, and what happens if it still fails? Should someone be alerted? If so, then how? Via email, text, or phone? How long should they have to respond before the situation is escalated further? How should downstream processes compensate? Should they all pause, or is moving forward without the output of this automation process okay? Having these answers gives you a sense of how your system can respond to and recover from an issue.
4. Remember that lightning can strike twice. If a failure occurs that you weren’t prepared for, move it to the top of your list for developing a contingency for – because it’s likely to happen again. It’s easy to write off unusual events as flukes, but really, there are almost always symptoms of some systemic danger you haven’t yet fully accounted for. Don’t let yourself get bitten twice!
5. Maintain good relationships with third parties. Many of your automation tasks are going to involve moving data to or from third parties. Know who your contacts are and how to get ahold of them when you need to. Touch base with them occasionally to make sure that they’re still the right individuals, and update your records if you need to. Familiarize them with your use cases so that they’re fully briefed and ready to help when the situation arises.
6. Test, test, test, and back up! Test your automation processes anytime you change them, and make sure you include tests of their impacts on downstream processes as well. Wherever possible, include stress tests and edge-case tests so that you’re not just testing the optimal situation. And always make sure that any data that your automation systems come anywhere near are regularly backed up (preferably in an automated fashion)! You don’t want the first time your contingency plans are tested to be when you’re in the middle of a crisis.

Just like with the electricity in our homes, data process automation enables us to save a tremendous amount of time doing routine tasks. As long as we recognize the risks and dangers involved, prepare properly for any changes, and keep an eye out for situations that seem out of the ordinary, it will continue to be an integral part of any organization’s data success.