A Practical Framework for Building Reliable Automation
Automation starts with identifying tasks that are repetitive, predictable, or prone to human error, and recognizing where those patterns exist in your day-to-day work. From there, you look at how the work is structured and choose the right tools based on the type of automation required, whether that’s data collection, system integration, or workflow automation.
Once the tools are selected, the process is built in clear, modular components, then tested to ensure it produces consistent and reliable results. Finally, the system is regularly audited and maintained to make sure it continues to function as environments, systems, and requirements evolve.
Identify What Can Be Automated
It’s time to take a look at your daily tasks and see what you can automate. Here are some criteria and questions that can help you decide which tasks are worth automating:
- Does the process follow clear, concise, steps and rules?
- Are the inputs and outputs predictable?
- Is human error common?
Repetitive tasks are those that have, usually, well-defined steps with predictable inputs and outputs.
An example of this would be generating sales reports from your CRM, and calculating sales totals from the past 24 hours. Each step on how to read the reports and perform all of the necessary calculations is well-documented and consistent. Another example would be taking daily backups. The steps to generate the backups are well-defined, and it generates consistent results each time.
Using the aforementioned daily backups task, here is an example of how automating the process can save time and provide more consistent results. Let’s say that you have 200 different systems that you need to backup daily. The entire backup workflow takes you 1-2 hours to complete. Most of the time coming from clicking through different applications, reviewing the next procedural step, and moving onto the next system.
If you were to automate that process you would no longer need to click or read the next steps as it would be built into the process. You could even feed a list of systems to the automated workflow so that new systems are added (or old systems removed) from the backups. So what would normally take 1-2 hours could end up taking minutes. This also improves backup consistency by removing human error from the task.
Error-prone tasks are those where the likelihood of mistakes increases due to human limitations rather than the process itself. These typically fall into three categories: fatigue, complexity, and repetition drift.
Lack of sleep or working late to complete tasks can lead to fatigue. Fatigue is a common cause of errors because attention decreases, increasing the likelihood of mistakes.
An example of this would be missing critical line items while parsing through system logs after an application crash. If this is a business-critical application, any extra downtime could be costing the company large sums. Having an automated process in place that detects these specific line items would aid in faster application recovery.
Sometimes a process can be long and complex, which can result in cognitive overload. This can make it very difficult to really focus on what you are doing and you may skim over parts or spend longer performing the tasks than originally necessary.
An example of this would be on-boarding new customers into your infrastructure. When you need to tie them into multiple different systems (storage, DRaaS, IaaS, etc.) all at once to deliver their services. Each component is necessary to the next and even one small piece missed can cascade to the other systems and cause the entire stack to fail. If these tasks were automated it would ensure that no part is missed in the on-boarding and all systems would come online successfully.
Even the best of us fall victim to repetition drift. Repeating the same task, ad nauseam, is enough to make our eyes gloss over.
An example of this would be copying data from multiple spreadsheets into a master sheet or database. With no real challenge or anything interesting our minds tend to wander, and the cells of a spreadsheet tend to meld together. Something as simple as pasting a value in a wrong cell can cause the entire sheet or database to lose its integrity. An automated task that would read data from these sheets and input them into the proper cells or database ensures that the integrity of our data is maintained.
Find the Right Tools
Once you have chosen a task to automate, the next step is to determine what tools are best suited for the task at hand. Not all automation tools are created equal. Some are better suited for web-based tasks (web scraping), some are used to connect applications , while others for automating workflows using AI. Choosing the correct tool is just as critical as the task you are looking to automate.
Here are several examples of different problems and the tools that could be used for them:
- Company ABC is looking to perform competitor pricing analysis to ensure that they are staying competitive in the current market. If this process was done manually, they would need to navigate to multiple sites, click through each page, and extract all of the pertinent information. Once obtained, they would need to store it somewhere that is easily accessible.
Python would be a solid option for this task as it has several libraries that are built specifically for these types of tasks:
- Web Scraping: Scrapy, Requests, BeautifulSoup
- Data Manipulation: Pandas
- Data Storage: Can load data into spreadsheets. Can also connect directly to databases
- Tech Corp is experiencing growing pains due to an influx of customers using their SaaS product. The time to on-board and integrate new customers into their system is a bottleneck. Customers are becoming frustrated as they aren’t getting their services delivered quickly enough.
n8n would be a great option for this. There are several different integrations designed to streamline this type of workflow:
- Webhooks: Trigger automated actions/events
- Project Tracking: Jira, Asana
- Databases: MongoDB, MySQL
- Email: Outlook, Gmail
This is not an exhaustive list of each category. It demonstrates that different tools are specialized for different workloads.
Build the Automation Process
After finding a task to automate and the correct tools for the task, it’s finally time to start building out the process. While this may seem like an overwhelming task its important to break the task down into small components. This will allow you to focus on a single element at a time, but it is also how you will create your process. You may already have a documented step-by-step process you can use.
One thing that will assist you in creating this system is to keep similar logic together. If you are building a web scraping application, you would want to keep all login and authentication logic coded together. You could have another section for scraping the web pages. Another section could be specifically for parsing that data and extracting the necessary components. If your system is no longer scraping data then you know to go to the section of code where you have the scraping portion coded and begin troubleshooting.
While proprietary restrictions prevent sharing source code, this case study walks through the workflow design, implementation approach, and operational improvements gained through automation.
Verify the Automation Actually Works
Now that the automation process has been built out, it’s time to start testing. You want to make sure that your process is as robust as possible. This will help maintain the integrity of the results. If you are automating daily backups, can the process see new machines that are added, or ones that need to be removed?
Here are some questions to ask yourself when you are testing out your system:
- If any invalid data is found, how does my process handle it?
- If I run it multiple times using the same criteria do I achieve the same outcome?
- If I hand this process off to another individual would they be able to run it without error?
- If I make a typo, what happens?
- If I enter text where there should be numbers, what happens?
What this does is to help you start thinking outside of just the standard step-by-step process. Automation only handles scenarios you explicitly account for so you need to build in certain steps for error handling, or any other variables that may be presented.
Audit and Improve the Process
Your automation process is completed, tested, and has been running for some time. It’s time to ensure that it’s still working as intended and adjust as needed. Maybe there are some components that are no longer needed that can be removed, or others that you would like to add in. Just like your car, regular tune-ups will keep your process running smoothly. Here are some examples of why we want to perform regular audits:
- Systems evolve
- APIs change
- Dependencies break over time
If your automation process is keeps backups or archives of certain items, it is crucial that those are tested as well. Something simple like opening the files and verifying data, or restoring a backup of a machine to ensure that it’s working as intended.
It’s recommended to keep these audits on a schedule. Any time the automation process is adjusted, an audit should be performed. There is no one size fits all but here are some example schedules.
- Weekly checks of the system output
- Monthly reviews of the automation process
- Quarterly audits of archival data
Conclusion
Automation isn’t just about replacing manual tasks. It’s about fully understanding the issue at hand, selecting the right tools, and thoughtfully building a solution. The framework outlined should give you a solid understanding of the process. This will ensure that you continue to build reliable automated systems in the future.
