Yesterday, April 15th 2015, we experienced a service issue with our SmartFlow tablet applications for both iOS and Android devices. Today, we are providing an incident report that explains the details of the event and the actions we have taken in the aftermath.
Beginning at approximately 4:50AM PDT, the SmartFlow application experienced time-outs which prevented users from accessing information on the tablet. At the root of the problem was a database issue caused by high erroneous activity, leading to symptoms which include: missing vehicles, inability to load or submit inspections, and incorrect worksheet information. This was discovered and corrected by 7AM PDT.
Corrective & Preventative Measures:
1. Fail Safe Mode (detailed article found here)
We know the importance of our services to your shop’s day to day workflow and do not take this lightly. To ensure no loss of data and the ability to create inspections with your tablet 100% of the time, we have developed a fail-safe method for performing inspections and have already begun the testing phase with a dozen shops. We will extend “Fail Safe Mode” to more shops at the start of next week and are currently scheduled to go live by the end of May.
2. Monitoring Improvements
In order to detect and prevent similar issues in the future, we have taken efforts to tighten our monitoring and recovery system. We are implementing additional monitors into every layer for our application and services. The new insights into the out-dated software and systems allow us to revisit previous designs, optimize performance, and improve the overall experience for our users.
3. Information Resources
In the event where an AutoVitals service is temporarily unavailable, we understand that you want to know everything about what’s going on, when to expect a solution, whether there is a workaround, and so forth. In an effort to improve our communication with you, we have introduced the system status page and will be heavily utilizing it in the future to keep you informed.
Once again, we apologize for the recent service issue and the impact and inconvenience on you and your shop. The trust you have placed in AutoVitals does not go unappreciated, and we are committed to maintaining that trust. We thank you for your patience and continued support.