Trucker Path app outage yesterday

615914

{"text":"Tuesday noon approximately 9:00am – 2:30pm PST, Trucker Path’s app experienced a widespread database outage. This resulted in systematic failure across login, search, status updates, and other services fetching from the database. At approximately 2:30pm PST we resolved the issue. This was an extremely severe issue. We want to apologize to all our users, provide more technical detail on why it happened, as well as share the lessons we learned. \n\nBackground\n\nThe key problem that caused this outage stemmed from a database issue. After accumulating months of faulty data entries, the database and CPU was overloaded. Subsequent services that fetched info from this database completely failed, resulting in many users unable to login/search/see updates. Like a warehouse that gets completely clogged with junk, its loading docks will be unable to ship/receive any loads. After several hours of troubleshooting, our backend engineers were able to identify and isolate the faulty data from the main database. \n\nResponse\n\nWe started receiving feedback from users within minutes of the failure, however, the resolution took much longer than anticipated. The central database houses several different and critical functions; thus it was difficult to pinpoint the root cause. Only after a few hours did our senior engineering teams diagnose and resolve the issue.\n\nNext Steps\n\nWe know for every passing second, thousands of our users are impacted with real world consequences. We’re going to improve our systems to detect anomalous configurations. Our escalation process needed improvement, and we’ve added internal control mechanisms for critical issues to directly ring our senior management teams. We also need to work on our communication with our users – we cannot guarantee mistakes won’t happen, but we can guarantee we will give you guys updates within 30min of any outage. \n\nConclusion\n\nMillions of truckers rely on Trucker Path’s app to keep goods moving and our economy running. Our users expect Trucker Path to always work, and we take that expectation seriously. This was a painful lesson for us, but it motivates us to continuously enhance our service.\n\nWe’re sorry.\n","videos":"[]","link":"{}","pics":"[]","canComment":true,"externalShare":false}

988175

{"replyTo":"","text":"reboot your servers daily...the end.","replyToPid":0}

229979

{"replyTo":"

988175 发表于 2020-07-22 10:31:04 reboot your servers daily...the end.

","text":"Your a complete computer illiterate.  Rebooting won’t do hack for a database error.  ","replyToPid":389966}