Degraded performance of address validation and label purchase
Incident Report for Shippo
Postmortem

Summary

Between August 19, 2016 10:38 am and 11:09 am, our systems experienced higher response times. Address validation and some USPS labels requests timed out and returned 500 server errors.

Root Cause

The problem was caused by a temporary outage of the USPS Address Validation API, which is an external dependency of Shippo's address validation endpoint. The address validation and label generation APIs incorrectly handled the USPS API timeout and returned a 500 internal server error instead of a well-formatted error message.

Actions taken

  • We will correct the misreporting of timeouts as 500 server errors.
  • We will remove the address validation dependency for USPS label requests.

We sincerely apologize for the interruption in service caused by this outage. We know that your business relies on Shippo and that such incidents have direct consequences on your operations. It's our highest priority to offer a reliable and performant service and we're committed to preventing any such incidents in the future.

Posted Aug 19, 2016 - 17:12 PDT

Resolved
The system is fully operational. All label and rating requests are being processed successfully.
Posted Aug 19, 2016 - 12:06 PDT
Monitoring
All systems are fully operational again. Our engineering team is closely monitoring the system.
Posted Aug 19, 2016 - 11:20 PDT
Investigating
Address validation requests and some label purchase requests are currently timing out. Our engineering team is working on a resolution.
Posted Aug 19, 2016 - 11:02 PDT