Connectivity issues in Asia/Pacific region
Incident Report for ftrack
Postmortem

Connectivity issues in Asia/Pacific region

Summary

Users hosted in the Asia region experienced downtime with their ftrack service. The outage resulted in the service being completely inaccessible for customers hosted in the Asia-Pacific region. The outage affected the service for three and a half hours.

Cause

The outage was caused by a system failure in a service managed by the cloud provider (Google). An automatic repair was initiated but failed to repair the service.

Resolution

ftrack was able to work around the issue and made the service operational again. The root cause was later resolved by the cloud provider.

Post-mortem

Google cloud platform has committed to conduct an internal investigation of this issue and make appropriate improvements to its systems to help prevent or minimize future recurrence.

ftrack will also investigate and take action to minimize the risk of cloud provider outages in the future. We will also look over our internal processes and policies to ensure information about an outage is available to our customers on our status page (status.ftrack.com) as soon as possible during an outage.

Outage history

All times are in CEST.

2019-05-08 01:59 AM - Internal service failure detected.

2019-05-08 02:14 AM - ftrack service outage started.

2019-05-08 05:45 AM - ftrack service available again.

2019-05-08 10:19 AM - The root issue was resolved by the cloud provider (Google).

Posted May 08, 2019 - 17:46 CEST

Resolved
This incident has been resolved.
Posted May 08, 2019 - 13:02 CEST
Monitoring
Instances are accessible again and we continue to monitor the situation.
Posted May 08, 2019 - 07:14 CEST
Identified
Hosted instances in the Asia/Pacific region are experiencing connectivity issue. We're working to restore connectivity as quickly as possible.
Posted May 08, 2019 - 05:32 CEST
Investigating
Hosted instances in the Asia/Pacific region are experiencing connectivity issue. We're working with our hosting provider to restore full capacity as soon as possible.
Posted May 08, 2019 - 03:56 CEST
This incident affected: ftrack Services and Components (Hosted applications).