Degraded performance for clients in the Asia Pacific region

Incident Report for ftrack

Postmortem

Root Cause:

During routine maintenance, a configuration change was made to the Kubernetes clusters in Google Cloud, which inadvertently caused an internal service to stop responding. This caused a partial outage and affected the ability of users to access certain features.

Resolution:

Our team immediately identified the issue and worked to resolve it as quickly as possible. The configuration change was rolled back, which restored the service.

Impact:

During the partial outage, users may have experienced difficulty accessing certain features. We apologize for any inconvenience this may have caused.

Next Steps:

We will continue working closely with our cloud service providers to ensure our services remain reliable and available to our users. We will also thoroughly review our change management policies and procedures to prevent similar incidents in the future.

Posted Mar 08, 2023 - 09:42 CET

Resolved

This incident has been resolved.
Posted Mar 08, 2023 - 00:15 CET

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Mar 08, 2023 - 00:14 CET

Identified

Due to a configuration change in the Asia Pacific region, ftrack workspaces hosted in the region are experiencing a partial outage.

We have identified the issue and are implementing a fix.
Posted Mar 07, 2023 - 23:30 CET
This incident affected: ftrack Services and Components (Hosted applications).