Ellby Cloud Platform - Connection Timeout Issues in EllbyConnect – Incident details

Connection Timeout Issues in EllbyConnect

Resolved
Degraded performance
Started 5 months agoLasted 22 hours 57 minutes

Affected

EllbyConnect Service

Degraded performance from 1:15 AM to 9:23 PM, Operational from 9:23 PM to 12:12 AM

Updates
  • Resolved
    UTC
    Resolved

    We are no longer seeing Timeout issues in the EllbyConnect platform. We are continuing to work with Microsoft engineering to minimise the risk of future platform changes such as these. This incident is now resolved.

  • Update
    UTC
    Update

    The EllbyConnect service is operating normally after the Microsoft changes were reverted. We will continue to monitor under load and resolve the issue after further monitoring shows no further issues.

  • Monitoring
    UTC
    Monitoring

    We have been informed by Microsoft that a background update to a component of their execution platform that correlates to the timing of our issue is the likely cause of the degraded performance. This change is in the process of being reverted and we expect normal operations to resume once that is completed over the next hour. We will monitor operations post this rollback and resolve the issue once we are seeing normal operational performance.

  • Update
    UTC
    Update

    Microsoft have escalated this to the internal engineering team as a likely platform issue, we will advise once we have further updates.

  • Identified
    UTC
    Identified

    We are continuing to work with Microsoft support for a resolution to this issue. We will advise when we have an ETA for resolution.

  • Investigating
    UTC
    Investigating

    We are seeing increased Timeout errors in executions within EllbyConnect, we are currently working with Microsoft support to determine the source of the issue. We will update further as we make progress.