Latency in Netapp causing slow read of data for all customers in North Europe

Dear Customers,

Following the service disruption affecting the Microsoft Azure North Europe region on October 8, we have completed our post-incident analysis and would like to share our findings and the actions taken to reduce the risk of recurrence.

Incident Summary

The incident affected multiple customer environments hosted in the North Europe region, resulting in degraded performance and, for some customers, temporary service unavailability. Customers experienced symptoms including environments responding slowly or becoming unavailable, instances failing to start, and individual platform services remaining in a pending state.

Our teams worked throughout the incident to restore the affected environments, and all customer services were successfully returned to normal operation. Following the recovery, we continued monitoring the region and confirmed that the environment remained stable with no further incident-related issues reported.

Root Cause

Our post-incident investigation identified a period of high storage activity in the North Europe region. The analysis found that infrastructure operations across multiple clusters were occurring within the same time window, contributing to increased write activity on the underlying storage platform.

This condition affected the ability of some platform components to start and operate normally and contributed to the broader service degradation experienced by customers.

Resolution and Corrective Actions

During the incident, our teams performed recovery actions across the affected environments and worked closely with our Cloud Service Provider while service availability was being restored.

Following the investigation, we introduced additional safeguards across the North Europe clusters to stagger infrastructure operations that had previously been occurring at approximately the same time. This reduces simultaneous storage activity and lowers the likelihood of the same conditions recurring.

We are also continuing to evaluate additional improvements to the underlying storage architecture as part of our short- and longer-term resilience work.

Incident Response and Communication

In addition to the technical measures, we have reviewed our response to this incident and identified opportunities to improve how quickly broader service disruptions are communicated.

Closing

We sincerely apologize for the disruption this incident caused to our customers and their business operations.

The incident and the feedback received have provided valuable input for improvements to both our platform resilience and our incident response processes.

Thank you for your patience, understanding, and continued trust in Jedox.

Should you have any additional questions or feedback regarding this incident, please contact our Support Team.

Dear Customers,

we would like to provide a final update regarding the service disruption reported yesterday in Cloud Region North Europe.

Over the last 24 hours, we have closely monitored the affected services and are pleased to confirm that no further issues related to this incident have been reported to our teams. All customer environments continue to operate normally, and we consider the service fully restored.

In parallel, we continue to work closely with our cloud service provider, Microsoft, on the postmortem analysis. The objective is to establish a comprehensive understanding of the root cause and define a robust set of corrective and preventive actions to further strengthen platform resilience.

We have also identified opportunities to improve our incident response process. As an immediate follow-up, we will review and enhance our communication procedures to ensure that service disruptions are published to the Status Page in a more timely and consistent manner.

As the service has remained stable and no additional impact has been observed, this incident is now being marked as Resolved and will be closed on our Status Page. We will publish a follow-up Root Cause Analysis (RCA) within the next 7 days, once the postmortem process has been completed and the agreed actions have been finalized.

We would like to sincerely apologize for any disruption this incident may have caused to your business operations. We greatly appreciate the feedback received from our customers during this event and want to assure you that it will be carefully considered as part of our continuous efforts to improve the reliability, transparency, and quality of our services for all Jedox customers.

Should you have any further feedback regarding this incident, our response, or our communication throughout the event, please do not hesitate to contact us.

Thank you for your patience, understanding, and continued trust in Jedox.

Dear Customers,

Since our last update, we have continued working closely with our cloud service provider to further investigate the regional disruption affecting the Microsoft Azure North Europe region.

At this time, all customer environments remain operational and stable. In addition to the recovery actions already completed, further corrective measures have been implemented to reduce the likelihood of a similar issue occurring again.

A detailed postmortem analysis is being conducted jointly with our cloud service provider to identify the root cause and document any additional preventive actions. The findings will be reviewed as part of our standard incident management process.

As a precaution, the incident will remain open on the Status Page while we continue to monitor the platform and gather feedback from customers. Based on the stability of the environment and any reports received, a decision regarding incident closure will be made tomorrow.

If you continue to experience any issues with your environment, please report them through our customer portal, so that our teams can investigate promptly.

We appreciate your patience and cooperation throughout this incident.

Dear Customers,

We would like to provide an update regarding the service disruption affecting customer environments hosted in the Microsoft Azure North Europe region.

Service has now been restored, and all customer applications and platform services are currently operational. Our teams have completed the required recovery actions, and customer environments are available again.

While service availability has been restored, we continue to work closely with Microsoft Azure engineering teams to identify the underlying root cause of the regional disruption. As a precaution, the incident will remain open on the Status Page while we continue to monitor the environment and validate long-term stability.

At this time, we are not aware of any remaining service impact. However, if you experience any issues with your environment, please report them through My Support so that our teams can investigate them promptly.

We appreciate your patience and understanding throughout this incident. Further updates will be provided if additional information becomes available from our ongoing investigation with Microsoft.

Dear Customers,

We are currently experiencing a major service disruption affecting multiple customer environments hosted in the Microsoft Azure North Europe region.

Current Impact

Customers may experience one or more of the following issues:

• Environments responding slowly or becoming unavailable
• Instances failing to start
• OLAP and other platform services remaining in a pending state or failing to start
• Connectivity issues when accessing environments
• Intermittent service degradation across affected instances

Timeline

The first degradation of services was noticed around 12:00 UTC. At that time, our engineers immediately began investigating the issue and working to restore services for affected customers.

As the incident evolved, the impact expanded beyond individual environments and developed into a regional outage within Microsoft Azure North Europe, affecting customers across the region.

Investigation Status

The incident has been classified as Critical and is being actively investigated. Our teams are working closely with Microsoft Azure to identify the root cause and restore full service.

Based on our findings, elevated latency within the affected Azure region appears to be preventing some services from starting correctly and causing others to remain in a pending state.

As part of our mitigation efforts, we have been:

• Restarting affected services and instances
• Repairing impacted environments where possible
• Removing and recreating failed RPC components when required
• Performing recovery actions to restore service availability while Microsoft investigates the underlying platform issue

Next Update

We are continuing to work with Microsoft and monitor the situation closely. The next update will be provided as soon as additional information becomes available or significant progress has been made.

We apologize for the disruption and appreciate your patience and understanding while we work to restore full service.

If you have any further questions or concerns, please do not hesitate to contact our Support Team.

Thank you for your continued partnership and patience!