August 2024

We experienced an incident with a SATA storage unit on August 14th in the late afternoon. This second-generation storage unit was scheduled to be replaced in the last quarter of 2024.

Since then, we have developed a 4th generation of storage units that are even more robust and faster. Although we have never encountered such an incident since the deployment of the second generation, this reinforces our decision to upgrade our storage architecture for the Cloud Storage server range. The analysis of other storage units currently in production has not revealed any technical anomalies, but we will expedite their replacement.

At the same time, we will also work on improving our backup system. While it performed as expected, the restoration process is too slow for our liking. Although the restoration of VPS on our other ranges is super fast, restoring large hard drives is not sufficiently efficient. Our goal is to implement a faster restoration system than the one currently in place. Our R&D teams are already working on this, and the new system is expected to be deployed by mid-September.

We sincerely apologize for this incident, which was beyond our control.

Clients affected by this incident are eligible for the SLA provisions outlined in the contract. The SLA can be triggered upon simple request to support.

Restoration efforts are ongoing. Here is an accurate update on the current operation. The IDs listed below are displayed in the main information box of the VPS on the PulseHeberg client area, under the operating system.

At this stage, 96.77% of the affected VMs are online following the restoration. This phase is taking longer as it involves the largest VMs.

Restorations completed for the following VMs:

NCE 03 - 76887

NCE 03 - 78326

NCE 03 - 82176

NCE 05 - 1095

NCE 05 - 1179

NCE 05 - 1251

NCE 05 - 1287

NCE 05 - 1792

NCE 05 - 2459

NCE 05 - 303

NCE 05 - 340

NCE 05 - 71025

NCE 05 - 71062

NCE 05 - 71087

NCE 05 - 733

NCE 05 - 809

NCE 05 - 969

NCE 05 - 100

NCE 05 - 1123

NCE 05 - 1264

NCE 05 - 1539

NCE 06 - 79259

NCE 06 - 79779

NCE 06 - 2051

NCE 06 - 78569

NCE 06 - 80066

NCE 06 - 79507

NCE 06 - 79522

NCE 06 - 79540

NCE 06 - 80894

Restorations in progress for the following VMs:

NCE 06 - 71836

Restoration efforts are ongoing, and here is an update on the current operation. The IDs listed below can be found in the PulseHeberg client area, in the main information section of the VPS under the operating system.

At this stage, 93.54% of the affected VMs are online following the restoration. The current restoration phase is taking longer as it involves the largest VMs.

Restorations completed for the following VMs:

NCE 03 - 76887

NCE 03 - 78326

NCE 03 - 82176

NCE 05 - 1095

NCE 05 - 1179

NCE 05 - 1251

NCE 05 - 1287

NCE 05 - 1792

NCE 05 - 2459

NCE 05 - 303

NCE 05 - 340

NCE 05 - 71025

NCE 05 - 71062

NCE 05 - 71087

NCE 05 - 733

NCE 05 - 809

NCE 05 - 969

NCE 05 - 100

NCE 05 - 1123

NCE 05 - 1264

NCE 05 - 1539

NCE 06 - 79259

NCE 06 - 79779

NCE 06 - 2051

NCE 06 - 78569

NCE 06 - 80066

NCE 06 - 79507

NCE 06 - 79522

NCE 06 - 79540

Restorations in progress for the following VMs:

NCE 06 - 71836

NCE 06 - 80894

Restoration efforts are ongoing, and here is a detailed update on the current operation. The IDs listed below are available in the PulseHeberg client area under the main information section of the VPS operating system.

At this stage, 90.32% of the affected VMs are online following the restoration. The current phase is taking longer as it involves the largest VMs.

Restoration completed for the following VMs:

• NCE 03 - 76887

• NCE 03 - 78326

• NCE 03 - 82176

• NCE 05 - 1095

• NCE 05 - 1179

• NCE 05 - 1251

• NCE 05 - 1287

• NCE 05 - 1792

• NCE 05 - 2459

• NCE 05 - 303

• NCE 05 - 340

• NCE 05 - 71025

• NCE 05 - 71062

• NCE 05 - 71087

• NCE 05 - 733

• NCE 05 - 809

• NCE 05 - 969

• NCE 05 - 100

• NCE 05 - 1123

• NCE 05 - 1264

• NCE 06 - 79259

• NCE 06 - 79779

• NCE 06 - 2051

• NCE 06 - 78569

• NCE 06 - 80066

• NCE 06 - 79507

• NCE 06 - 79522

• NCE 06 - 79540

Restoration in progress for the following VMs:

• NCE 05 - 1539

• NCE 06 - 71836

• NCE 06 - 80894

Here is a detailed VM-by-VM status update of the ongoing restoration operations. The IDs listed below are visible in the PulseHeberg client area in the main information section of the VPS under the operating system.

At this stage, 87.09% of the affected VMs are online following the restoration. The current restoration phase is taking longer as it involves the largest VMs.

Restoration completed for the following VMs:

• NCE 03 - 76887

• NCE 03 - 78326

• NCE 03 - 82176

• NCE 05 - 1095

• NCE 05 - 1179

• NCE 05 - 1251

• NCE 05 - 1287

• NCE 05 - 1792

• NCE 05 - 2459

• NCE 05 - 303

• NCE 05 - 340

• NCE 05 - 71025

• NCE 05 - 71062

• NCE 05 - 71087

• NCE 05 - 733

• NCE 05 - 809

• NCE 05 - 969

• NCE 05 - 100

• NCE 05 - 1123

• NCE 06 - 79259

• NCE 06 - 79779

• NCE 06 - 2051

• NCE 06 - 78569

• NCE 06 - 80066

• NCE 06 - 79507

• NCE 06 - 79522

• NCE 06 - 79540

Restoration in progress for the following VMs:

• NCE 05 - 1264

• NCE 05 - 1539

• NCE 06 - 71836

• NCE 06 - 80894

Here is a detailed VM-by-VM status report of the restoration operations currently in progress. The IDs listed below are displayed in the PulseHeberg client area in the main information section of the VPS under the operating system.

At this stage, 83.87% of the affected VMs are online following the restoration. The current restoration phase is taking longer as it involves the largest VMs.

Restoration completed for the following VMs:

• NCE 03 - 76887

• NCE 03 - 78326

• NCE 03 - 82176

• NCE 05 - 1095

• NCE 05 - 1179

• NCE 05 - 1251

• NCE 05 - 1287

• NCE 05 - 1792

• NCE 05 - 2459

• NCE 05 - 303

• NCE 05 - 340

• NCE 05 - 71025

• NCE 05 - 71062

• NCE 05 - 71087

• NCE 05 - 733

• NCE 05 - 809

• NCE 05 - 969

• NCE 05 - 100

• NCE 05 - 1123

• NCE 06 - 79259

• NCE 06 - 79779

• NCE 06 - 2051

• NCE 06 - 78569

• NCE 06 - 80066

• NCE 06 - 79507

• NCE 06 - 79522

Restoration in progress for the following VMs:

• NCE 05 - 1264

• NCE 05 - 1539

• NCE 06 - 71836

• NCE 06 - 80894

• NCE 06 - 79540

Here is a detailed status update of the ongoing restoration operations, VM by VM. The IDs listed below are displayed in the main information section of the VPS under the operating system in the PulseHeberg client area.

At this stage, 80.64% of the affected VMs are online following the restoration. The current restoration phase is taking longer because it involves the largest VMs.

Restorations completed for the following VMs:

• NCE 03 - 76887

• NCE 03 - 78326

• NCE 03 - 82176

• NCE 05 - 1095

• NCE 05 - 1179

• NCE 05 - 1251

• NCE 05 - 1287

• NCE 05 - 1792

• NCE 05 - 2459

• NCE 05 - 303

• NCE 05 - 340

• NCE 05 - 71025

• NCE 05 - 71062

• NCE 05 - 71087

• NCE 05 - 733

• NCE 05 - 809

• NCE 05 - 969

• NCE 05 - 100

• NCE 05 - 1123

• NCE 06 - 79259

• NCE 06 - 79779

• NCE 06 - 2051

• NCE 06 - 78569

• NCE 06 - 80066

• NCE 06 - 79507

Restorations in progress for the following VMs:

• NCE 05 - 1264

• NCE 05 - 1539

• NCE 06 - 71836

• NCE 06 - 80894

Restorations pending for the following VMs:

• NCE 06 - 79522

• NCE 06 - 79540

Here is a detailed status report of the ongoing restoration operations, VM by VM. The IDs listed below are displayed in the main information section of the VPS under the operating system in the PulseHeberg client area.

At this stage, 77.41% of the affected VMs are online following the restoration. The current restoration step is taking longer as it involves the largest VMs.

Restorations completed for the following VMs:

• NCE 03 - 76887

• NCE 03 - 78326

• NCE 03 - 82176

• NCE 05 - 1095

• NCE 05 - 1179

• NCE 05 - 1251

• NCE 05 - 1287

• NCE 05 - 1792

• NCE 05 - 2459

• NCE 05 - 303

• NCE 05 - 340

• NCE 05 - 71025

• NCE 05 - 71062

• NCE 05 - 71087

• NCE 05 - 733

• NCE 05 - 809

• NCE 05 - 969

• NCE 05 - 100

• NCE 05 - 1123

• NCE 06 - 79259

• NCE 06 - 79779

• NCE 06 - 2051

• NCE 06 - 78569

• NCE 06 - 80066

Restorations in progress for the following VMs:

• NCE 05 - 1264

• NCE 05 - 1539

• NCE 06 - 71836

• NCE 06 - 80894

Restorations pending for the following VMs:

• NCE 06 - 79507

• NCE 06 - 79522

• NCE 06 - 79540

Here is a detailed VM-by-VM status of the ongoing restoration operations. The IDs listed below are displayed in the PulseHeberg client area in the main information box of the VPS under the operating system.

At this stage, 67.74% of the affected VMs are online following restoration. The current restoration step is taking longer as it involves the largest VMs.

Restorations completed for the following VMs:

• NCE 03 - 76887

• NCE 03 - 78326

• NCE 03 - 82176

• NCE 05 - 1095

• NCE 05 - 1179

• NCE 05 - 1251

• NCE 05 - 1287

• NCE 05 - 1792

• NCE 05 - 2459

• NCE 05 - 303

• NCE 05 - 340

• NCE 05 - 71025

• NCE 05 - 71062

• NCE 05 - 71087

• NCE 05 - 733

• NCE 05 - 809

• NCE 05 - 969

• NCE 05 - 100

• NCE 06 - 79259

• NCE 06 - 79779

• NCE 06 - 2051

Restorations in progress for the following VMs:

• NCE 05 - 1123

• NCE 05 - 1264

• NCE 05 - 1539

• NCE 06 - 71836

• NCE 06 - 80066

• NCE 06 - 78569

Restorations pending for the following VMs:

• NCE 06 - 79507

• NCE 06 - 79522

• NCE 06 - 79540

• NCE 06 - 80894

Here is a detailed status update of the ongoing restoration operations, VM by VM. The IDs listed below correspond to those displayed in the PulseHeberg client area under the main information section of the VPS operating system.

Restorations completed for the following VMs:

• NCE 03 - 76887

• NCE 03 - 78326

• NCE 03 - 82176

• NCE 05 - 1095

• NCE 05 - 1179

• NCE 05 - 1251

• NCE 05 - 1287

• NCE 05 - 1792

• NCE 05 - 2459

• NCE 05 - 303

• NCE 05 - 340

• NCE 05 - 71025

• NCE 05 - 71062

• NCE 05 - 71087

• NCE 05 - 733

• NCE 05 - 809

• NCE 05 - 969

• NCE 06 - 79259

• NCE 06 - 79779

Restorations in progress for the following VMs:

• NCE 05 - 100

• NCE 05 - 1123

• NCE 05 - 1264

• NCE 05 - 1539

• NCE 06 - 2051

• NCE 06 - 71836

• NCE 06 - 80066

Restorations pending for the following VMs:

• NCE 06 - 78569

• NCE 06 - 79507

• NCE 06 - 79522

• NCE 06 - 79540

• NCE 06 - 80894

61.29% of the VPS servers have been restored and are fully operational. Restoration efforts are ongoing.

At this stage, 45.16% of the affected VPS servers are back online from the backups. Restoration operations are ongoing.

File system integrity tests revealed errors on the file system of the affected storage unit. As a result, it is not possible to restart this unit at this time.

Given our inability to quickly restore access to the storage unit, we made the decision to proceed with restarting the VPS from our backups.

We have backups for all VPS affected by this incident, taken between August 13th at 8:00 PM and August 14th at 3:00 PM. Some VPS are particularly large, so the restore operation may take some time.

We will first restore the smaller VPSs, before dealing with the larger servers.

Since 15:49, we have detected errors on a SATA storage unit affecting 32 VPS servers in the Nice, France area. The incident started as a partial disruption with minimal impact on our customers. However, the situation has gradually deteriorated.

At 4:19 PM, our technical team isolated the storage unit from the rest of the production environment in order to perform integrity checks.