infra issues continue

Hello,

sadly, the infra issues seem to continue. Not all VMs are affected, but right now it looks as if the internal connection between both machines have issues again and several VMs were paused due to the latency of the underlying Gluster file system. Infra is working to finding the reason for those issues and I hope to be back with positive news soon.

Florian

Hello,

sadly, the infra issues seem to continue. Not all VMs are affected, but
right now it looks as if the internal connection between both machines
have issues again and several VMs were paused due to the latency of the
underlying Gluster file system. Infra is working to finding the reason
for those issues and I hope to be back with positive news soon.

a quick status update:

Several sites are back, but some VMs refuse to start, as the virtualization platform is unable to determine the status, while checking on the console confirms the VMs are not powered on.

We're working on that, with a priority on getting the website back online.

Keeping you posted...
Florian

Hello,

another status update:

Things seem to take longer than planned. We're in progress of setting up a temporary, separate server to bring back the most important sites (web, BugZilla) soon, and will keep you posted.

Sorry for all inconveniences,
Florian

Hello,

a quick update:

Website and Bugzilla are back, Ask should follow soon. There's still a chunk of work left, root of issues to be found, and a major change of the setup due, further outages could still occur - but at least for the moment it's working. Kudos to the infra staff for an enormously long working day and their amazing efforts!

Florian

Indeed. Brave and patient work.. and prolly stressful too.
Million thanks lot to all!
Cor