Hello,
I wanted to give all of you a quick infrastructure heads-up. You might have seen that from time to time, there are certain hiccups, which is due to the fact that in the last weeks, the popularity of our sites and therefore the use of the infrastructure has *heavily* increased. After some thorough checking, we can (at least relatively sure) rule out configuration issues as well as attacks, so it's actually good news - grown interest and demand. The result, sometimes 503 errors and timeouts, is bad of course.
We have therefore decided to change the way we plan our systems migration. During the last admin meeting in September, the plan was to make a clean start using centralized configuration management. This is a logical and senseful step, but unfortunately, this requires huge amounts of time - putting the documentation into a configuration language is tedious.
In order to eliminate the bottlenecks, we will now first migrate to the new, much stronger hardware and newer software, configuring services manually, to finally get back to a stable state. In other words: Replicating the existing setup with relatively few changes, but using stronger hardware. My personal plan is, even if it involves some lenghty nights and working weekends, to have the web services migrated this month, if all goes well.
Only thereafter, when the bottlenecks are eliminated, we will start with centralized configuration and further planning the growing of the infrastructure.
We are sorry for the inconveniences the hiccups cause. The admin team is doing what they can, but as for many people in the project, time of course is a limiting factor, and it's not so easy to overcome the laws of physics. Plus, scaling certain tasks to too many people isn't a solution either.
We'll keep you posted. I'd like to take a chance to thank all those silent workers in the background, especially Alex, Alin, Andreas, Christian, Erich, Friedrich, Robert and Uwe, who already spent many hours and endless nights to ensure stable services. Thanks a lot folks, you all rock!
Florian