Most Powerful Open Source ERP

[SOLVED] Incident: Nov, 12nd 2020 11:00 GMT: some requests on Production CDN are failing

Rapid.Space Incident. Read the article for real time information.
  • Last Update:2020-11-22
  • Version:001
  • Language:en

2020/11/22 finished move back [SOLVED]

8:45 GMT Europe region has been updated.

Since now all requests passing via our frontend infrastructure are correctly handled. We are monitoring the situation, we will also work on fixing this issue regarding future developments.

2020/11/20 moving back to previous version

08:30 GMT In order to provide correctly working service to our customers, we are going to move back to previously correctly working version. This procedure will be applied in non-intrusive way, outside of working hours of given CDN regions.

09:30 GMT North America region has been updated.

16:00 GMT China and Asia regions have been updated.

2020/11/19 applying the fix, held due to information gathering

10:00 GMT We confirmed that workaround applied on 2020/11/16 is not enough, there are still some errors coming from CDN itself. Fortunately we have a configuration fix to the failing component, which we are going to apply now.

12:00 GMT We are holding for now with applying the fix, as further analysis provided us more information. We will keep you updated about evolution of this situation.

2020/11/16 workaround applied

We applied workaround on the CDN to one of the CDN components. It's not the final solution, but the errors are going to drop. We are working on the final solution.

2020/11/13 more information gathered

Thanks to analysing logs, we found out that amount of errors seen by the users risen from ~0,15% to ~0,45% since 2020/10/12. This date is correlated with recent change in our infrastructure.

It's normal to observe some some errors in the CDN, as they can come from backend begin unavailable, but the observed difference comes from CDN itself, which is real problem here.

2020/11/12 11:00 GMT users informed about the problem

Users informed about the problem, which can be seen in the browser with the message Server Connection Closed. We are investigating the issue.