Again we would like to apologise for the issues experienced here. We have now completed our investigation and we believe the cause of this particular issue was the result of 3 unrelated problems:
We experienced a significant amount of inbound spam into the Spam Experts platform, this resulted in mail being delayed and queued for quite some time whilst it handled this.
One of the filtering nodes was out of action due to a software fault.
Another node was experiencing performance issues which has been identified as a hardware fault.
This resulted in the cluster not only having a significant "spam attack" but it was also running 2 nodes down which also impacted performance slightly. We now have the node with the software fault back in action and for the hardware fault, this has been logged with our hardware vendor to have the required replacement part sent over as soon as possible.
This unfortunately was a case of multiple events occurring at the same time which resulted in the slower filtering performance and delays in mail receipt.
We can confirm that the inbound attack has stopped and the filtering cluster is now operating correctly minus the one node which we are waiting on replacement hardware for.
We have also provided feedback to SpamExperts on a way to improve the software performance under such heavy load, this should improve performance significantly if such a situation ever reoccurs.
For now, we are going to leave this in a monitoring status until the hardware fault has been resolved.
As before, we do apologise for any disruption this has caused.
Monitoring
Again we would like to apologise for the issues experienced here. We have now completed our investigation and we believe the cause of this particular issue was the result of 3 unrelated problems:
We experienced a significant amount of inbound spam into the Spam Experts platform, this resulted in mail being delayed and queued for quite some time whilst it handled this.
One of the filtering nodes was out of action due to a software fault.
Another node was experiencing performance issues which has been identified as a hardware fault.
This resulted in the cluster not only having a significant "spam attack" but it was also running 2 nodes down which also impacted performance slightly. We now have the node with the software fault back in action and for the hardware fault, this has been logged with our hardware vendor to have the required replacement part sent over as soon as possible.
This unfortunately was a case of multiple events occurring at the same time which resulted in the slower filtering performance and delays in mail receipt.
We can confirm that the inbound attack has stopped and the filtering cluster is now operating correctly minus the one node which we are waiting on replacement hardware for.
We have also provided feedback to SpamExperts on a way to improve the software performance under such heavy load, this should improve performance significantly if such a situation ever reoccurs.
For now, we are going to leave this in a monitoring status until the hardware fault has been resolved.
As before, we do apologise for any disruption this has caused.
Investigating
We're aware that some inbound email is being delivered later than expected. We're investigating the cause and will provide an update shortly.
