Problems, like buses, tend to come one after the other. Over the last week we experienced some intermittent hits on our servers that put quite a strain on the database, following a stalled unscheduled reboot incident last Friday evening (and unrelated). While the database is able to recover from such occurrences most of the time, sometimes it is unable to tidy up cleanly and tables remain open and/or marked as crashed, which requires repeated manual interventions and restart.
The pressure on the database is substantially driven by bad bots, looking for weaknesses in the infrastructure (this has been a constant for years though). We already have 2 firewalls to deal with this, but we have added a new recipe to the menu, which should help.
The network is much more stable again now, and the database logs are error free. We will be monitoring this carefully.
If you have a business critical project hosted on ourlocality, or plan anything that may go viral or attract TV coverage, do let us know in advance so we can look at improving the resilience of the service.
