Affecting System
-
DNS Servives
-
22/09/2020 15:30
- 05/10/2020 22:24
-
Last Updated 05/10/2020 22:24
We are aware that many of our services are showing as down or offline, including web services
and e-mails.
We have identified this as DDoS Attack on our customer DNS services.
Our server engineers are currently working on redirecting this attacks and reviewing options
to prevent this happening again in the future.
Update @ 19:00 22/09/20: We are still working to mitigate this issue and roll out robust
preventative measures, normal service will resume as soon as possible and we thank you for
your patience.
Update @ 01:00 23/09/20: The attack has ceased and we've taken some steps to better protect
some infrastructure. We will review the situation over the coming hours and make a report in
due course. Please accept our apologies for any inconvenience.
Update @ 12:35 23/09/20: The attack has recommenced. We have deployed a secondary NS server
as of yesterday evening in a different IP location, however it appears the glue records are
taking some time to propagate around the Internet. We are continuing to review and look at
options.
Update @ 14:38 23/09/20: We came to the conclusion that the new DNS server built off-network
wasn't serving requests fast enough and this was leading to fatal timeouts, despite the
hardware being over-specified (on paper). We abandoned this server and built another server
in a different location over the last two hours, and changed the DNS glue records for
ns1.unlimited.uk.net and ns1.thundercloud.uk again to this new server and are waiting
propagation to the wider Internet. ns1.thundercloud.uk appears to be largely serving traffic
to the Internet as of this update, and we await ns1.unlimited.uk.net to similarly propagate.
Unfortunately, the propagation time of a change of glue records is longer for .uk.net domain
a .uk domain. We continue to monitor.
Update @ 15:27 23/09/20: The attack is ongoing, but DNS lookup services are restoring to
availability as the new DNS glue records propagates around the Internet. Traffic levels
appear nominal for the time of day (excluding attack bandwidth).
Update @ 16:24 23/09/20: The attack is ongoing, but the situation appears to be stable for
customers using our standard DNS servers. Customers who have created their own DNS server
glue records, can you please contact support via ticket to be told of the new IP's for the
ns1.* DNS servers. Where we have access to the registrar record to update these, then we are
attempting to do so but this may not be possible in all cases. We will continue to monitor
and will update if there is any material change.
Update @ 20:47 23/09/20: The attack is ongoing, but has now also added our main website to
the list of targets. We have now put the site behind CloudFlare to restore direct access to
the customer support portal. DNS for unlimitedwebhosting.co.uk is now with CloudFlare and is
propagating around the Internet, but we expect this to take a few more hours. Service to the
cloud servers seems otherwise stable, ns1.unlimited.uk.net and ns1.thundercloud.uk continue
to provide service and egress traffic is roughly nominal for this time of the evening.
Update @ 14:18 24/09/20: The attack is ongoing, and we've had to relocate our main website IP
again. Otherwise, customer services continue to improve and traffic is largely nominal for
the time of day. However, the attack has now spread to involve our group company zFast, and
it's DNS servers are similarly being attacked. We will be migrating ns1.liveboxserver.com
and ns1.liveboxserver.uk to a new host, and awaiting for DNS propagation. We will be moving
zFast also behind CloudFlare however please our primary objective is to protect DNS services
and customer websites.
Update @ 14:48 24/09/20: For customers who have created their own DNS/NS records for their
domain, then please use 185.53.57.60 in place of 149.255.60.9 and 185.53.57.130 in place of
149.255.60.10. Please don't get confused as to what replaces what! Where possible our
engineers we are updating custom name server records from our end to reduce required
customer intervention.
Update @ 14:51 24/09/20: Please also note that all our support ticket systems are working via
e-mail - if you have any trouble with accessing the Client Portals then just send a request
to the support address.
Update @ 12:00 25/09/20: The attack against our prior DNS services ceased around 01:00 and
has not yet restarted, so these are now answering queries again - which is helping
propagation where it has not already happened. We will not be reverting the change to the
ns1.* servers, and most issues which we are dealing with are DNS related. All servers are
serving data, and traffic appears to be what would be expected on a Friday. Our apologies
for any unanswered live chat and phone calls, please raise any issues via ticket either
through the relevant client portal or e-mailing the relevant support address. Engineers are
actively triaging and dealing with tickets.
Update @ 12:12 25/09/20: There is a little confusion about the new DNS servers when speaking
to people, so can we please confirm / emphasize:
For Fastnet Hosting, your NS servers should be ns0.thundercloud.uk
(149.255.60.1) and ns1.thundercloud.uk (185.53.57.60). If you are using custom name servers
based on your domain, then please update these servers to use the IP addresses 149.255.60.1
and 185.53.57.60.
For zFast, Relic Host, No-Wires, 1st
DNS and Web Hosting Payments, your NS servers should be
ns0.liveboxserver.uk (149.255.60.2) and ns1.liveboxserver.uk (185.53.57.130). If you are
using custom name servers based on your domain, then please update these servers to use the
IP addresses 149.255.60.2 and 185.53.57.130.
Update @ 15:30 25/09/20: The situation continues to be stable. We will shortly move
zfast.co.uk and the other sites to a different location behind CloudFlare for additional
security. We anticipate that the domain records will have updated worldwide by now, however
if there is any remaining propagation then please wait for this to complete.
Update @ 20:00 25/09/20: The situation continues to be stable and no further DDoS activity
has been observed since 01:00. We have completed the move of zFast, Relic Host, No-Wires,
1st DNS and Web Hosting Payments to a new location behind CloudFlare. We will be undertaking
background tasks over the weekend, however these should have minimal customer impact. We
will observe the situation over the weekend and Monday, and if there is no further activity
then we issue a full fault report via e-mail thereafter. We will update this status is there
is any significant change.
Update @ 21:00 26/09/20: There has been no recurrence of the
attack, and we have continued to fix up some features which were slightly broken due to the
infrastructure changes (e.g. VPS control panel, blog redirection, etc). Observed traffic
remains nominal for the weekend.
Update @ 22:20 05/10/20: There has been no recurrence of the
attack for 10 days, and we are now closing this network status. We will be issuing a fault
report in the next two days via e-mail to the customer base. Many thanks for the kind words
and supportive tickets over the period, they have been greatly appreciated by all at
Unlimited!