Server Status

Below is a real-time overview of our servers where you can check if there's any known issues.

Server Name	HTTP	FTP	POP3	Server Load	Uptime
Bach
Baracus
Bronte
Bullseye
Fleetwood
Hermes
Hyrule
Joshua
Kipling
Murdoch
Nash
Orwell
Rosie
Shakespeare
Shaw
Wilde

Server Maintenance (Resolved) Medium

Affecting Server - Shaw

10/03/2024 02:00 - 10/08/2024 14:48
Last Updated 10/01/2024 09:05

We will be taking the server down to tweak some BIOS settings and to perform a stress test on the CPU as it appears to be suffering from overheating. We expect this to take between 30 - 60 mins but have set aside a longer window just to cover any unexpected issues we run into whilst running the tests.

Drive Swap (Resolved) Medium

Affecting Server - Fleetwood

09/13/2024 04:00 - 09/13/2024 05:00
Last Updated 09/13/2024 07:41

Our monitoring systems have picked up a failed drive in this server. We will be taking the server down to replace the failed drive at 4am UK time on the 13th Sept. Downtime is expected to be around 30 mins.

Update: This was completed as scheduled

Hardware test and BIOS update (Resolved) Medium

Affecting Server - Orwell

09/05/2024 02:00 - 09/05/2024 08:00
Last Updated 09/05/2024 08:45

Maintenance Details: Full hardware test and BIOS update
Start Date/Time: Thursday 5th September, 2am EST
Expected Downtime: 4 - 6 hours
Details: The server has been suffering from random reboots for the past few days so we need to take the server offline to fully check and replace any faulty hardware to hopefully find and fix the underlying cause of the issues.

Dark Fib (Resolved) Medium

Affecting System - Router Software Upgrade

06/19/2024 00:01 - 06/20/2024 08:19
Last Updated 06/19/2024 05:51

This notification is to inform you of scheduled maintenance that may affect your service if your website(s) are hosted in NYC. Please review the details of this notification to determine how it will affect your service.

Event Type: Router Software Upgrade

Start: 2024-06-19 00:00:00 Eastern

End: 2024-06-19 4:00:00 Eastern

Location(s): New York City Metro (NY1)

Client Affecting: Yes

Details: Our dark fiber carrier to 165 Halsey will be upgrading router software. They expect the upgrade to complete in 15 - 20 minutes. During this period, there may be brief moments of packet loss as traffic shifts to other carriers and internet exchanges.

US Datacenter Maintenance (Resolved) Medium

Affecting Other - Network

05/13/2024 07:07 - 06/13/2024 15:26
Last Updated 05/13/2024 07:09

We have just received the following notification from the datacenter that houses some of our US based servers. The maintenance is to upgrade a number of routers to fix a software bug. The maintenance window is scheduled for 02:00 EST and 14:00 EST on the 13th May:

The JunOS version on the router will be upgraded to fix a software related bug. This bug is known to cause routing related issues and will be installed with high priority to prevent unexpected network downtime. We expect brief unavailability in 15 to 20 minute periods during the upgrade process. The large maintenance period has been scheduled to allow preparation prior to the upgrades and monitoring post upgrades.

Disk Issues (Resolved) High

Affecting Server - Hyrule

05/03/2024 06:21
Last Updated 05/05/2024 09:17

Our monitoring systems picked up some RAID issues on the server. On checking we could see a disk had dropped out of the RAID array. We have taken the server down to investigate.

Update: We are unable to recover the disk that has vanished. The working disk is also showing a small number of errors. As such we will be migrating data over to a new server rather than replacing all the drives in the current server. We plan to do this in the early hours of Monday, which means the server will be running on a single disk over the weekend but we will be closely monitoring it. The server should be deemed "at risk" over the weekend but it should keep ticking over until we are able to get the new box set up and data migrated.

IP addresses will be moved over to the new server so there shouldn't be any impact for clients.

Update: This has been completed and websites are running on a new box. There was no downtime since IP addresses were moved over to the new server.

Main IP Change (Resolved) High

Affecting Server - Nash

04/30/2024 09:24 - 04/30/2024 10:05
Last Updated 04/30/2024 10:05

We will be changing the main IP address of the Nash server due to issues we are having with BT internet and the current main IP.

We expect around 10 mins of downtime whilst this takes place. Websites will continue to remain on the same IP address as before. It is only cPanel/WHM that runs on the main IP.

Update: This was completed with only a few seconds of downtime.

Scheduled Downtime (Resolved) High

Affecting Server - Orwell

03/27/2024 10:00 - 03/30/2024 07:47
Last Updated 03/27/2024 09:59

The Orwell server will be taken down between 11am EST and 1pm EST today (27^th). This is due to power load balancing taking place in the datacenter.

Server Down (Resolved) High

Affecting Server - Shaw

03/13/2024 09:04 - 03/15/2024 07:51
Last Updated 03/13/2024 17:15

The "Shaw" server has gone unresponsive and we are unable to access it remotely. We have issued a support request with the onsite datacenter technicians and are currently awaiting an update from them. We will obviously be doing all we can to get the server back up ASAP.

Update: We have established a KVM session on the server and the server is up and running but unable to connect to external networks. This points to a network issue at the datacentre and we are awaiting their response.

Update: The DC have replaced the optic cable as well as swapped to the 2nd NIC interface but the issue remains. They believe the issue is at switch level and have escalated the issue to the senior network engineer.

Update: The network access has been restored to the server. We can now also see that a hard drive isn't being picked up in the server but we will schedule a time to check this out in further detail in the coming days. Any clients with open tickets will get a full RFO once we receive it from the datacentre.

EU Router Upgrades (Resolved) Medium

Affecting System - Multiple EU DC's

11/18/2023 01:30 - 11/21/2023 10:25
Last Updated 11/16/2023 10:44

One of our upstream providers will be rolling out software updates to all of their JuniperMX routers in the early hours of Saturday morning (UK time).

There is expected to be around 20 minutes of disruption whilst the routers switch over to the new software.

This will impact all servers hosted in London and Amsterdam (both shared and dedicated server clients).

The work on the routers in AMS is scheduled to take place between 1.30am and 2.30am and the work on the routers in LDN is scheduled to take place between 4am and 5am.

NYC Partial Network Outage (Resolved) High

Affecting System - NYC Network

10/25/2023 08:05
Last Updated 10/26/2023 08:14

There is currently a partial network outage in our upstreams NY1 facility caused by a fiber break between PNJ and 165 Halsey. Zayo (the company responsible for the fiber cable) has taken offline the ring to perform a repair. When the fiber break has been repaired servers will be accessible. This only impacts services hosted in NYC. All other locations remain online.

All servers are up and running, it's just the network that is currently unreachable due to an external fiber cable break so as soon as the broken cable is repaired connectivity will be restored to the building and things will be back to normal.

We don't have a timescale in terms of how long Zayo will take to fix the cable break as they aren't a direct provider of ours so we only get limited information. The cable break has however taken down a lot of internet connectivity in the NYC area so we imagine it will be done as quickly as possible to minimize the disruption for all customers in that area.

Update 1: You can view live updates on the cable repairs here - https://tranzact.zayo.com/?fbclid=IwAR3jvhdP0KLy8iwKA_ZOC0giwqEwysojd8YTXB72AfITyHuZx5-UmO5dgio#!/case-detail/5004z00001tVV9sAAG/TTN-0007089358/

Update 2: Zayo have completed their work to what they thought was the initial cause of the issue but they still aren't seeing any connectivity. Please follow the link above for the latest updates from the 3rd party fiber carrier.

We have also heard from another of our upstream partners that there is a redundant fiber link into the datacentre (both provided by Zayo) but the primary link was damaged a couple of days ago and this recent work has cut the backup link. Our upstream are also actively working on other solutions to get network connectivity back into the building should the Zayo work not be resolved very soon.

Update 3: This issue was resolved yesterday afternoon when Zayo finally managed to fix the fiber link that had been split in multiple places. We will be sending a full RFO to all impacted customers as soon as we receive one from our upstream.

Network outage (Resolved) High

Affecting Server - Bach

02/28/2023 10:56 - 03/18/2023 08:30
Last Updated 02/28/2023 11:10

There is a network issue impacting the Bach server. The server is up but network is unreachable. The update we have from the datacentre is:

We are experiencing an issue with our VC Juniper switches in our core network.

Techs are onsite and hard cycling the devices so we can regain access.

We will update when we have more info.

Update: The network is back up after the switches were rebooted. The datacentre are continuing to monitor things and to look through the switch logs to try and find the underlying cause of the issue. It was down for around 20 mins.

Drive replacement (Resolved) Medium

Affecting Server - Shakespeare

01/19/2023 11:50
Last Updated 01/19/2023 13:07

We are seeing one of the hard drives in the Shakespeare server showing a small number of errors. To avoid any future issues we are going to remove the drive from the RAID array and swap it out with a fresh drive.

We plan to do this work at around midday UK time on the 19th Jan. We have placed a downtime window of 60 - 120 mins for the work but if all goes to plan it shouldn't take that long.

Drive swaps normally pass without incident but we do have backups taken from this morning should things not go to plan.

Update: The drive swap has been completed and RAID is currently rebuilding in the background. Load on the server will be higher than normal until this completes. Downtime was around 50 mins.

Server Down (Resolved) High

Affecting Server - Shakespeare

01/16/2023 16:04
Last Updated 01/17/2023 09:33

We have noticed the server shakespeare.hyliahub.com is down.

Upon checking there has been a massive mistake at the datacentre and they have reinstalled an operating system on this server rather than another one they were working on.

This has wiped all data on the server meaning we are going to have to restore from backups.

We are awaiting the DC to reinstall the correct operating system and once done we will install cPanel, Cloudlinux and then start the backup restoration ASAP.

Please accept our sincere apologies about this.

Update: We have begun restoring accounts. Customers who pay for daily backups will have backups restored from the backup taken in the early hours of the 16th Jan (today) and customers who are on standard weekly backups will have accounts restored from the last weekly backup which ran on the 10th Jan.

The accounts are restored in alphabetical order and based on past experience the restoration should complete in around 6 - 8 hours. Some sites with usernames at the start of the alphabet will already be back online but accounts with usernames toward the end of the alphabet will take longer to come back up.

Apologies again about this unexpected outage, we are working as fast as possible to get everything back up and running.

Update: The restoration completed at around 10pm and all accounts are back online. Please get in touch if you find any issues with websites.

Disk Failure (Resolved) High

Affecting Server - Rosie

01/08/2023 21:06
Last Updated 01/09/2023 15:30

Updates as soon as we get them from the DC

18.30: There is an issue with the network at the Data Center and we are checking with the engineers

20.15: A disk failure was found and our engineers are working on resolving the issue. We will restore all accounts to fresh server if this fails.

Update: We have exhausted all possibilities with regards repairing the existing server. As such we have taken the decision to spin up a new server and restore all accounts from backups. Unfortunately this means it will be an extended outage. We will update again when we have got the new server ready and are ready to restore backups. We would estimate it would take around 2 hours to spin up a new server and install the OS then another 12 hours at a minimum to restore all accounts but will keep this status page updated.

Update: We have successfully spun up a replacement server and have now started the process of restoring all accounts from our offsite backups. The backup restoration started at around 9.15am UK time and there are 286 accounts to restore. We will try and provide updates every 2 hours with details on how the restoration is progressing.

Update: After just under 3 hours 176 accounts have been restored with 110 left to go.

Update: The restoration completed in around 5.5 hours which was well ahead of schedule. Sites using the D9 hosting nameservers should already be back online. If you are using 3rd party DNS servers or Cloudflare you just need to update the IP address to point to:

116.202.84.239

If you find any issues with websites please do get in touch with tech support and we will treat it as a priority.

Server Down (Resolved) High

Affecting Server - Shakespeare

01/07/2023 14:52
Last Updated 01/07/2023 19:35

At around midday UK time our monitoring systems notified us that the Shakespeare server was unreachable.

We issued a reboot request but the server didn't come back up so we contacted the datacentre. They replied afer 90 minutes saying they had also issued a power cycle to the server.

The server still didn't come back up so we requested iDRAC logins for the server to access a remote console session. We then found the server was unable to boot due to corrupt meta data in the file system.

We are currently trying to fix the data corruption issue but at the same time have spun up a spare server and have started the process of restoring backups to this new server should we be unable to recover from the data corruption.

If the backup restoration completes before we are able to fix the data corruption issue on the old server we will point nameserver IPs to the new server to get your sites up quicker. Customers that pay for our daily backup service will have a backup restored from the early hours of the 7th Jan whilst customers who have standard weekly backups will have a backup restored from the last weekly backup which ran on the 3rd Jan.

It's not possible to give an exact timeframe on when the backup restoration will complete as it depends on a number of factors but in our experience 8 - 12 hours would be a reasonable estimate.

We will update clients directly when either the backup restoration has completed or the data corruption issue on the old server has been fixed.

Update: All accounts were successfully restored from backups in around 5 hours.

If you are using a 3rd party DNS service such as Cloudflare or Sucuri you need to log into the service and update your A record to point to the following IP address:

195.206.165.225

If you are unsure how to make this change please provide us with the login details to your 3rd party DNS service and we will be happy to make the change for you.

If you notice any issues with your sites post restoration please don't hesitate to open a ticket with technical support.

Drive Replacement (Resolved) Medium

Affecting Server - Nash

02/21/2022 10:34
Last Updated 02/22/2022 09:01

Our monitoring systems have picked up a failed drive on the Nash server. We are just taking a backup of all accounts and will then ask the sever to replace the failed drive.

If all goes well downtime should be 30 - 60 minutes.

Update: This was completed without any issues.

Server Down (Resolved) High

Affecting Server - Wilde

01/27/2022 10:26
Last Updated 01/27/2022 14:44

We are aware that the "Wilde" server is currently unreachable. We are unable to access the server remotely via KVM so the issue has been escalated to the datacentres onsite team and they are checking on it.

Initial thoughts are that it could be a PDU failure in the rack. We will post more updates as and when we get them from the datacentre team.

Update: The Wilde server came back online around 1 hour ago. There was a failed PDU at the NYC datacentre which affected the rack that the server is located in and caused the server to lose power. The failed PDU was replaced and the server is back online.

Apologies for any inconvinience caused.

Disk Replacement (Resolved) Medium

Affecting Server - Shaw

11/10/2021 13:31 - 01/24/2022 11:50
Last Updated 11/10/2021 21:43

Our monitoring systems have picked up a failed drive in the server, we will shortly be taking the server down to check this in more detail but will try and keep any downtime to a minimum. We will post more updates as we have them.

Update: Unfortunately when rebooting the server to check the status of the disk this caused some data corruption which we are still trying to recover from. Most services on the server are running although websites aren't coming up at present. We are currently spinning up a replacement server that we can restore backups from should we need to go down that route but we will continue to try get services back up on the current server at the same time.

Final Update: Earlier today the server suffered a RAID failure which caused file system errors that we were unable to fully recover from. As such, the decision was taken to spin up a replacement server and restore the backups that were taken just prior to the old server suffering the file system errors.

The restoration is progressing well and we expect it to fully complete within the next 4 - 6 hours, however it is possible that the restoration of your account has already been completed.

If you are using the D9 Hosting nameservers for your website(s) then no action is needed on your part, your service will return to normal within the next few hours if it isn't already back up and running already.

If you are using a 3rd party CDN service such as Cloudflare then action IS needed on your part since your websites now reside on a different IP address - switching the IP address from the old server would have added a lot more time to the restoration process since we need to wait for the datacentre's billing team to make the changes and they are notoriously slow!

So if you are using Cloudflare or a 3rd party DNS service please log in and update the A record for your domain(s) to point to the following IP address:

185.150.190.169

If you would like any help with this then please do let us know and we would be happy to make the changes for you.

If you are a reseller customer and use non-D9 Hosting nameservers then you will need to log into your domain registrar and update your custom nameservers to use the IP address listed above. If you would like any help with this please do let us know.

If you previously had a dedicated IP address your websites are currently using the shared IP address listed above, we will be moving your websites back over to your origianl dedicated IP address when the datacentre have updated things at their end which is likely to be early next week. We will be in touch with customers on an individual basis to let them know when their dedicated IP has been reassigned.

If you are seeing a cPanel "default website" page when visiting your website or are receiving an "invalid password" error when accessing emails then it means your account is still in the restoration queue and things will be back to normal shortly. If you are still seeing this message after 6 - 8 hours please let us know and we can double check everything for you.

Apologies for the long email, and apologies for the extended downtime. Whilst rare, RAID failures are one of those issues that crop up from time to time and don't have a quick easy fix, hence the importance of having reliable offsite backups that we are able to restore from when needed!

Thanks again for your patience and understanding, we appreciate it.

Drive Replacement (Resolved) Medium

Affecting Server - Shaw

10/19/2021 08:22
Last Updated 10/20/2021 08:04

Our monitoring systems have picked up a failing drive on the Shaw server. We are currently taking a full backup of all accounts on the server and will then proceed to have the datacentre swap the drive. If all goes well then downtime is likely to be 60 - 120 mins.

Update: We were able to hot-swap the drive so there was no downtime needed. RAID rebuild is currently in progress and should complete within the next 4 - 6 hours.

Drive Swap (Resolved) Medium

Affecting Server - Hermes

10/12/2021 09:07
Last Updated 10/12/2021 16:05

Our monitoring systems have picked up a failed drive in the Hermes server. We will be taking the server down to replace the failed drive. Downtime is likely to be 30 - 60 mins.

We are currently taking a full backup of all accounts on the server before proceeding. Once the backup has finished (ETA ~6hrs) we will take the server down to swap the drive.

Update: This has been completed and server is back online. Load will be slightly higher than normal until the RAID rebuild has completed.

Drive Replacement (Resolved) Medium

Affecting Server - Hyrule

10/08/2021 09:12
Last Updated 10/08/2021 15:54

Our monitoring systems have picked up a failed drive in the Hyrule server. We will shortly be taking the server down to replace the failed drive. Downtime is likely to be 30 - 60 mins if all goes to plan. We do have a backup of all accounts on the server should we run into any major issues.

Update: This has been completed and the server is back online

Drive Swap (Resolved) Medium

Affecting Server - Fleetwood

09/28/2021 14:40 - 10/08/2021 09:14
Last Updated 09/28/2021 14:41

We will shortly be taking the Fleetwood server down to replace a failed hard drive. Downtime shouldn't be more than 30 - 60 minutes.

Server Down (Resolved) High

Affecting Server - Kipling

08/10/2021 13:28
Last Updated 08/10/2021 15:45

The server "kipling.hyliahub.com" has gone unresponsive and we are unable to access IPMI for the server. We have a ticket open with the datacentre and will post more updates as we have them.

Update: The rack the server was in suffered a failed PDU hence the outage. The datacentre replaced the failed PDU with a new one and the server is back up.

Baracus network issue. (Resolved) Critical

Affecting Server - Baracus

06/08/2021 13:57 - 06/15/2021 10:58
Last Updated 06/08/2021 14:03

Earlier today we were alerted to a load issue on the Baracus server. Our usual methods to fix this haven't worked and it looks to be a hardware issue. In our experience, it is better to restore accounts to a new server rather than to have the datacentre take the server offline for 6 - 12 hours to run an extended hardware check. We will be starting the data restoration from Monday's backup (clients who pay for the daily backup will have the Tuesday backup restored) and when all data has been restored IP addresses will be moved over to the new server. The old server will remain online with mail services still available but HTTP will be offline to keep the load under control.

Network Maintenance (Resolved) Medium

Affecting Other - Nash, Baracus & Fleetwood Servers

05/25/2021 07:00 - 05/25/2021 09:14
Last Updated 05/20/2021 09:01

We have received a notification from our datacentre that some scheduled network maintenance will be taking place on Tuesday the 25th May at around 7am GMT (+1/BST) and this will cause around 30 - 60 minutes of network disruption for the server that your hosting account resides on.

Please find the full details below:

"Please note we are planning scheduled maintenance to replace a top of rack switch with newer generation hardware capable of 40G networking.

The downtime for this is expected to last 30-60 minutes. All network connectivity within rack D1D10 will be offline for this duration, however all servers will remain powered on. No action is required on your part.

Once the switch has been replaced all servers will regain connectivity automatically."

Scheduled Network Maintenance (Resolved) Medium

Affecting Server - Hermes

05/11/2021 07:00 - 05/11/2021 08:00
Last Updated 05/11/2021 08:33

Please note we are planning scheduled maintenance to replace a top of rack switch with newer generation hardware capable of 40G networking.

The downtime for this is expected to last 30-60 minutes. All network connectivity within rack D1D11 will be offline for this duration, however all servers will remain powered on. No action is required on your part.

Once the switch has been replaced all servers will regain connectivity automatically.

Drive Swap (Resolved) Medium

Affecting Server - solus.hyliahub.com

04/30/2021 22:00 - 04/30/2021 23:00
Last Updated 05/01/2021 05:53

We will be taking down the "Kraken" VPS node at around 10pm EST this evening to replace a failing drive in the server. Downtime shouldn't be anymore than 60 - 120 minutes but with any drive swap it is always a good idea for clients to take backups of any important data prior to the work commencing.

Update: This was completed successfully

Server Problems (Resolved) Medium

Affecting Server - Hermes

02/12/2021 01:00
Last Updated 02/23/2021 16:01

The newest updates will be posted at the top of this page

18th Feb: 06.29 EST: The restoration of websites has completed and sites have been back online for around 1 hour. Any clients using a 3rd party DNS provider or custom (non-D9) nameservers will need to update their A records to:

213.166.86.58

More details on the restoration is due to be emailed to all impacted clients in the next 30 mins.

18th Feb: 03:45 EST: Last night we took the decision to provision a second replacement server in our UK facility as we have better quality hardware available to us at shorter notice in the hope that the restoration would complete sooner. We are pleased to report that this has made a vast improvement to the restoration times and we hope to have service restored by 6am EST. As the server is in a different facility an IP address change will be involved (for the vast majority of customers no action will be needed) but we thought this would be better than waiting for another 24 hours for the original restore to complete.

17th Feb, 12.12 EST: The restoration has now been running for around 12 hours (we got out time zones mixed up in the previous update!) and just over 30% of accounts have been restored to the new hardware. We will continue to monitor the restoration and make any needed tweaks to the settings to ensure everything is restored as quickly as it possibly can be.

17th Feb, 01:55 EST: The restoration of the backups onto a new server has now been running for around 6.5 hours and is around 10% completed. When all of the data has been restored from the backups we will move the IP addresses over from the current server to the new server and sites will begin to load.

16th Feb, 16:40: We have just received the logins for the replacement server and will proceed with the setup. We will then start the backup restoration process. We will update this page as and when we have more info.

16th Feb, 11:52 EST: We are still waiting for the datacentre to provision a new server so we can begin to restore backups. We heard from them around 90 minutes ago that the server was undergoing final testing but as of yet it hasn't been delivered to us.

16th Feb, 08:21 EST: Unfortunately our efforts to fix the file system corruption on the server are proving to be futile and MySQL remains down. At the present time our only option is to wait for the datacentre to provision the new hardware for us so that we can restore data from our backups. We have reached out to their management team and asked them if they can speed up the delivery for us but as yet we haven't received a response. We will update this page as soon as we get more information.

16th Feb, 04:55 EST: Over the past few hours we have observed some file system corruption on the server caused by a bad hard drive and this is causing services to fail, namely the MySQL service. Other services are up and running without any issues (HTML driven websites, emails, DNS) but MySQL driven websites will show an error message.

We are doing all we can to restore service but depending on how severe the file system corruption is we may need to move to new hardware and restore data from our backups.

As it currently stands this is where we are at:

1. We have our senior admins trying to fix the problem to bring up MySQL driven websites on the current hardware

2. We have ordered a replacement server with the datacentre and have asked for the setup to be expedited although they have informed us this could take up to 48 hours due to having less staff "on the shop floor" to comply with safe social distancing measures, for us this time delay is unacceptable but we are left with little option other than to wait for them to do the build

As soon as the new hardware is provisioned we will begin restoring websites from our backups. When the restoration has been completed we will then sync any accessible data from the old hardware over to the new and then migrate IP addresses across to the new server.

We appreciate that nobody likes it when their websites are down, least of all us, but as you can imagine, we are becoming swamped with support tickets so we would request that you visit this page as any updates we have we will post here. Submitting multiple tickets about the issue slows down our ability to fix the issue.

FAQ's
======

Q: Do you have an ETA?

A: It is not possible to give an ETA as there are too many unknowns that are out of our control at present. Namely the time it takes the datacentre to provision the new hardware and how long (if at all) we are able to recover from the file system corruption.

Q: How did this happen?

A: The cause appears to have been a failing drive in the RAID 10 array. We attempted to replace the drive last week but the datacentre informed us the RAID array wouldn't take the new drive so we waited for them to get an exact match in stock. They attempted to add the new drive again last night but as before the RAID array wouldn't accept it. They then re-added the original failing drive to the server and this is where the file system corruption appears to have taken place.

Apologies again for the inconvenience caused but rest assured we are doing all in our power to get things back up online as soon as possible.

Scheduled Maintenance (Resolved) High

Affecting Server - Hyrule

11/30/2020 12:30
Last Updated 11/30/2020 17:00

We will be taking the Hyrule server down to run a network test to diagnose an underlying issue on the server. Downtime is likely to be 60 - 120 minutes.

Update: This was completed as scheduled well within the timeframe.

Server Migrations (Resolved) Medium

Affecting Other - Kipling and Shaw

07/30/2020 12:17 - 07/30/2020 13:13
Last Updated 07/30/2020 13:12

We will be migrating all accounts on the Kipling and Shaw servers over to new hardware running CentOS 7.

Very little, if any, disruption is anticipated and all clients impacted by this have been emailed.

Update: This has now been pushed back to this Thursday (30th July)

Update: This has now been completed. Please contact support if you have any issues.

Drive Swap (Resolved) Medium

07/28/2020 00:00
Last Updated 07/29/2020 09:54

We will be taking down the Goron server at midnight EST on Tuesday to replace a failed hard drive.

If all goes to plan downtime shouldn't be any more than 60 minutes.

Sporadic packet loss (Resolved) Medium

Affecting Server - Hyrule

01/09/2020 09:33
Last Updated 01/14/2020 13:43

We have been seeing an ongoing issue with the "Hyrule" server.

From time to time we are seeing sporadic packet loss on the server which is causing general sluggishness when accessing sites and emails.

We believe we may have found the underlying cause of the issue and have put in place a fix but only time will tell as this has been successful as the issue is very sporadic.

To complete this work we will need to do a number of server reboots and we've got these scheduled to begin at 11am EST on the 9th January. The server will have brief periods of downtime whilst these reboots take place.

Once the work has been completed we will continue to monitor the server to ensure the problem has been resolved. If the problem reappears our next course of action will be to set up a new box and migrate sites off the troublesome hardware.

Update: We believe we have put a fix in place but we will need to monitor the server for a number of weeks to ensure the underlying problem has been resolved.

Scheduled Maintenance - January 9th 2020 (Resolved) Medium

Affecting Server - Murdoch

01/09/2020 00:30 - 01/09/2020 09:33
Last Updated 01/09/2020 09:33

We will be taking the "Murdoch" server offline at 00:30 EST on Thursday 9th January to perform a BIOS update and RAM test on the server.

We expect the server to be down for around 2 - 4 hours whilst this takes place.

Update: This has now been completed.

Drive Swap (Resolved) Medium

Affecting Other - Kraken VPS Node

01/07/2020 00:30
Last Updated 01/07/2020 15:57

We will be taking down the "Kraken" VPS node on the 7th January at 00:30 EST to replace a failed drive in the RAID array, we don't expect this to take any more than 60 minutes unless we run into problems.

Update: This has been completed and the RAID array has rebuilt successfully.

RAM Replacement (Resolved) High

Affecting Server - Hermes

11/04/2019 16:43
Last Updated 11/05/2019 15:29

We have been seeing flaky performance from the Hermes server over the past 48 hours which has resulted in a couple of unscheduled reboots.

The server is now (09:15 EST) being taken down to replace the faulty RAM stick.

Update: This has now been completed and the server is back online. We will keep a close eye on the server to ensure it remains stable and the problem has been 100% resolved.

NYC Network Issue (Resolved) Critical

Affecting System - NYC network

11/05/2019 10:15
Last Updated 11/05/2019 10:36

Our monitoring systems have just notified us that the datacenter we use in NYC lost all network connectivity around 5 minutes ago and as such all servers hosted in this facility are unavailable. This is affecting any companies with servers in the building and not just D9.

We will post more info as an when we get it.

Servers hosted in the UK and in Miami are unaffected.

Update: The network has come back up. Downtime was around 10 minutes. We will try and get more info from the datacentre to find out what caused the network blip.

Update from the DC: There appears to have been a routing engine failure, everything has rolled over to our backup routing engine.

Server Migration (Resolved) Medium

Affecting Server - Hermes

10/31/2019 11:09 - 11/04/2019 16:43
Last Updated 10/31/2019 13:35

We are migrating the "Hermes" server over to new hardware running CentOS 7.

We are pleased to let you know that the Hermes server migration has now been completed and your websites are running on the new hardware.

As we mentioned in our earlier emails, the IP address that your website resides on hasn't changed so there is no action needed on your part.

We would kindly ask you to check your websites to ensure they are working correctly after the migration and to let us know ASAP if you have any issues.

One potential issue you may face could be caused by the new default PHP version we are using on the server. The new server uses PHP 7.2 as the default PHP version whereas the old server was running the old end of life PHP 5.6 as the default version.

Some older scripts may not be compatible with PHP 7.2 and may throw up error messages when you visit your sites. If you find this is the case for your website you can easily switch back to PHP 5.6 by logging into your cPanel and clicking on the "Select PHP Version" icon.

If you are facing issues connecting to emails using an email client (Outlook, Thunderbird, etc) then please try removing and then re-adding the profile and this should resolve your issue.

Many thanks for your patience throughout the migration and please do let us know should you have any questions or concerns.rs.

MySQL Issues (Resolved) High

Affecting Server - Hermes

10/29/2019 09:54 - 10/31/2019 09:38
Last Updated 10/29/2019 10:15

We are seeing issues with the MySQL service on the Hermes server, as such websites will currently be showing a MySQL related error message. We are aware of the problem and are working on getting things back up and running.

Packet Loss (Resolved) High

Affecting Server - Hyrule

10/17/2019 09:32
Last Updated 10/24/2019 14:31

We are seeing sporadic packet loss on the Hyrule server. We are investigating this and hope to have the issue resolved ASAP.

During the packet loss clients may experience slower than normal loading of websites and emails.

Update: This was resolved and the server and the network has been stable now for the past 6 days, we will however continue to keep a close eye on things

Packet Loss (Resolved) High

Affecting Server - Hyrule

09/09/2019 08:31 - 09/18/2019 09:02
Last Updated 09/10/2019 08:43

We are seeing sporadic packet loss on this server and the datacentre are investigating this with a matter of urgency. Sites and email may see brief outages due to the packet loss.

Update: The issue was caused by the datacentres DDOS protection system. An IP
address on the server was under heavy attack from malicious DNS traffic.
We have identified and blocked the source of the attack and things look
to be back to normal, we will however continue to closely monitor the
server for the next few hours. Many thanks for your patience during this
disruption.

Update - 10th Sept: Unfortunately the DDOS attack from yesterday has returned, although it isn't as large so shouldn't cause as much disruption. We are working with the datacentre to bring things under control.

Network Issue (Resolved) High

Affecting Server - Shakespeare

09/06/2019 16:29
Last Updated 09/07/2019 06:59

The shared server shakespeare.hyliahub.com is down due network issues. We are waiting for the data centre to respond. We will keep post the status of the server.

Update:

The issue with the Shakespeare server has now been resolved.

It took longer than expected to be provided with a KVM connection from our UK datacentre (iomart), hence the prolonged outage.

When we were able to access the server via KVM we found there was an issue with the network service on the server. After doing some research on the issue we found this was caused by a Cloudlinux bug that was found in their software earlier this week and this is what prevented the network from coming up:

https://www.cloudlinux.com/cloudlinux-os-blog/entry/issues-with-cloudlinux-os-7-7-update-on-systems-with-kernel-v3-10-0-862-or-lower

We managed to apply a fix, reboot the server and everything is back online.

We believe this has fully resolved the issue but will continue to monitor things for the next few hours to ensure everything remains stable.

Many thanks for your patience during the outage, it is much appreciated.

Disk Swap (Resolved) Medium

Affecting Server - Kipling

07/04/2019 02:00 - 07/22/2019 13:13
Last Updated 07/03/2019 08:28

The Kipling server has a failed drive in the RAID array. We will be taking the server down at 2am EST on Thursday 4th July to replace the failed disk.

Network Upgrades (Resolved) High

Affecting System - NYC Network

06/29/2019 09:00 - 07/02/2019 15:56
Last Updated 06/29/2019 11:20

A number of client Dedicated Servers and the shared server Orwell will be offline for periods on the 29th June due to a network upgrade being carried out in our NYC data center. All affected clients have already been notified via email, but you can find full details of the migration below:

Event Summary: Our data center in NYC will be migrating your server to their new network. We do not expect downtime to last the entire maintenance period.

The new network was designed from the ground up for improved reliability through automation and increased redundancy, more affordable bandwidth upgrades, and more DDoS protection at no additional charge.

This is the final part of the network upgrade. No action is needed from your end and the server will remain powered on throughout the process, it will simply be hooked up to a new switch and then we will update the network settings on the server via a KVM interface.

How Much Downtime Will There Be: Although the maintenance window lasts for 12 hours, we have been informed that downtime per server is likely to be around 60 - 180 minutes. As there are thousands of servers being migrated to the new network it isn't possible to say when this downtime will occur during the window, but as soon as we see the server has been removed from the old network, we will be logging into the server and updating the network settings ready for it to come back up on the new network.

Server IP Changes (Resolved) High

Affecting Other - Servers: Goron, Hermes, Kipling, Shaw, Wilde

06/26/2019 09:17 - 06/28/2019 09:15
Last Updated 06/26/2019 09:20

The servers Goron, Hermes, Kipling, Shaw and Wilde will be undergoing some maintenance over the next 2 days. This is the work we originally had penciled in for Saturday that we emailed clients about this on the 25th.

We have been able to bring this forward so we will be doing it ASAP. Doing the work now will mean we can run the new IP addresses on a 2nd NIC in conjunction with the old, this will result in a small period of downtime but will mean we can avoid the prolonged downtime had we been forced to wait until Saturday. Clients will be emailed the new IP as soon as migration is completed.

We attempted to do this on Goron yesterday afternoon/evening (25th) but the data center assigned an incorrect new IP block to the servers meaning we couldn't bring them up and this resulted in 2 - 3 hours downtime. We are currently waiting on new (correct!) IP addresses to be assigned and brought up at which point we can move sites over to the new IP addresses.

Many thanks for your continued patience on this, it is much appreciated.

Server IP Change (Resolved) High

Affecting Server - Hermes

06/25/2019 16:00 - 06/26/2019 09:17
Last Updated 06/25/2019 16:00

The Hermes server will be undergoing some maintenance over the next few hours. This is the work we originally had penciled in for Saturday that we emailed clients about this morning.

We have been able to bring this forward so we will be doing it ASAP. Doing the work now will result in a small period of downtime but will mean we can avoid the prolonged downtime had we been forced to wait until Saturday. Clients will be emailed the new IP as soon as migration is completed.

Server IP Change (Resolved) High

06/25/2019 14:01 - 06/26/2019 09:17
Last Updated 06/25/2019 14:09

The Goron server will be undergoing some maintenance over the next few hours. This is the work we originally had penciled in for Saturday that we emailed clients about this morning.

We have been able to bring this forward so we will be doing it ASAP. Doing the work now will result in a small period of downtime but will mean we can avoid the prolonged downtime had we been forced to wait until Saturday. Clients will be emailed the new IP as soon as migration is completed.

Scheduled Maintenance - June 11th, 2019 (Resolved) High

Affecting Server - Bullseye

06/11/2019 04:30 - 06/14/2019 08:15
Last Updated 06/11/2019 09:31

Event Summary: A network upgrade will require a change of IP address. For the majority of customers no action will be needed but there will be a period of downtime whilst the maintenance takes place. Please review the details for more information.

Affected Server: Bullseye

Maintenance Date: Tuesday 11th June

More Details: The data center we use in NYC have recently completed the build of a brand new network. The new network was designed from the ground up for improved reliability through automation and increased redundancy, more affordable bandwidth upgrades, and more DDoS protection at no additional charge.

The data center are now in the process of migrating all servers over to the new network. For 95% of servers this is a simple case of hooking up the server to a new switch, but for a small number of servers an IP address change is required as the current IP addresses are owned by a 3rd party rather than the data center directly, so cannot be moved over to the new network.

Do I need to do anything?: If you are using D9 Hosting nameservers for your domains, then no, no action will be needed on your part. D9 Hosting will take care of everything for you.

If you are using a 3rd party DNS service or firewall (Eg: Cloudflare or Sucuri) you will need to update the A records to point to the new IP address once the migration has been completed. If you are unsure how to do this, our technical support team will be happy to help. We will email all clients with the new IP address details as soon as the IP change has been completed.

Reseller and Multi Site clients using custom nameservers will need to update nameserver IP addresses to point to the new IP addresses. We would be happy to do this for you, but we will need you to provide us with your domain registrar login details.

What time will this take place?: We are unable to give an exact time frame as some of the work needs to be done by the data center, but we aim to start the work at around 04:30 am EST with the aim to have everything completed by 09:30 am EST, but this time frame does depend on the data center.

Will there be downtime?: Yes. There will be a period of downtime whilst we configure a 2nd NIC on the server to use the new IP block and then make network changes to the server via a KVM connection. We will then need to configure all sites on the server to use the new IP addresses and update all server licenses to use the new IP. When this has been done all nameserver IP addresses will be updated so sites resolve from the new IP addresses. There may also be some additional downtime whilst propagation takes place but this will depend on the users local ISP.

Scheduled Maintenance - June 4th, 2019 - IP A (Resolved) High

Affecting Server - Hyrule

06/04/2019 04:30 - 06/04/2019 13:16
Last Updated 06/04/2019 13:16

Update: THIS HAS NOW BEEN COMPLETED

Event Summary: A network upgrade will require a change of IP address. For the majority of customers no action will be needed but there will be a period of downtime whilst the maintenance takes place. Please review the details for more information.

Affected Server: Hyrule

Maintenance Date: Tuesday 4th June

More Details: The data center we use in NYC have recently completed the build of a brand new network. The new network was designed from the ground up for improved reliability through automation and increased redundancy, more affordable bandwidth upgrades, and more DDoS protection at no additional charge.

The data center are now in the process of migrating all servers over to the new network. For 95% of servers this is a simple case of hooking up the server to a new switch, but for a small number of servers an IP address change is required as the current IP addresses are owned by a 3rd party rather than the data center directly, so cannot be moved over to the new network.

Do I need to do anything?: If you are using D9 Hosting nameservers for your domains, then no, no action will be needed on your part. D9 Hosting will take care of everything for you.

If you are using a 3rd party DNS service or firewall (Eg: Cloudflare or Sucuri) you will need to update the A records to point to the new IP address once the migration has been completed. If you are unsure how to do this, our technical support team will be happy to help. We will email all clients with the new IP address details as soon as the IP change has been completed.

Reseller and Multi Site clients using custom nameservers will need to update nameserver IP addresses to point to the new IP addresses. We would be happy to do this for you, but we will need you to provide us with your domain registrar login details.

What time will this take place?: We are unable to give an exact time frame as some of the work needs to be done by the data center, but we aim to start the work at around 04:30 am EST with the aim to have everything completed by 09:30 am EST, but this time frame does depend on the data center.

Will there be downtime?: Yes. There will be a period of downtime whilst we configure a 2nd NIC on the server to use the new IP block and then make network changes to the server via a KVM connection. We will then need to configure all sites on the server to use the new IP addresses and update all server licenses to use the new IP. When this has been done all nameserver IP addresses will be updated so sites resolve from the new IP addresses. There may also be some additional downtime whilst propagation takes place but this will depend on the users local ISP.

Network Issue (Resolved) High

Affecting Server - Virtualizor

04/25/2019 13:30
Last Updated 04/25/2019 13:36

The VPS node "Kraken" has stopped responding to ping, we are however able to access the server via the IPMI connection and it's up and running fine, so it appears to be a network issue.

We have contacted the datacentre about this. Users with VPS on the Kraken server will be unable to access their VM's at this time.

Update: This has now been resolved.

Wordpress DDOS Attack (Resolved) High

Affecting Server - Shakespeare

04/15/2019 08:54
Last Updated 04/16/2019 15:13

We are seeing a huge number of IP addresses attempting to access wp-login.php and xmlrpc.php files across the server and this is causing intermittent website outages. We are currently working on getting this resolved ASAP.

Update: We have discovered the high number of connections is actually an underlying symptom of an InnoDB MySQL database corruption and this has been causing the intermittent issues. We are currently working on a permanent fix for this. There may well be periods of MySQL unavailability whilst this work is carried out.

Update: We have finished repairing the InnoDB database tables, and have also moved the MySQl data directory to a new partition with more free space. We have also increased the space of the /tmp partition. All sites are now up and running and this should solve the problem going forward. There is just a small tweak needed to the CageFS settings which may cause a few minutes of MySQL outages but we plan to do this during off peak hours.

Update: We just need to stop MySQL for a few minutes to complete this maintainence and we are proceeding with this now. (15.20 UK time)

Update: This has now been fully resolved. Please contact support if you have any issues.

Server Unresponsive (Resolved) High

Affecting Server - Hyrule

03/12/2019 16:25
Last Updated 03/12/2019 16:39

The Hyrule server has gone unresponsive. We have just issued a reboot request.

Update: Server back online after a reboot

Server Maintainence (Resolved) High

Affecting Server - Bullseye

01/31/2019 16:00
Last Updated 02/01/2019 18:48

Update 5:

The IP switch was completed at approx 10.45am EST and websites came back online shortly afterwards. There are a few small tasks still pending that we need to complete but these should not affect websites. If you continue to have issues then please don't hesitate to open a technical support ticket with the details and we will be more than happy to help.

Thanks for all of your patience during the extended maintainence window, it's much appreciated.

Update 4:

The data sync has now been completed. We are now just waiting for the onsite USA datacenter technicians to swap the IPs from the old server to the new, at which point things will be back online.

Update 3:

The data transfer from the faulty server to the new server is progressing well and as long as the faulty server remains stable it should be completed by 9am - 10am EST today (Feb 1st). Once the data migration has been completed we can then have the onsite USA datacenter technicians move the IP blocks from the faulty server to the new, at which points sites will start to load again.

Email has continued to be up throughout the migration, although you may see a brief period of email unavailability during the IP switch over.

So all in all, we would hope to have sites back online by midday EST, although it does depend on how quickly the datacenter are able to carry out our IP address migration request.

Update 2:

We have made good progress in the past few hours. We've been able to stabalise the faulty server by disabling the HTTP service. Whilst this does mean that websites are unavailable, it's kept the email services online and also allowed us to start migrating data off the faulty server and onto the new server without having to pull accounts from the backup server.

At the time of writing, the main bottleneck with getting things back to normal is going to be when the USA datacenter technicians move the IP addresses from the old server to the new. Unfortunately this is a specialist task that they can only do between the hours of 9am and 5pm EST.

We hope we will have the data migrated onto the new server by 9am EST, and will then be requesting the IP address switch.

Whilst it's still early days, if things keep progressing as they are, we would hope to have websites back online by tomorrow morning EST, although this does depend on the faulty server continuing to play ball with us!

We will continue to work throughout the night and will post more updates in the early hours of Feb 1st EST.

Update 1:

Our monitoring systems have picked up a potential issue with the RAID card on the server, and as such we have had to take the server down to check this.

A RAID card failure, at worst, can cause complete data loss.

Whilst this hasn't occurred yet, the server is showing signs that it could be about to give up the ghost completely and as such we have a stand by server currently being set up for deployment.

Once the server is ready we will then be migrating websites off the old failing server and onto the new hardware. We don't yet know if we will be able to keep the old server stable enough to pull the data directly off there and onto the new server or if we will have to restore from backups.

In the meantime we are trying to keep the faulty server as stable as possible, but this may mean disabling all but core services (Email) or taking the server offline completely.

We don't have an ETA on when the migration will be completed as it depends on how much data there is to migrate but based on past experience of a full server restore, it could be anything from 24 - 48 hours but we will know more in the coming hours.

We will continue to post updates on our server status page as and when we have them:

https://d9clients.com/serverstatus.php

Whilst we appreciate that having websites offline for an extended period is a massive inconvenience, please rest assured we are doing all we physically can to get things back up and running as soon as possible.

If you would prefer not to wait for our restoration process to complete, and if you have your own cPanel generated website backup available. Please let us know and we can restore the backup for you onto a different server.

{$signature}

Server Overloaded (Resolved) High

Affecting Server - Bullseye

01/15/2019 14:36
Last Updated 01/15/2019 15:00

We have received alerts that the server is overloaded and some services are inaccessible.

We are checking it now.

Update: The server is now back online and stable.

DDOS Attack (Resolved) High

Affecting System - Shaw, Wilde, Goron & Hermes Servers

12/10/2018 12:28
Last Updated 12/10/2018 14:33

We are seeing sporadic DDOS attacks on the following servers:

- Shaw

- Wilde

- Hermes

- Goron

We are aware of the issue and are working on blocking offending IPs to keep sites online. There will however be periods when sites load much slower than usual depending on the size of the incoming attacks, but we are doing our best to keep things stable.

Update: Everything has now gone back to normal. Please open a ticket if you continue to have issues.

Drive Replacement (Resolved) High

Affecting Server - Shaw

09/25/2018 08:30 - 10/02/2018 14:25
Last Updated 09/25/2018 14:51

We will be taking the "Shaw" server down at 6am EST on the 25th September to replace a failed hard drive. Downtime should be around 30 - 60 mins.

Update: This has had to be pushed back to 8.30am EST due to the limited availability of datacenter technicians at the original time.

Update: This is now in progress

Update: We are requesting an update from the datacenter as the drive replacement is taking longer than expected.

Update: The faulty disk has been replaced with a new one. Unfortunately, the server was getting stuck in the BIOS on boot so we are performing a BIOS update in an attempt to resolve the issue.

Update: Thank you for your patience. The server is back online, We have replaced the board of the server with a new once hence all the network configurations were lost. We have configured the IPs on the new network interface and the server is now accessible and sites are back up.

Please be informed that the RAID rebuilding is in progress, it's currently at 58% and should be completed in the next couple of hours. The server may be slightly sluggish during this time.

Drive Replacement (Resolved) High

09/11/2018 23:30 - 09/05/2018 00:30
Last Updated 09/07/2018 10:12

Our monitoring systems have found a faulty hard drive in the RAID array. We will be taking the server down at 11.30 pm EST on Tuesday to replace the drive.

Update: This is taking longer than originally expected and we are awaiting an update from the datacenter as to what the issue is.

Update: We have tried to replace the faulty disk and could see that the disk on the port p0 also seems to be faulty. The server is not booting up after the reboot. We have tried to rebuild the array but it doesn't look like it is going to be possible based on the 2 disks that have failed. As such we are now reinstalling the OS on the server, we will then proceed to install cPanel and then start the restoration of accounts from the most recent backup (3rd Sept).

In terms of a timescale we estimate it will take around 2 - 3 hours for the OS installation, cPanel installation and hardening. Accounts will then be restored from the backup server in batches and will start to come up again as soon as they have been restored. Based on the amount of data to transfer we would expect to have all accounts back online within 24 hours, although many will be sooner depending on the order in which they are restored.

Update: cPanel is currently being installed and hardened. We estimate the restoration will start within the next 90 - 120 minutes at which point the first batch of accounts should start coming up shortly thereafter.

Update: The 1st batch of backups is currently being pulled from the backup server and restored.

Completed Restores:

===============

Restoration of accounts starting with "A" is completed.

Restoration of accounts starting with “B” is completed.

Restoration of accounts starting with “C” is completed.

Restoration of accounts starting with “E, F” is completed.

Restoration of accounts starting with “G, H” is completed

Restoration of accounts starting with “I” is completed

Restoration of accounts starting with “J, K, L” is completed.

Restoration of accounts starting with "N" completed.

Restoration of accounts starting with “P, Q” is completed.

Restoration of accounts starting with "D" is completed

Restoration of accounts starting with “O” is completed

Restoration of accounts starting with “M” is in progress

Ongoing Restores:

==============

Update: 98% of accounts have now been restored. 2 more accounts pending.

Update: All restores now completed as of around 2 hours ago. We are just going through and verifying things. We will be emailing all clients shortly with more information.

Update: All accounts has been restored.

Drive Replacement (Resolved) High

Affecting Server - Bullseye

08/20/2018 23:00 - 08/20/2018 02:00
Last Updated 08/22/2018 08:20

Our monitoring systems have found a faulty hard drive in the RAID array. We will be taking the server down at 1am EDT to replace the drive.

Update: We have had to push this back as the DC didn't receive a drive shipment that they were expecting. We will update this post again once we receive confirmation of the date.

Update: This has been scheduled for Monday 20th August, 11pm EST.

Update: This has now been completed.

Server Maintenance (Resolved) High

Affecting Server - Hyrule

06/14/2018 10:52 - 07/19/2018 10:19
Last Updated 06/14/2018 15:00

We will shortly be taking the server "Hyrule" down for some maintenance as we are seeing some sporadic packet loss on the server. We need to perform a live OS test to see if the issue is server or network related. Downtime is expected to be around 30 minutes.

Update: Server is back up and the datacentre are working on the sporadic packet loss issue

Update: We are still working on a few niggling issues with the server, you may experience brief moments of downtime (usually 5 - 10 minutes) and we are also aware of the server throwing up cPanel license errors from time to time but we are working on bringing the server back to its usual stable self!

Hardware Migration (Resolved) High

Affecting Server - Shakespeare

06/04/2018 13:46
Last Updated 06/06/2018 08:46

Update: This has now been completed

As per our email this past Friday we are in the process of migrating the Shakespeare server over to new hardware.

The initial migration has been completed and websites are now loading from the new hardware.

There is one outstanding issue which is causing certificate errors when logging into cPanel and this will be resolved in the early hours of the 5th June when the final IP switch takes place.

Update: The final IP switch has taken place but we are seeing an issue with some mail clients reporting "unsupported SSL protocols" (or similar) error messages. We are aware of this and are working on it. In the meantime you can still access emails via Webmail:

https://shakespeare.hyliahub.com/webmail

Server Down (Resolved) High

Affecting Server - Shakespeare

05/30/2018 08:39 - 06/01/2018 10:56
Last Updated 05/30/2018 09:12

The Shakespeare server is currently down and not responding to Ping. The datacenter are currently setting up a KVM session so we can investigate. We will post more details as soon as we have them.

Update: The server is now back online.

bullseye.d9hosting.com (108.61.50.138) down (Resolved) Medium

Affecting Server - Bullseye

03/11/2018 10:13 - 03/11/2018 16:33
Last Updated 03/11/2018 16:33

The server is online now, we have checked the websites and the websites are loading as well.

Server Down (Resolved) Critical

Affecting Server - Shakespeare

02/09/2018 16:47
Last Updated 02/09/2018 17:17

Update: The server has come back up after a reboot.

The Shakespeare server has gone unresponsive. We are waiting on the DC to hook up a KVM unit for us so we can investigate further.

UK Network Upgrade (Resolved) Medium

Affecting System - UK Datacenter

02/07/2018 23:00 - 02/09/2018 10:49
Last Updated 02/07/2018 09:29

Our UK network provider will be carrying out scheduled network upgrades from 23:00 to 06:00 starting on the 7th Feb.

There is expected to be around 20 minutes of network downtime during this period affecting all UK servers.

This maintenance is the final piece of work required to fully migrate the network into a brand new Cisco platform which has been a large ongoing operation between the datacenter and Cisco.

Bullseye will be taken down for maintenance (Resolved) Medium

Affecting Server - Bullseye

12/31/2017 12:01 - 12/31/2017 00:00
Last Updated 12/31/2017 12:42

The server will be taken down for maintenance, we will inform once the maintenance work gets completed.

Update: THis has now been completed and the server is back online.

Packet Loss (Resolved) Critical

Affecting System - PNJ Switch

11/06/2017 09:47 - 11/23/2017 16:03
Last Updated 11/06/2017 09:48

We are seeing an issue with a switch at the NJ DC. Everything is up but there is sporadic packet loss. We have reported the issue to the datacenter but you may notice slower than usual performance whilst the issue is resolved.

Affected servers are:

bronte
wilde
hermes
kipling
shaw
goron

Wilde emergency maintenance (Resolved) Critical

Affecting Server - Wilde

10/05/2017 07:12
Last Updated 10/05/2017 14:42

We are aware of the ongoing issue with wilde server. We are working on this to resolve the issue as soon as possible.

Update: We have tested the RAM but the sticks are fine. We are now running long smart tests on all drives in the server but will try keep the server as stable as possible during this time.

Update: We will be taking the server down in the next 20 minutes to replace a hard drive in the server and then proceeding to rebuild the RAID array.

Update: The drive has been replaced and the RAID rebuild is currently in progress. We have stopped MySQL on the server to keep the load down so the rebuild will complete quicker.

Update: We have managed to stabilize the server but are still seeing issues with the RAID array which we continue to work on. We do also have a new server up and ready to migrate data to should we need to but we will continue to work on the server for the next few hours and if any further work is going to occur that will incur excessive downtime we will send out an email to affected customers.

Update: The issue has been resolved. We finally tracked it down to a rogue mod_security rule that was rolled out by comodo in an automatic update!

Drive Replacement (Resolved) Medium

Affecting Server - Kipling

09/29/2017 00:00 - 09/29/2017 14:17
Last Updated 09/29/2017 14:17

We will be taking the server down to replace a failed hard drive at midnight on the 29th Sept. Downtime is expected to be 60 - 120 mins.

Update: This was completed. Downtime was around 20 mins.

RAM Upgrade (Resolved) Medium

Affecting Server - Shaw

09/26/2017 01:00 - 09/26/2017 08:23
Last Updated 09/25/2017 10:03

We will be taking the Shaw server down at 1am EST on the 26th Sept to add more RAM to the server. Downtime is expected to be 30 - 60 mins.

RAM Upgrade (Resolved) Medium

Affecting Server - Hermes

09/21/2017 01:00
Last Updated 09/21/2017 08:09

We will be taking the Hermes server down for a RAM upgrade at 1am EST on the 21st September. Downtime is expected to be 30 - 60 mins.

Update: This has now been completed

Multiple IP Blocks Down (Resolved) Critical

Affecting System - US Datacenter

08/25/2017 16:57
Last Updated 08/25/2017 18:19

There is currently an ongoing issue at the US Datacenter affecting a number of different IP blocks. The blocks are currently going up and down sporadically but the DC is working on the issue.

Clients affected are those with VPS's on the node "Kraken" and also a handful of clients with Dedicated Servers.

Shared and Reseller servers are unaffected so far.

Update: This was caused when the datacenter needed to do an emergency reboot of a switch to solve some BGP issues. This has now been resolved.

Network Maintenance (Resolved) Medium

Affecting Other - US Datacenter

07/12/2017 02:00 - 07/17/2017 12:38
Last Updated 07/10/2017 13:25

This notification is to inform you of scheduled maintenance that may affect your D9 Hosting service. Please review the details of this notification to determine how it will affect your service. (all times US eastern).

Event Type: Network Upgrade
Start Time: 07-12-17 02:00 EST
End Time: 07-12-17 03:00 EST
Location(s): New York City Metro
Device(s): BR1.PNJ1/BR2.PNJ1
Client Affecting: Possible

Event Summary: Important maintenance will be performed on one or more devices that may affect network connectivity to your services. Configuration updates will be applied that may affect traffic into the network in NY/NJ causing higher than normal latency or brief periods of packet loss.

Network Maintenance (Resolved) Medium

Affecting Other - US Datacenter

07/12/2017 02:00 - 07/17/2017 12:52
Last Updated 07/10/2017 13:25

This notification is to inform you of scheduled maintenance that may affect your D9 Hosting service. Please review the details of this notification to determine how it will affect your service. (all times US eastern).

Event Type: Network Upgrade
Start Time: 07-12-17 02:00 EST
End Time: 07-12-17 03:00 EST
Location(s): New York City Metro
Device(s): BR1.PNJ1/BR2.PNJ1
Client Affecting: Possible

Event Summary: Important maintenance will be performed on one or more devices that may affect network connectivity to your services. Configuration updates will be applied that may affect traffic into the network in NY/NJ causing higher than normal latency or brief periods of packet loss.

Packet Loss (Resolved) High

Affecting Server - Wilde

07/10/2017 13:02 - 07/10/2017 13:23
Last Updated 07/10/2017 13:03

We are currently seeing some packet loss on this server which is resulting in websites not resolving for some users. The datacenter are working on the issue for us.

Packet Loss (Resolved) High

Affecting Server - Bullseye

07/03/2017 09:18 - 07/10/2017 13:03
Last Updated 07/03/2017 09:18

We are seeing some packet loss on the server bullseye.d9hosting.com

We are currently investigating the issue.

High Server Load (Resolved) High

Affecting Server - Wilde

06/05/2017 15:12 - 06/06/2017 10:55
Last Updated 06/05/2017 14:13

The Wilde server is experiencing high load for the past few hours, we are currently aware of the issue and investigating to find the cause of the issue.

Server Migration (Resolved) High

Affecting Server - Hyrule

05/30/2017 12:42
Last Updated 05/30/2017 16:57

We will be switching the IP addresses over from the old Hyrule hardware to the new at around 9am EST on Tuesday 30th May, around 1 hour of downtime is expected.

All affected clients have already been emailed about this work.

Update: IP change has been completed, we are now stopping MySQL on both servers and doing a final sync of MySQL to esnure data consistency

Update: This has been completed

Issue with BBU on Hyrule server (Resolved) High

Affecting Server - Hyrule

05/23/2017 12:51 - 05/30/2017 12:42
Last Updated 05/24/2017 08:44

There is currently an issue on the hyrule server with the BBU-battery backup unit and this is causing high load on the server.

Our admins are currently aware of the issue and are working on the server. We are attempting to replace the batterie or raid card with the data center and there will be a server down time for the next 45 to 60 minutes. We will let you know when the problem is resolved.

Update: The datacenter are still working on this and are testing the BBU to see if it's charging at all, if not it will either need to be replaced or worse case scenario we would need to replace the RAID card but this would require a full data restore so we are hoping to avoid that.

Update: Server has been brought back online whilst we run additional tests on the BBU. We will check the BBU status again at 9pm EST when the BBU should've fully charged and we can see if it is holding the charge or still failing, at which point we can decide how to proceed.

Update: Server is back online but we are seeing issues with MySQL which we are currently working on

Update: MySQL issue has been fixed. Although due to the BBU issue there will be sub-optimum website performance until the issue is resolved. We will have more info in around 12 hours time once the tests on the battery have completed.

Update: The battery unit on the RAID controller is completely dead, so rather than swap out the RAID controller and risk any data loss and having to restore from backups, we are putting up a new box and will be migrating all data over to the new machine. We will be emailed all clients shortly with more info.

High Server Load (Resolved) High

Affecting Server - Hyrule

05/23/2017 09:09 - 05/23/2017 14:00
Last Updated 05/23/2017 09:10

We are seeing abnormally high load on the Hyrule server that is causing very slow website performance. We are monitoring and working on the server and hope to have the issue resolved ASAP.

Web Server Upgrades (Resolved) Medium

Affecting Other - All Servers

04/19/2017 08:51 - 04/26/2017 15:25
Last Updated 04/26/2017 15:25

We will be migrating from EasyApache 3 to EasyApache 4 on all Shared and Reseller Servers from the 19th April onwards.

Each server will be migrated one by one, and there will be around 30 minutes of website downtime per server.

The updates will all be done during off peak hours (between the hours of 1am and 6am)

Currently Completed:

Shaw

Hermes

Shakespeare

Kipling

Hyrule

Bullseye

Goron

Wilde

Baracus

Fleetwood

Nash

UK VPS Network Issue (Resolved) High

Affecting Other - IP Range

04/24/2017 13:37 - 04/25/2017 10:55
Last Updated 04/24/2017 15:11

We are seeing a network issue for VPS's on the IP range 185.3.166.*

The issue is only affecting 5 VPS's but we have escalted the issue to the datacentre and they are working on it.

Update: We have just received an update from the datacentre, which is copied below. We have seen a couple of IP's come back up for a short period then go down again, so hopefully it wont be too long before the issue is resolved:

"We are still working on an issue with the compute layer of a small section of our infrastructure. Though many of you have your services back online, we are still working hard to bring back those clients with offline services."

Server Maintenance (Multiple Servers) (Resolved) Medium

Affecting Other - All Shared and Reseller Servers

03/29/2017 14:32 - 03/30/2017 14:21
Last Updated 03/29/2017 14:34

We are upgrading Apache on all Shared and Reseller servers on the 30th and 31st of March during off-peak hours. We don't expect any disruption but in the unlikely event there is any downtime, this notice is to make clients know we are aware of the issue and will be working on it.

Server Maintenance (Multiple Servers) (Resolved) High

Affecting System - Multiple Servers

03/20/2017 01:00 - 03/21/2017 04:00
Last Updated 03/21/2017 09:51

We will be carrying out scheduled maintenance on the following servers on the dates and times listed below. You can find details of this maintenance and how it will affect you below.

Servers: Goron, Hermes & Hyrule [Completed]

When: Monday 20th March @ 01.00 EST

Servers: Kipling, Shaw & Wilde [Completed]

When: Tuesday 21st March @ 01.00 EST

Expected Downtime: 2 to 3 hours

What: We will be taking the servers down to add an additional SSD RAID array to it. This will allow us to improve the overall performance of the server by offloading the MySQL service to the new SSD array.

All clients with accounts that reside on these servers have already been notified via email.

Server Migration (Resolved) Medium

Affecting Server - Bullseye

01/26/2017 10:00 - 01/27/2017 10:24
Last Updated 01/27/2017 10:24

As per our previous email notifications, we will be migrating the Bullseye server over to new hardware on the 26th January.

We will be migrating the IP addresses from the old hardware to the new at around 10am EST on the 26th January, around 1 hours downtime is expected.

Update: This is now in progress.

Update: IP's migrated but we need to do a final MySQL sync, so at the present time it looks like we'll be running approx 1 hour behind the scheduled window.

Update: Migration completed

UK VPS Network Issues (Resolved) Critical

Affecting System - All UK VPS's

11/15/2016 08:57
Last Updated 11/16/2016 09:01

There is currently a network issue affecting all UK VPS's. We are aware of it and the DC are working on fixing the issue.

Update: This is due to a power outage at the Namesco/Everest Datacentre in Reading, UK. They are currently working to get power restored in the building and we will update you as and when we get any updates.

Update: Power has now been restored and the datacentre team are now working on fixing network issues that have been caused by the power outage.

Update: As of 14:40 the datacentre are still working on bringing all affected services back online.

Update: VPS's are now slowly coming back online but there are a lot of machines to bring back up so please hold tight whilst everything continues to be brough back online.

Update: [17:22] We are still waiting on our provider in the datacentre to get the majority of services back online. We've seen a small number of IP's come back up but the majority are offline. We aren't able to get any concrete information as to why this is but we will continue trying and post any updates as soon as we get them.

Update: All servers are now back online and we will be emailing a full RFO to all clients as soon as we receive one from the datacenter.

Kernel Updates (Resolved) High

Affecting System - All Servers

10/25/2016 08:42 - 11/07/2016 20:03
Last Updated 10/27/2016 08:44

We will be updating all servers to patch against the "Dirty Cow" kernel vulnerability over the following days:

25th October
=========

Now - bullseye [completed]
Now - goron [completed]

26th October
==========

1am EST - Hermes [completed]
2am EST - Hyrule [completed]
3am EST - Kipling [completed]

2am GMT - Baracus [completed]
3am GMT - Fleetwood [completed]
4am GMT - Nash [completed]

27th October
=============

1am EST - Kipling [completed]
2am EST - Shaw [completed]
3am EST - Wilde [completed]

28th October Onwards
===============

All clients Dedicated Servers & VPS. Clients will receive an email advising of upgrade times.

Downtime will likely be 15 - 30 minutes per server.

VPS Node Migrations (Resolved) High

Affecting System - Old VPS Node

10/20/2016 09:00
Last Updated 10/20/2016 16:07

We are currently migrating all VPS's running on old CentOS 5 nodes over to new nodes. We expect this to be completed by 13:00 EST on the 20th October.

If you are going to be impacted by this you will have already received an email giving you more details on the 19th October.

Network Issues Incident Report (Resolved) Low

09/22/2016 08:43
Last Updated 09/30/2016 09:52

Event Type: Network Outage

Location(s): Piscataway, NJ

Event Type: Network Outage

Location(s): Piscataway, NJ

Event Summary: A large portion of the Data Center's IP space was targeted by an attack exceeding 500 Gbps in size that caused several major upstream links to be completely saturated.

The DC made several immediate changes to the network infrastructure that prevented the attack from causing complete unavailability.

While the current infrastructure can handle the attacks as a whole, the attack traffic did not come in through balanced routes, causing specific uplinks to be completely saturated.

The DC have now started the process of upgrading uplinks to all upstream providers to multiple 100 Gbps links, which will allow them to completely absorb the attack without saturation in the near future. This permanent solution is expected to be in place soon (date TBA as they are still coordinating with upstream providers) and is considered to be the highest priority project to ensure long term availability and quality of service

Network Issues on a number of Servers (Resolved) Critical

Affecting Other - goron, hermes, hyrule, kipling, shaw, wilde, keats, kraken, bronte

09/20/2016 09:53
Last Updated 09/21/2016 09:45

Update: From the US datacenter:

"Our network gear was targeted by a sophisticated attack, causing significant packet loss across a great portion of our NYC Metro network. The attack was manually mitigated and we're currently working on a permanent solution to prevent this sort of attack automatically

There was a network failure earlier today on the above servers. Our datacenter in the US was contacted and we have been told that this issue has now been fixed. We have not been told as yet what caused the issue, whether it was a network issue or there was some kind of attack. We will keep this updated as and when we get information. Please open a tech support ticket if you are still experiencing any issues.

MySQL Upgrades (Resolved) Medium

Affecting Other - All Shared and Reseller Servers

08/04/2016 08:52
Last Updated 08/08/2016 08:25

We will be performing MySQL upgrades on all Shared and Reseller servers over the coming days.

During the upgrade window there may be slower than usual performance and also brief moments of downtime for MySQL driven websites.

You can find the upgrade schedule below:

4th August - 4am EST (US East Coast)
=========================

Bronte [completed]
Bullseye [completed]

5th August - 1am GMT (UK Time)
===============================

Baracus [completed]
Fleetwood [completed]

5th August - 1am EST (US East Coast)
===================================

Goron [completed]
Hermes [completed]

6th August - 1am GMT (UK Time)
================================

Nash [completed]

6th August - 1am EST (US East Coast)
===================================

Hyrule [completed]
Kipling [completed]

7th August - 1am EST (US East Coast)
===================================

Shaw [completed]
Wilde [completed]

Sporadic Packet Loss (Resolved) High

Affecting System - US Network

07/01/2016 12:49 - 07/05/2016 12:30
Last Updated 07/01/2016 12:50

We have been seeing packet loss at present with the Choopa datacentre and all networks contained within it (our servers are on a network within the Choopa building). It's been going on for a couple of days now and we are waiting to hear about a full resolution. I must stress this isn't something we have any direct control over, since it is an issue with the entire facility rather than something specific to the D9 servers.

Servers Down (Resolved) Critical

Affecting Other - Multiple Servers

06/14/2016 14:43
Last Updated 06/14/2016 14:48

We currently have 4 servers down due to an automated system failure on behalf of our upstream providers, the servers should be back up shortly.

Update: This has now been resolved

Drive Replacement (Resolved) Medium

Affecting Server - Wilde

05/25/2016 01:00
Last Updated 05/25/2016 13:59

One of the hard drives in the RAID array has failed, so we are taking the server down at 1am EST on the 25th May to replace the drive. Downtime shouldn't be more than 30 - 60 minutes.

Update: The drive has been replaced and RAID is currently rebuilding

RAM Replacement (Resolved) Medium

Affecting Server - Bullseye

03/31/2016 02:00 - 04/14/2016 16:07
Last Updated 03/30/2016 13:19

Maintenance Notice:

The "Bullseye" Server will be taken down at 2 am EST on the 31st March to replace a couple of RAM sticks. Downtime shouldn't be more than 1 - 2 hours.

RAM Upgrade (Resolved) Medium

02/27/2016 02:00 - 02/27/2016 03:00
Last Updated 02/27/2016 11:54

We will be taking this server down at 2am EST on the 27th Feb so add some additional RAM to the server. Downtime is likely to be around 15 - 30 minutes.

Update: This has now been completed

Server Maintenance (Resolved) Medium

Affecting Server - Hyrule

01/28/2016 14:39
Last Updated 01/28/2016 14:54

We are currently stopping the MySQL and Apache services on this server so that we can re-mount the /home parition with quota support to fix an on-going issue with regards the reporting of disk space usage on this server. Services will be down for around 10 minutes.

Update: This has been completed and services back online

Network Issue (Resolved) High

Affecting System - Multiple Servers

10/07/2015 16:03 - 10/07/2015 17:40
Last Updated 10/07/2015 16:04

There appears to be a local network issue affecting multiple servers in our US datacenter, we have reported the issue and the DC techs are checking it.

Server Unresponsive (Resolved) High

Affecting Server - Hermes

08/24/2015 13:32
Last Updated 08/24/2015 14:07

Server has gone unresponsive, we are checking it.

Update: Server is back online after a reboot

Hyrule Server Load (Resolved) High

Affecting Server - Hyrule

06/22/2015 11:54 - 06/24/2015 13:39
Last Updated 06/22/2015 11:55

We have found a resource heavy user on the server and are currently moving this account over to a Dedicated Server. There are multiple accounts to move and whilst we rsync these over to a new Dedicated Server load will be high and you may experience sub-optimal performance when loading websites. We are working on getting the resource heavy accounts off the server ASAP whilst keeping the load down to a minimum to allow other sites to load.

Your patience is appreciated and performance will return to normal once these accounts are moved off the server.

Kernel Upgrades (Resolved) High

Affecting System - Multiple Servers

06/19/2015 23:00 - 06/22/2015 11:54
Last Updated 06/19/2015 08:17

Maintenance Details: Kernel Security Patch
Start & End Date: 19th June 23:00 - 20th June 06:00 EST
Expected Downtime: < 1 hour per server

We will be patching the following Dedicated Servers against the recently discovered CVE-2015-1805 Linux Kernel bug that if exploited, could allow an unprivliged user root access to the server.

Once we have applied the patch, the Dedicated Server will be rebooted to complete the maintenence. Downtime of no more than 1 hour is expected for each server:

Kipling
Shaw
Wilde

We will also be patching the following UK Dedicated Servers at 01:00 GMT on the 20th June:

Baracus
Fleetwood
Nash

You can view the full details of the bug that we are patching against here:

https://access.redhat.com/security/cve/CVE-2015-1805

Kernel Upgrades (Resolved) High

Affecting System - Multiple Servers

06/18/2015 23:00 - 06/19/2015 06:00
Last Updated 06/19/2015 08:15

Maintenance Details: Kernel Security Patch
Start & End Date: 18th June 23:00 - 19th June 06:00 EST
Expected Downtime: < 1 hour per server

We will be patching the following Dedicated Servers against the recently discovered CVE-2015-1805 Linux Kernel bug that if exploited, could allow an unprivliged user root access to the server.

Once we have applied the patch, the Dedicated Server will be rebooted to complete the maintenence. Downtime of no more than 1 hour is expected for each server:

Bronte
Bullseye
Goron
Hermes
Hyrule

We will also be applying the same patch to all UK VPS's at 1am GMT on the 19th June, again, only minimal downtime is expected.

You can view the full details of the bug that we are patching against here:

https://access.redhat.com/security/cve/CVE-2015-1805

Dedicated Server Kernel Pathces (Resolved) High

Affecting Other - Clients Dedicated Servers

06/17/2015 23:00 - 06/18/2015 08:21
Last Updated 06/18/2015 08:20

Maintenance Details: Kernel Security Patch
Start & End Date: 17th June 23:00 - 18th June 06:00 EST
Expected Downtime: < 1 hour per server

We will be patching all clients Dedicated Servers against the recently discovered CVE-2015-1805 Linux Kernel bug that if exploited, could allow an unprivliged user root access to the server.

Once we have applied the patch, the Dedicated Server will be rebooted to complete the maintenence. Downtime of no more than 1 hour is expected for each server.

All clients with UK Dedicated Servers please note we will start with the Kernel upgrades at 02:00 GMT, not EST.

You can view the full details of the bug that we are patching against here:

https://access.redhat.com/security/cve/CVE-2015-1805

Update: This has been completed and all servers are back online

VPS Kernel Security Updates (Resolved) High

Affecting System - All OpenVZ VPS's

06/16/2015 23:00 - 06/17/2015 06:00
Last Updated 06/17/2015 08:08

Maintenance Details: Kernel Security Patch
Start & End Date: 16th June 23:00 - 17th June 06:00 EST
Expected Downtime: 2 Hours

We will be patching all VPS's against the recently discovered CVE-2015-1805 Linux Kernel bug that if exploited, could allow an unprivliged user root access to the server.

Once we have applied the patch, the hardware node will be rebooted to complete the maintenence at which point a each VPS will start to come back online. Downtime of no more than 2 hours is expected.

You can view the full details of the bug that we are patching against here:

https://access.redhat.com/security/cve/CVE-2015-1805

Update: This has now been completed and all VPS's are back online

Bullseye server Down (Resolved) Critical

Affecting Server - Bullseye

02/23/2015 03:05 - 03/02/2015 14:27
Last Updated 02/27/2015 03:57

The server is up and back online. We are restoring the accounts now and it will be completed soon.

Restoration of all the accounts Completed'

Few account failed during restoration process. We are going to check the failed accounts now.

We will keep updating here.

We appreciate your patience.

Shaw Server Down (Resolved) Critical

Affecting Server - Shaw

02/23/2015 17:19 - 02/23/2015 21:58
Last Updated 02/23/2015 18:57

We are having network issue on the Shaw Server. We informed the issue on DC and the techs at Data Center is working on the issue.

Network issue with the Shaw server has been resolved.

Server is up and back online.

Server Unresponsive (Resolved) Critical

Affecting Server - Fleetwood

12/09/2014 13:51
Last Updated 12/09/2014 14:00

The server is currently unresponsive. We are checking it.

Update: Server has been rebooted and is back online

Server Down (Resolved) High

Affecting Server - Bullseye

11/14/2014 12:40
Last Updated 11/14/2014 14:17

The server has gone unresponsive, we are checking it.

Update: The server is back up after a reboot and we are monitoring it. Total downtime was around 35 mins.

Network Disruption (Resolved) Medium

Affecting System - Specific IP's

11/10/2014 16:56 - 11/11/2014 11:50
Last Updated 11/10/2014 16:56

There appears to be a network issue affecting a very small number of IP blocks in our US facility. We're checking it out.

MySQL Upgrades (Resolved) Medium

Affecting Other - All Shared and Reseller Servers

09/02/2014 01:00 - 09/06/2014 06:00
Last Updated 09/08/2014 08:49

We will be upgrading MySQL on all Shared and Reseller hosting servers in the early hours of Tuesday - Saturday morning, we'll be upgrading a small number of servers each night. There is likely to be around 1 hour of service interupption for MySQL driven websites.

Update: The MySQL upgrades on the servers Wilde and Shaw have been completed, the next batch of servers will be upgraded on Wednesday morning

Update: The MySQL upgrades on the servers Baracus, Hermes and Goron have been completed, the next batch of servers will be upgraded on Thursday morning

Update: The MySQL upgrades on the servers Kipling, Hyrule and Nash have been completed, the next batch of servers will be upgraded on Friday morning

Update: All servers have now been updated

Server Unresponsive (Resolved) High

Affecting Server - Shaw

08/13/2014 13:57
Last Updated 08/13/2014 16:04

The server has gone unresponsive. We're currently checking on it.

Update: This has now been resolved, we will continue to monitor the server

Server Outage (Resolved) High

Affecting Server - Nash

08/06/2014 11:38 - 08/06/2014 20:01
Last Updated 08/06/2014 11:44

The Nash server is suffering sporadic outages, we are checking on it.

Update: Server is back up and we're monitoring it

Server Maintenance (Resolved) Medium

Affecting Server - solus.hyliahub.com

07/29/2014 09:36
Last Updated 07/29/2014 10:02

We've discovered that the server isn't discovering one of the RAM sticks which is leading to performance issues. We have taken the server down to investigate, we hope to have it back online shortly.

Update: We have replaced the defective RAM and the server is back up and running. VPS's are being restarted one by one, all VPS's should be back up within the hour.

Intermittent Network Issue (Resolved) High

Affecting System - New Jersey Network

06/30/2014 15:30
Last Updated 06/30/2014 18:23

We are seeing intermittent packet loss on our New Jersey network. The datacentre are aware of the issue and are currently working on it.

UPDATE;

18.44 - the issue is now resolved

Apache, MySQL and PHP Updates (Resolved) Low

Affecting System - All servers

06/27/2014 10:49
Last Updated 06/28/2014 13:16

We will shortly be upgrading the Apache Web Server on all Shared, Managed VPS and Managed Dedicated Servers to address a security vulnerability found in older versions of Apache. Any Managed VPS clients running an older version of MySQL that is causing cPanel updates to fail will also be upgraded at the same time.

We will also be updating the default PHP version on all Shared Servers at the same time as the Apache upgrades. Managed VPS and Managed Dedicated Server customer wont have the PHP version changed unless you're running an old PHP version that is no longer supported by cPanel/EasyApache/

The majority of Shared servers were still running PHP 5.2.* as the default version, despite it going into "end of life" status back in January 2011 so we are upgrading the default PHP version to PHP 5.4.*

If you have a script that wont run on the new PHP version, you can switch back to PHP 5.2.* via cPanel > Select PHP Version. (This applies to Shared and Reseller Customers only)

No downtime is expected during the upgrades.

Kernel Upgrades (Resolved) High

Affecting Other - Multiple Servers

06/10/2014 02:00
Last Updated 06/10/2014 13:51

The following servers will be rebooted at 2am EST on Tuesday 10th June to fix the recently discovered CVE-2014-3153 kernel vulnerability:

bullseye.d9hosting.com
goron.hyliahub.com
hermes.hyliahub.com
hyrule.hyliahub.com
kipling.hyliahub.com
shaw.hyliahub.com
wilde.d9hosting.com

The following servers will be rebooted at 2am GMT on Tuesday 10th June:

baracus.hyliahub.com
fleetwood.hyliahub.com
nash.hyliahub.com

Downtime is likely to be 5 - 10 minutes unless a FSCK is needed.

Update: This has been completed and all servers are back online

UK VPS Network Maintenance (Resolved) High

Affecting System - UK VPS Network

02/15/2014 16:00 - 02/18/2014 09:49
Last Updated 02/14/2014 10:13

On 2014-02-15 we'll be performing some maintenance that will affect your service. We will be migrating UK VPS's to a new facility, as part of our expansion programme for services capacity and network offering. The migration will happen overnight, from 22:00-02:00 (GMT).

US Backup Server Maintinence (Resolved) Medium

Affecting System - Backup Server

02/06/2014 10:04 - 02/12/2014 15:18
Last Updated 02/06/2014 10:05

We are currently re-configuring the RAID level on our US backup server; vault.bkpvault.com. During this time backups will be unavailable for restoration.

New Jersey Network Maintenance (Resolved) High

Affecting System - New Jersey Network

12/04/2013 02:00 - 12/10/2013 14:09
Last Updated 12/03/2013 22:12

This notification is to inform you of scheduled maintenance that is taking place in our New Jersey facility. Please review the details of this maintenance below:

Event Type: Network Upgrade
Start Time: 2013-12-04 02:00 EST
End Time: 2013-12-04 02:45 EST
Device(s): br1.pnj1

Event Summary: We have worked with our vendor to isolate the root cause of a brief outage on Monday 2 December, 2013 which impacted some customers. The root cause is a recently discovered memory leak in the current software version. To resolve this condition, the OS level must be upgraded immediately and will require a router reload. The other core routers are not experiencing the defect but will be scheduled for upgrade shortly in non-emergency maintenance windows.

The purpose of this maintenance window is to apply an OS version upgrade to one of our core routers in the PNJ location. This upgrade will require a reload which may impact customer traffic for up to 15 minutes.

Server down (Resolved) Critical

Affecting Server - solus.hyliahub.com

10/20/2013 19:10 - 10/20/2013 19:41
Last Updated 10/20/2013 19:41

Kraken went unresponsive and we are working on it. So please hold on.

Server back online. Restarting each containers one by one.

Containers are back online.

Network Issue (Resolved) High

Affecting Other - Pennsylvania Network

08/27/2013 20:57
Last Updated 08/29/2013 13:45

There appears to be a network issue affecting the Pennsylvania datacenter that houses our budget dedicated server clients. We will post more info when we receive it from our upstream provider.

The issue is now resolved and we received the following info from the data center:

"We are aware of the network/routing issues recently arising, and have been working hard to resolve such. The is issue is due to a combination of issues/factors occurring all at once, including: Relocation of carrier circuits to the new PA facility from our old facility, maintenance from carriers on multiple circuits, defective RAM/Memory in a gateway router, BGP flaps from a backbone carrier, Defective carrier equipment card on a long haul transport route, a DDOS attack direct to some router interfaces w/carrier uplink attached, amongst a few other smaller issues. All of these issues hitting at one time, made it a little hard/complex to troubleshoot problems, and get a handle on what was actually going on.

We believe everything has been resolved to the point, and expect no other major service issues moving forward. There is some clean up work still needing to be done, and some circuits to still migrate, but we do not expect to see major network issues arise from them as per the last few days random problems. There may be a few minor hiccups, such as BGP updates, as some circuits are relocated to the new facility over the next week, and as we shift traffic around.

We than thank you for your patience and understanding with this, and we expect smoother network performance moving forward."

High Server Load (Resolved) High

Affecting Server - Wilde

08/09/2013 09:25 - 08/10/2013 08:57
Last Updated 08/09/2013 12:46

We are just doing the final data sync on the Wilde server before we complete the migration to the new datacenter, during this time load on the old server will be high and sites will be loading slower than usual but it will only be for an hour or two.

Update: We are going to disable httpd and mysql on the server just so we can complete the final sync quicker since it was badly overloading the server. We expect the services to be down for around 1 hour and once the sync is complete we will route traffic through the new server

MySQL Down (Resolved) High

Affecting Server - Bullseye

08/07/2013 11:51 - 08/09/2013 09:24
Last Updated 08/07/2013 11:51

We are aware of an issue with the MySQL service on Bullseye, we are checking it.

Server Unresponsive (Resolved) Critical

Affecting Server - Bullseye

07/30/2013 18:47
Last Updated 07/30/2013 22:09

Server is currently unresponsive, we have issued a reboot request

Update: Server back up after a reboot

Update: Server is overloaded, checking it

Update: Server is back up, we have tweaked a few settings and are monitoring it

Server nash.hyliahub.com is down (Resolved) Critical

Affecting Server - Nash

07/27/2013 13:05 - 07/31/2013 08:11
Last Updated 07/30/2013 05:51

**Please Scroll To The Bottom of the Page for Latest Updates**

The server has gone unresponsive again and the DC techs are checking on it.

Update: THere appears to have been a heating issue causing a kernel panic, we have re-thermal pasted the CPU, and checked the health of the heatsync. Will be attempting to bring the server back up momentaerially.

Update: The server died again as it was coming back up, onsite techs are still working on it. Looking like we may have to go for a chassis swap

Update: Server was up for a period but died again. We are working on getting replacement hardawre set up.

Update: Datacenter is sourcing replacement hardware whilst still trying to get the server back up. In the meantime if you have a local backup of your sites please open a ticket with technical support and we can set up your account on a different server and you can upload your data.

Update: We are unable to locate an exact hardware match in the UK facility so it looks like we are going to have to restore data to a US based server whilst the DC are able to order in a new box for the UK facility. Will keep posting updates as and when we get them. As we mentioned before, if you have local backups that you can restore than please let technical support know and we can recreate your account on another server.

Update: We have got a new server ready and will shortly start restoring data to that server from our Last good backup which is from the 24th July. There is just short of 400 GB of data to be restored. We will keep this page updated with the progress.

Restore Updates

==============

Update 1: Currently restoring /var (the database partition) and it's just over 15% done....now 23% done...../var is now 42% done....... now 65% done..... now 100% and completed.
Update 2: Currently restoring /tmp it's over 35% done........ now 95% done........now complete.
Update 3: Started restoring the /home partition ... 29% done ... 31%done .... 38% done...45%done
Update 4: Restoration of /home partition is still in progress. Currently 49% done...52% done...55% done.... 59% done... 63% done.. 67% done...73% done...76% done.....now 81% done.... now 87% done......now 92% done........now 98% completed.... now 100% completed.
Update 5: Bare-Metal Restore has been completed. Its time to reboot the server through HDD and after that the new Ip's will be assigned.
Update 6: Onsite techs are currently booting server into single user mode to reset the password, once done new IP's will be assigned
Update 7: We should be back up very soon, we can see signs of life in the IP's and are just waiting on final confirmation from the datacenter before we can make any needed nameserver IP changes

Update
=======

The server is now back online with new IP's. Nameserver IP's have been updated and your sites will shortly start to resolve from the new IP addresses if they are not already doing so. If you are using a 3rd party DNS service and just pointing your A records to the server, please update them to use the following IP:

209.140.31.226

We will be emailed all affected clients with an incident report shortly.

Server Unresponsive (Resolved) Critical

Affecting Server - Bullseye

07/29/2013 08:18 - 07/29/2013 09:28
Last Updated 07/29/2013 08:18

The Bullseye server is currently unresponsive, we are checking it.

Server Unresponsive (Resolved) High

Affecting Server - Nash

07/25/2013 15:46 - 07/26/2013 09:39
Last Updated 07/26/2013 08:36

The server nash.hyliahub.com is not responding. We are checking it.

Update: Unfortunately we have no further info we can give you other than the datacenter are still working on the server but if/when we get more information we will post it here.

Update: We are able to get access to the server for brief periods but soon after it goes inaccessible, looks like it could be a PSU issue. Still checking it.

Update: We have been able to bring the server back up. We are closely monitoring it.

Update: We are rebooting the server to install a new kernal.

Update: Server booted into new kernal successfully but disk IO is still very high. We are checking the drives for errors. During this time http may be up and down.

Server Unresponsive (Resolved) High

Affecting Server - Nash

07/13/2013 04:32 - 07/13/2013 08:01
Last Updated 07/13/2013 06:14

The server is currently unresponsive, we are checking on it.

Update: Datacenter techs are checking on the server

Pennsylvania Datacenter Move (Resolved) Medium

Affecting Other - Clients Dedicated Servers

07/10/2013 08:17 - 07/11/2013 10:09
Last Updated 07/10/2013 08:19

As per our recent emails, a number of clients with Dedicated servers located in PA are being moved (the servers are being physically moved) to our upstream providers new facility 3 miles down the road. Downtime for the move is expected to be 2 - 4 hours. Full details of the move are in the email that was sent out to all affected clients last week.

UK Servers Unreachable (Resolved) High

Affecting System - UK Cheltenham Network

07/04/2013 11:58 - 07/08/2013 03:07
Last Updated 07/04/2013 14:42

There is currently a network issue with the UK datacenter that houses our Shared and Reseller UK servers. We are seeing lots of packet loss which is causing sites to time out, the DC is currently working on the issue.

Update: There is a massive DDOS attack which is overloading the main switch leading to the packet loss, DC techs are working to mitigate the attack

Update: The IP that was being attacked (it wasn't one of ours, but another client on the same switch) has been null routed and servers are accessible again.

IP rerouting to new hardware (Resolved) Medium

Affecting Server - solus.hyliahub.com

05/24/2013 09:00 - 05/28/2013 05:52
Last Updated 05/24/2013 11:14

As per our email notice, we will be routing the IP's on the VPS node kraken.hyliahub.com to new a new box. We have opened a maintinance window for this from 9am - 11am EST on Friday 24th May, you may experience sporadic outages until the new routing is in place.

Update: The IP's have been routed to the new server successfully, we are now working to bring the VPS's back online. Thanks for your patience.

Update: This has now been completed and all VPS's are back online, apologies that we went slightly over the 2 hour window we gave ourselves, it was due to an unforseen firewall issue. Please contact tech support if you have any issues after the migration, you may see a few duplicate emails come through from your email accounts but this is normal and nothing to worry about.

wp-login.php DDOS (Resolved) Medium

Affecting Other - Multiple Servers

05/20/2013 16:12 - 05/23/2013 08:45
Last Updated 05/20/2013 16:14

We are seeing a reoccurance of the wp-login.php DDOS attack on a smale scale against a small number of our servers, as such we have temporarily suspended global access to wp-login.php pgaes until the attack subsides. If you need access to the admin area of a Wordpress based site in the meantime please follow the instructions in this guide:

http://d9clients.com/knowledgebase/113/How-To-Enable-Wordpress-Admin-Area-Access-For-Your-IP-Address.html

DDOS Attack (Resolved) High

Affecting Server - Nash

05/14/2013 04:46 - 05/15/2013 08:26
Last Updated 05/15/2013 07:14

We are currently experiencing a DDOS attack against the server nash.hyliahub.com which is causing sub-optimal performance, we are currently working on brining this under control.

Update: We have implemented the global wp-login.php block and this has brought the server load back to normal levels. We will leave the block in place whilst the DDOS subsides and then remove it. If you need access to a Wordpress admin area in the mean time, please follow this guide:

http://d9clients.com/knowledgebase/113/How-To-Enable-Wordpress-Admin-Area-Access-For-Your-IP-Address.html

Update: This how now subsided, and we have removed the global wp-login.php block.

Update: The issue has returned again today, we are working on it.

Server Reboot (Resolved) Critical

Affecting Server - solus.hyliahub.com

05/14/2013 08:19 - 05/14/2013 08:51
Last Updated 05/14/2013 08:51

Server has gone unresponsive so we are rebooting it, it should be back up shortly. As there have been an abnormal amount of errors on this node recently we are arranging for new hardware to replace the existing box which should fix these ongoing issues.

Update: Server is back up and all VPS's online

Filesystem Error (Resolved) Critical

Affecting Server - solus.hyliahub.com

05/13/2013 00:26 - 05/13/2013 02:21
Last Updated 05/13/2013 01:15

We are facing some file system related issues on hardware node Kraken. We are working on this issue, so please wait for further updates from our end.

Update:

Issue resolved and server back online.

Server Reboot (Resolved) High

Affecting Server - solus.hyliahub.com

04/26/2013 06:00
Last Updated 04/26/2013 06:10

Server went inaccessible so we have rebooted, VPS's should be back online shortly.

Update: Server is back online and VPS's up and running, please open a support ticket if you have any problems.

Server Unreachable (Resolved) Critical

Affecting Server - Hermes

04/25/2013 11:02 - 04/25/2013 12:47
Last Updated 04/25/2013 12:46

Server is currently unresponsive, we are checking it.

Update: Server is back up, please open a support ticket if you are still facing issues

Network issue (Resolved) Critical

Affecting Server - Hyrule

04/22/2013 20:35 - 04/24/2013 12:21
Last Updated 04/22/2013 20:37

We are facing a network issue with our server. our network engineers are working on this to resolve this issue asap.

wp-login.php DDOS Attack (Resolved) High

Affecting System - Multiple Servers

04/11/2013 17:10 - 04/22/2013 04:09
Last Updated 04/12/2013 10:27

For Instructions On How To Access Your Wordpress Admin Area. Please See The Following Knowledgebase Article:

http://d9clients.com/knowledgebase/113/How-To-Enable-Wordpress-Admin-Area-Access-For-Your-IP-Address.html

Friday Evening Update:

The blocking of wp-login.php files has had the desired effect, and servers have remained stable throughout the day. Whilst the attacks do seem to have slowed down over the past few hours, we have seen this behaviour over the past couple of days, only for things to peak up again after a few hours of being idle, so with that in mind we will be leaving all wp-login.php files disabled over the weekend and then reassess things on Monday. The last thing we want to do is let our guard down, and find ourselves back to square one over the weekend when we wont have as many staff on shift the handle the situation.

Users can still access their Wordpress admin areas by whitelisting their IP address, as shown in this article.

======================================================

Friday Morning Update:

As the attacks show now sign of slowing down, we have now taken the steps to globally disable access to all wp-login.php files on all shared and reseller servers. This has already been implemented on all servers apart from Bullseye which was brought down by the DDOS attack a short time ago. Once Bullseye comes back up we will implement the temporary fix in an effort to keep the servers online throughout this industry wide attack.

======================================================

Historic Updates - for information purposes only, please refer to the updates above for the most recent info

Over the past 24 hours there has been a widespread DDOS campaign targetting the popular CMS script Wordpress, inparticular the file used to log into the admin area of the script; wp-login.php. Large numbers of IP addresses from across the world have been attacking any files they can find, resulting in massive issues for web hosts and the stability of web servers.

Whilst we have done our best to manage the situation, a number of our servers have been hit hard, some have even been taken offline by the attack.

To get everything back online on the servers that have been hit the hardest (at the time of writing these are the servers Hermes, Kipling and Wilde) we have had to take the only option left that was to available to us, and that is to disable access to all wp-login.php files on the server, and the block any IP's that attempt to access a file with that name.

We understand that this will mean a small number of genuine users will be locked out of the server for a short period (60 mins), but to put it into context, this measure has also allowed us to block malicious IP addresses that were causing the issues, at a rate of around 1,000 per minute. So you can see the sort of numbers we are having to deal with across multiple servers.

So What We Are Asking From You Is This
==============================

- If you are on one of the servers where we are temporarily disallowing access to wp-login.php and you DON'T need urgent access then please sit tight and bare with us, normal service will be resumed once the attacks subside

- If you DO need URGENT (we must stress this part, please only request access if absolutely vital) access to the admin area of your blog, please open a ticket with technical support and provide us with:

1. The URL of the blog you want access to

2. Your IP address - this can be found at www.whatismyip.com

Thanks for your patience in this matter, we know it's going to be a bit of a pain for the next few days but it was either this, or have the servers going down every few minutes when the DDOS overloaded them! Also, if you want more details on the nature, and the scale of the attack, you can see how the rest of the web hosting community is dealing with the issue in this forum thread.

wp-login.php brute force/ddos attack (Resolved) High

Affecting Server - Hermes

04/11/2013 15:47 - 04/11/2013 17:27
Last Updated 04/11/2013 15:48

The Hermes server has just been hit by the wp-login.php ddos attack that has been going round the net for the past 12 - 24 hours and is currently inaccessible, we are working on it. Please see the "Wilde" network issue down the page for full details on the issue.

httpd service down (Resolved) High

Affecting Server - Wilde

04/11/2013 06:54 - 04/11/2013 17:27
Last Updated 04/11/2013 12:09

The httpd service on this server has gone down, causing websites on the server to become inaccessible. It appears to be due to a large scale brute force attempt from lots of IP's focusing on a lot of different wp-login.php pages. We are aware of the issue and are working on it.

Update: We are now seeing the wp-login.php brute force issues on a number of other servers, and after some research we have discovered this is a wide scale brute force attack which is affecting lots of other web hosts. More details can be found here. We are currently testing out mod_security rules on a server that is under attack, if successful this would block the IP's making the brute force attempts and bring service levels back to normal.

Update: We have managed to stabalise all servers apart from Wilde, which is being hit particularly hard by this brute force/ddos attack. We are still working towards a solution on Wilde and really appreciate your patience on this matter.

httpd down (Resolved) High

Affecting Server - Nash

04/11/2013 04:27
Last Updated 04/11/2013 05:28

httpd has crashed, which is causing websites to time out. We are aware of this issue and are currently working to get the service back online.

Update: We have brought httpd back online and the server is stable. We will continue to monitor the server for the next few hours.

UK Network Maintenance (Resolved) Medium

Affecting Server - Baracus

04/09/2013 02:00 - 04/09/2013 04:58
Last Updated 04/08/2013 05:34

We will be performing the network maintenance on Tuesday, April 9th at 6AM GMT. The purpose of this maintenance is to optimize the network configuration.

The servers which will be involved in the scheduled maintenance are:

baracus.hyliahub.com

The maintenance window should not be longer than 30 minutes. You should not experience downtime for longer than 5 minutes.

Nash server : network issue (Resolved) Critical

Affecting Server - Nash

03/25/2013 16:03 - 03/26/2013 00:00
Last Updated 03/26/2013 06:49

Nash server is down due to a network config error and we are working on getting it back up

Update : 2 Our network engineers are still working on the network configuration.

Update 3: We apologize profusely for the unacceptable downtime on Nash that occurred overnight and into this morning, we have now resolved the issue with the internal network issue/IP routing and your site and emails are back online.

Server Unresponsive (Resolved) High

Affecting Server - solus.hyliahub.com

03/14/2013 09:13
Last Updated 03/14/2013 09:39

The server kraken.hyliahub.com has gone unresponsive, we are checking on it.

Update: Server is back online after a reboot

Network Issue with server (Resolved) Critical

Affecting Server - Baracus

03/10/2013 13:49 - 03/10/2013 13:58
Last Updated 03/10/2013 13:51

We are now facing a network issue with the server.

Our network engineers are working on this.

This should be fixed asap. We will keep you updated.

Server Unresponsive (Resolved) High

Affecting Server - Hyrule

03/07/2013 12:54 - 03/08/2013 03:53
Last Updated 03/07/2013 14:57

A hacked account on the server has flooded the mail queue and sent it unresponsive. We've currently issued a reboot request so the server should be back online shortly. Thankyou for your patience.

Update: The server is back up after a reboot and FSCK, apologies for any inconvenience caused.

Atlanta Network Maintenance: Sunday Febuary 2 (Resolved) Low

Affecting Other - Network

02/24/2013 05:00 - 03/07/2013 14:57
Last Updated 02/22/2013 03:56

The Atlanta datacenter that houses our Shared and Reseller servers will be performing Network Maintenance on Sunday February 24th between the times of 5am to 7am EST. We do not anticipate that you will experience any downtime and there should be no interruptions in your service.

Server Unresponsive (Resolved) High

Affecting Server - Hermes

02/13/2013 03:14
Last Updated 02/13/2013 04:25

The server has gone unresponsive, we are checking it and will get it back up ASAP.

Update: The server is back online, we have temporarily stopped the httpd and mysql services to keep the load down whilst we investigate what caused the server to go down

Update: We have re-enabled http and mysql on the server, we found an abusive user on the server which we believe caused the server to become overloaded. The user has been suspended and load has gone back to normal.

network issue with server (Resolved) Critical

Affecting Server - Hermes

02/12/2013 20:03 - 02/13/2013 04:11
Last Updated 02/13/2013 03:53

Unfortunately we are facing a network issue with our shared server hermes.hyliahub.com.

Our network engineers are working on this. We hope that the issue will be resolved as soon as possible.

We really do apologize for the inconvenience this might have caused you.

Further updates will be notified here.

Network issue with Server (Resolved) Critical

Affecting Server - Shaw

02/06/2013 09:56 - 02/07/2013 01:29
Last Updated 02/06/2013 09:59

We are currenlt facing a network issue with our shared server shaw.hyliahub.com.
Our network engineers are working on this and this should be fixed asap.

Server Down (Resolved) High

Affecting Server - Hyrule

01/10/2013 06:12 - 01/10/2013 06:59
Last Updated 01/10/2013 06:13

Server has gone unresponsive, we are rebooting it

Goron Server Not Responding (Resolved) High

01/09/2013 18:13
Last Updated 01/09/2013 18:49

Goron server isn't responding and we're unable to establish kvm and apc. We have contacted the data center and will update when we hear more details.

Currently server Goron is up and running fine.

RAM & MOBO Swap (Resolved) High

Affecting Server - solus.hyliahub.com

12/28/2012 06:00 - 12/31/2012 06:13
Last Updated 12/27/2012 17:34

We will be taking down the VPS hardware node "Kraken" this Friday (December 28th) at 6 am EST to perform a RAM and Motherboard swap to resolve ongoing stability issues we have been facing over the past couple of weeks.

The hardware swap should take no more than 30 minutes, and once the server is back online individual VPS's will be booted up one by one, which usually takes 1 hour to complete. So total downtime should be around 2 hours.

Thanks for your continued support.

RAM Replacement (Resolved) Medium

Affecting Server - solus.hyliahub.com

12/19/2012 23:59 - 12/31/2012 06:13
Last Updated 12/15/2012 16:16

We will be taking down the VPS node Kraken at midnight on Weds 19th December EST to replace the RAM in the server.

Downtime will be 1 - 2 hours.

UK Network Issue (Resolved) High

Affecting System - UK Network

12/13/2012 10:02 - 12/13/2012 14:09
Last Updated 12/13/2012 10:42

There is currently an issue on part of the UK network we use which is causing connections to the following services/services to fail:

- Nash (UK shared & reseller server)

- Some UK VPS's

The datacenter technicians are already working on the issue and should have it resolved soon.

Update: We are now back online, apologies for any inconvenience caused.

Update: The network is experiencing issues again, onsite techs are already working on it.

Server Unresponsive (Resolved) High

Affecting Server - solus.hyliahub.com

12/10/2012 05:29 - 12/11/2012 10:06
Last Updated 12/10/2012 05:30

The VPS node kraken.hyliahub.com has gone unresponsive, load was high a couple of hours ago but we brought it under control however it has now gone unresponsive so we are working on it.

Server Unresponsive (Resolved) High

Affecting Server - Wilde

12/07/2012 03:32 - 12/07/2012 03:49
Last Updated 12/07/2012 03:49

The server wilde.d9hosting.com is currently unresponsive, we are checking it.

Update: Server is back online after a reboot

Kernal Upgrade (Resolved) Medium

Affecting Server - Kipling

11/02/2012 05:37 - 11/02/2012 06:07
Last Updated 11/02/2012 05:38

We will shortly be rebooting the server to complete a Kernal upgrade on kipling.hyliahub.com, downtime will be around 10 - 30 minutes.

Server Performance Issues (Resolved) High

Affecting Server - Nash

10/29/2012 10:24
Last Updated 10/29/2012 10:40

Server is suffering performance issues, we are checking it.

Update: An abusive account on the server was sending large amounts of spam, overloading the server. The account has been suspended and the server load is back to normal.

Server Unresponsive (Resolved) High

Affecting Server - Hermes

10/02/2012 03:16 - 10/02/2012 03:45
Last Updated 10/02/2012 03:20

The server hermes.hyliahub.com has gone unresponsive. We have issued a reboot request.

Update: Server is back up after a reboot. We will keep a close eye on it for the next few hours.

RAM Replacement (Resolved) High

Affecting Server - solus.hyliahub.com

10/02/2012 01:00 - 10/02/2012 03:05
Last Updated 10/02/2012 03:05

We have discovered a corrupt memory module in the server, so we will be taking it down at 1am EST on October 2nd to replace the corrupt module. When the new module is installed the server will be booted back up, and the VPS's will come back up one by one. Downtime is expected to be between 60 - 180 minutes.

Update: This has been completed. Changing the RAM took around 10 minutes, and then all individual VPS's booted up one by one. Total downtime was between 15 minutes and 1 hour depending on the individual VPS.

Server Down (Resolved) High

Affecting Server - Hermes

09/28/2012 15:23
Last Updated 09/28/2012 15:29

The server has gone unresponsive, we are checking it now.

Update: The server is back up after a reboot, we will keep an eye on it for the next few hours.

Drive Replacement (Resolved) Medium

Affecting Server - Kipling

09/10/2012 16:12 - 09/11/2012 16:22
Last Updated 09/10/2012 16:13

One of the drives in the RAID array is showing signs of deteriation, so we are going to replace it to be on the safe side. We don't expect any downtime but load on the server will be higher than normal whilst the RAID rebuild completes.

Drive Replacement (Resolved) Medium

Affecting Server - Kipling

08/20/2012 08:14
Last Updated 08/23/2012 09:57

We have found 2 bad drives in the RAID array, we are replacing them 1 by 1. During this time you may experience sub-optimal performance on the server.

Update: The 1st drive has been replaced and the RAID rebuild completed, we will now proceed with the 2nd drive.

Update: This has been completed

Server Inaccessible (Resolved) High

Affecting Server - Wilde

08/16/2012 09:41
Last Updated 08/16/2012 09:51

The server is currently inaccessible, we are checking it.

Update: Server has been rebooted and is now back online

Degraded RAID Array (Resolved) High

Affecting Server - Kipling

07/23/2012 10:55
Last Updated 07/23/2012 20:59

One of the disks in the RAID array is failing, which is causing instability on the server. We have contacted the DC to get the bad drive replaced.

Update: Apologies for the lack of updates, still going through the RAID rebuild process which by nature is very resource intensive until it's done, and can take a few hours to complete.

Update: We've brought httpd back online so sites are now loading again, RAID rebuild is 72% complete so the server will be slower the usual until it completes.

Update: The RAID rebuild on KIPLING server is now complete!

Kernal Upgrades (Resolved) Medium

Affecting Other - All Shared Servers

07/20/2012 00:30 - 07/23/2012 11:06
Last Updated 07/19/2012 03:15

We will be updating the kernals on all shared servers during the early hours (EST and GMT) or Friday July 20th. To complete the kernal upgrades each server will need to be rebooted, which will cause sites and email to go offline until the server has booted up in the new kernal. Downtime should be no more than a few minutes, unless we run into issues.

The servers will be upgraded one by one and we will be starting work at 3.30am GMT (12.30am EST), servers to be upgraded are:

Baracus
Bullseye
Goron
Hermes
Hyrule
Kipling
Nash
Shaw
Wilde

Server Unreachable (Resolved) High

Affecting Server - Baracus

07/13/2012 03:10 - 07/13/2012 05:33
Last Updated 07/13/2012 05:33

The server is currently unreachable, we are checking it.

Update: We are not able to access the server via IPMI either, so we are having to escalate the issue to the datacenter, we will update this page when we get more info.

Update: The server is now back online, apologies for any disruption caused.

Server Inaccessible (Resolved) High

Affecting Server - Nash

06/30/2012 04:36
Last Updated 06/30/2012 04:43

The server is currently inaccessible, we are checking it.

Update: Server is back online after a reboot

Sites not resolving (Resolved) High

Affecting Server - Wilde

06/18/2012 16:00 - 06/18/2012 16:51
Last Updated 06/18/2012 16:51

There appears to be an issue with HTTP and/or Named on the server, we are checking it.

Update: BIND appears to have broken beyond repair, so we've switched to NSD for the time being to get sites back online. We'll open an issue with cPanel regarding BIND and switch back from NSD > BIND once we hear back from cPanel.

Update: We have resolved the issue with BIND and have reverted back to it, please open a support ticket if you have any issues.

Server Unreachable (Resolved) High

Affecting Server - Baracus

05/29/2012 03:41 - 05/29/2012 04:09
Last Updated 05/29/2012 04:09

The server baracus.hyliahub.com is currently unreachable due to a network wide issue at the UKServers datacenter, they are aware of the issue and are working to bring the network back online ASAP.

Update: This has now been resolved

Nameservers Not Responding (Resolved) High

Affecting Server - Hermes

05/19/2012 03:34 - 05/19/2012 04:26
Last Updated 05/19/2012 04:25

The nameservers on the server hermes.hyliahub.com are not responding, as such sites are not loading. We are working on it.

Update: We are unable to get the nameservers to respond on any IP's, so have contacted cPanel regarding the issue. We will keep this page updated when we receive further information.

Update: We found that an automated named.conf update broke the nameserver configuration on the server and caused it to stop responding to external NS queries. We have reverted to a backup of the named.conf file and the nameservers now resolve again and sites are back online.

DDOS Attack (Resolved) High

Affecting System - Atlanta Datacenter

05/16/2012 16:37 - 05/17/2012 08:45
Last Updated 05/16/2012 16:39

Servers in the Atlanta datacenter were unavailable for a few minutes earlier this evening when the data was under attack from a massive DDOS. This has been nullified and the network is accessible again, apologies for any inconvinience caused.

UK Network Issue (Resolved) High

Affecting Server - Nash

04/26/2012 04:05
Last Updated 04/26/2012 10:01

Update: This should now be resolved, please let us know if it persists.

We have had reports of intermittent connectivity issues along certain transit routes with certain ISP's when accessing certain UK servers. The DC is checking on this now and we will update you when we get further information.

Drive Replacement (Resolved) Medium

Affecting Server - Kipling

04/15/2012 02:00
Last Updated 04/16/2012 16:06

Update: Drive replacement and RAID rebuild is now completed

Update: RAID rebuild is 90% complete

Update: The drive has been replaced and the RAID rebuild is currently in progress, it is currently 54% complete

Update: This has been pushed back to the early hours of Sunday Morning (15th April)

One of the drives in the servers RAID array is degraded, so we are replacing the failing drive in the early hours of Saturday morning (EST). The server will be taken down to replace the drive and when back online the RAID array will be rebuilt, which will cause sub-optimum performance until the RAID rebuild has been completed.

Atlanta Network Maintenance - 14th April (Resolved) Medium

Affecting Server - Bullseye

04/13/2012 15:43 - 04/15/2012 03:54
Last Updated 04/13/2012 15:43

One of our upstream providers (cogent) is doing some maintenance on their circuit April 14th.

Start time: 3:00 am eastern 04/14/2012
End time: 6:00 am eastern 04/14/2012

We are taking steps to work around this and clients should not be affected but we want to make sure this is posted so all can be aware if anything changes.

UK Network Issue (Resolved) High

Affecting Other - Network

04/11/2012 05:29 - 04/13/2012 11:05
Last Updated 04/13/2012 11:05

Update: The official cause of the outage we received from the DC was:

"The Datacentre suffered an unplanned network disruption at 11:24, 11/04/2012, they have yet to find the cause but believe it to be unrelated works at Telehouse London. Whilst re-routing the backup feed, their connection was restored"

They will also be installing secondary routing equipment to handle the instant fall back to the backup feed, as clearly this would have prevented today's outage.

Update: The network is now back up, it was down for around 30 mins.

There appears to be an issue with the UK network, effecting the shared/reseller server:

Nash.hyliahub.com as well as clients with UK based VPS's on the IP ranges 178.239.167.*

We have contacted the DC about this.

Sporadic Outages (Resolved) High

Affecting Other - Atlanta Network

04/03/2012 05:06
Last Updated 04/05/2012 09:53

5th May Outage Cause: The datacenter have provided us with the following information regarding the Atlanta network outage experienced on May 5th:

"There was maintenance performed by Cogent. Thus we had to close the layer 2 link to Cogent. They anticipated the completion of the maintenance by 7:00 EST.

Cogent's maint completed, the circuit was restored, the circuit was activated and all is functioning normally."

Update, Thurs May 5th: The network issue reappeared for around 45 mins this morning meaning we were unable to access our client area to answer tickets. Updates were posted on our Facebook and Twitter pages. Apologies for this unacceptable Atlanta network performance of the past few days, unfortunately it is out of our control and we are at the mercy of the datacentre, we will be taking it up with their management to see if we can get any further updates for you.

Update: The following has just been sent to us by the datacenter:

-- Begin DC Update --

"There is a routing loop between some routers which belong to to the service providers through which the packets are routed to us. This is mainly related to not-fully propagated routes across the service providers around the world. Thus if a packet goes through the network which does not have updated records, it loops until the TTL of this packet is expired.

We are working on this issue along with the DC techs and will try to get it resolved as quick as possible. The updates will follow on our forums.
The network loop is noticed through nLayer, Tinet.net and perhaps others are involved. That means if packets from your end-devices are routed through these carriers, the packets will be looped until TTL of the packet is expired.

We still work along with the DC techs to get this resolved but everything is fine inside our network. We just need the routes to get fully propagated to other service providers around the world."

-- End DC Update --

We are experiencing sporadic outages due to a network issue at the Atlanta Datacenter. This appear to be effecting a handful of servers and is causing intermettent connections.

The status updates provided by the DC indicate the issue is caused by a routing loop effecting multiple IP ranges, they are working on it at present. As the servers are all up, and the issue is at the DC end, then unfortunately we will need to wait for their updates regarding this, as it's effecting their entire network, not just our servers. So we will post updates here as and when we get them.

This is only effecting servers located in Atlanta, US. Servers in the UK, Miami, California, Dallas and Pennsylvania are uneffected.

Kernal Updates (Resolved) Medium

Affecting Other - Multiple Servers

03/21/2012 23:00 - 03/21/2012 06:31
Last Updated 03/21/2012 06:30

The following servers will have their Kernals updated in the early hours of Wednesday 21st March. These are both UK servers, so the upgrade will take place in the early hours, GMT:

Nash

Baracus

The servers will be rebooted to complete the upgrade, downtime will be 10 - 20 mins.

Update: These have both been completed

Kernal Upgrades (Resolved) Medium

Affecting System - Multiple Servers

03/14/2012 04:46
Last Updated 03/14/2012 14:13

Update: Hyrule came back online earlier today after a hard reboot, apologies for any inconvinience caused.

Update: Onsite techs are checking on Hyrule

Update: This has been completed, although Hyrule hasn't come back online after the upgrade. We expect it's running a FSCK before it can come back online but have contacted the DC to confirm, will update when more info is available.

We are updating the kernal on the following servers:

Bullseye
Goron
Hermes
Hyrule
Kipling
Shaw
Wilde

The servers will be rebooted to complete the update, so will be down for 10 - 20 minutes whilst they boot back up.

Kernal Upgrades (Resolved) Medium

Affecting System - Multiple Servers

03/01/2012 03:46 - 03/02/2012 03:34
Last Updated 03/01/2012 03:47

We are currently upgrading the Kernals on the following servers, each server will need to be rebooted to complete the upgrade. Downtime will be 10 - 20 minutes per server:

Bullseye
Goron
Hermes
Hyrule
Kipling
Shaw
Wilde

Server Migration (Resolved) Medium

Affecting Server - Nash

02/15/2012 05:13 - 02/23/2012 08:16
Last Updated 02/23/2012 08:16

We will be migrating data on the server nash.hyliahub.com to new hardware on Monday February 20th.

The migration is expected to take between 3 - 5 days.

We will notify all clients via email when this has been completed, no downtime is expected.

Update: We are now running the final rsync and will be pointing IP's to the new server shortly

Update; The final rsync has overloaded the server, we've temporarily stopped httpd and mysql so that it completes quicker

Update: We have postponed the final rsync until tonight, we'll complete the changeover in the early hours of tomorrow morning to avoid disruption for you

Update: The migration has now been completed, please let us know if you have any issues

Server Load (Resolved) High

Affecting Server - Nash

02/09/2012 10:15
Last Updated 02/09/2012 10:42

The server is currently running sluggishly due to a large number of MySQL queries in "locked" status, causing high memory usage. We are working on this now.

Update: This has now been resolved

Server Down (Resolved) High

Affecting Server - Nash

01/23/2012 09:12
Last Updated 01/23/2012 09:30

Server has gone unresponsive, we are checking it.

Update: Server isn't atcually down, but memory usage is high which is causing the sluggishness, we are working on it.

Update: Load has been brought under control and sites are loading normally. Please let us know if you have any further issues.

MySQL 5.1 Upgrade (Resolved) Low

01/20/2012 03:00
Last Updated 01/20/2012 03:52

MySQL will be upgraded to version 5.1 at 3am (EST), on Friday 20th January.

Once MySQL has been upgraded, PHP will need to be rebuilt, during this time MySQL driven sites will show a 500 internal server error. The PHP rebuild will take no longer than 1 hour.

MySQL 5.1 Upgrade (Resolved) Low

Affecting Server - Hermes

01/21/2012 03:00 - 01/21/2012 03:13
Last Updated 01/18/2012 11:02

MySQL will be upgraded to version 5.1 at 3am (EST), on Saturday 21st January.

Once MySQL has been upgraded, PHP will need to be rebuilt, during this time MySQL driven sites will show a 500 internal server error. The PHP rebuild will take no longer than 1 hour.

MySQL 5.1 Upgrade (Resolved) Low

Affecting Server - Hyrule

01/23/2012 03:00 - 01/23/2012 03:22
Last Updated 01/18/2012 11:01

MySQL will be upgraded to version 5.1 at 3am (EST), on Monday 23rd January.

Once MySQL has been upgraded, PHP will need to be rebuilt, during this time MySQL driven sites will show a 500 internal server error. The PHP rebuild will take no longer than 1 hour.

MySQL 5.1 Upgrade (Resolved) Low

Affecting Server - Kipling

01/24/2012 03:00 - 01/24/2012 03:28
Last Updated 01/18/2012 11:00

MySQL will be upgraded to version 5.1 at 3am (EST), on Tuesday 24th January.

Once MySQL has been upgraded, PHP will need to be rebuilt, during this time MySQL driven sites will show a 500 internal server error. The PHP rebuild will take no longer than 1 hour.

MySQL 5.1 Upgrade (Resolved) Low

Affecting Server - Nash

01/25/2012 22:00 - 01/25/2012 03:55
Last Updated 01/18/2012 10:59

MySQL will be upgraded to version 5.1 at 3am (GMT), on Wednesday 25th January.

Once MySQL has been upgraded, PHP will need to be rebuilt, during this time MySQL driven sites will show a 500 internal server error. The PHP rebuild will take no longer than 1 hour.

MySQL 5.1 Upgrade (Resolved) Low

Affecting Server - Shaw

01/26/2012 03:00 - 01/26/2012 03:25
Last Updated 01/18/2012 10:58

MySQL will be upgraded to version 5.1 at 3am (EST), on Thursday 26th January.

Once MySQL has been upgraded, PHP will need to be rebuilt, during this time MySQL driven sites will show a 500 internal server error. The PHP rebuild will take no longer than 1 hour.

MySQL 5.1 Upgrade (Resolved) Low

Affecting Server - Wilde

01/27/2012 03:00 - 01/27/2012 05:34
Last Updated 01/18/2012 10:57

MySQL will be upgraded to version 5.1 at 3am (EST), on Friday 27th January.

Once MySQL has been upgraded, PHP will need to be rebuilt, during this time MySQL driven sites will show a 500 internal server error. The PHP rebuild will take no longer than 1 hour.

MySQL 5.1 Upgrade (Resolved) Medium

Affecting Server - Bullseye

01/18/2012 03:00 - 01/18/2012 04:00
Last Updated 01/18/2012 04:28

MySQL will be upgraded to version 5.1 at 3am (EST), on Wednesday 18th January.

Once MySQL has been upgraded, PHP will need to be rebuilt, during this time MySQL driven sites will show a 500 internal server error. The PHP rebuild will take no longer than 1 hour.

Update: This has now been completed

Server Unresponsive (Resolved) High

Affecting Server - Wilde

01/16/2012 03:32
Last Updated 01/16/2012 03:58

Server has gone unresponsive, we are checking it.

Update: Server is back online

MySQL 5.1 Upgrade (Resolved) Medium

Affecting Server - Baracus

01/14/2012 00:00 - 01/15/2012 05:04
Last Updated 01/15/2012 05:04

MySQL will be upgraded to version 5.1 at midnight (GMT), on Saturday 14th November.

Once MySQL has been upgraded, PHP will need to be rebuilt, during this time MySQL driven sites will show a 500 internal server error. The PHP rebuild will take no longer than 1 hour.

Update: This has been completed

Server Sluggishness (Resolved) High

Affecting Server - Nash

01/12/2012 03:25
Last Updated 01/12/2012 16:06

Load on this server is very high, causing it to be very sluggish, we are checking it now.

Update: We have temporarily stopped httpd on the server to enable us to investigate what is causing the load

Update: We've found the drives have stopped running as SATA and are running as PATA, which is causing the performance issues. We'll modify the kernal and reboot the server with the drives running as SATA which will should solve the issue. We are keeping httpd offline for the moment as it will enable us to work faster on the server, without experiencing the sluggishness.

Update: The issue has now been resolved.

Server Reboot (Resolved) High

Affecting Server - Wilde

01/09/2012 04:17
Last Updated 01/09/2012 04:44

Server is unresponsive, we have rebooted it. Updates to follow.

Update: Server is now back online, please open a support ticket if you have any issues

Drive Replacement (Resolved) Medium

Affecting Server - Kipling

01/05/2012 11:46 - 01/05/2012 17:07
Last Updated 01/05/2012 17:07

We will shortly be taking the server down to replace a failing drive in the RAID array. Once the drive is replaced, the RAID array will be rebuilt, this is a resource intensive process and may result in a bit of sluggishness until it has been completed.

Update: This is now complete, there was no downtime

Scheduled Network Maintenance (Resolved) Low

Affecting Other - Network

12/30/2011 00:00 - 01/05/2012 11:49
Last Updated 12/22/2011 09:02

We have the network maintenance scheduled between 2011-12-30 00:00:00 EST and 2011-12-30 01:00:00 EST in our DC in Atlanta.

Maintenance will be performed on hardware carrying some Atlanta upstream connections on Friday, December 30th. This maintenance will include a brief outage on network connections to PCCW/BtN. Downtime is not anticipated, but some sub-optimal routing may be experienced as traffic is shifted between providers.

Server Not Responding (Resolved) High

Affecting Server - Kipling

12/20/2011 03:23
Last Updated 12/20/2011 04:55

Around 10 minutes ago the server became unresponsive, it's still pinging but we are unable to access the box, so have contacted the DC. Updates to follow.

Update: The server is now back online, please open a ticket if you are still facing issues

Kernal Upgrade (Resolved) Medium

Affecting Server - Nash

12/14/2011 00:00 - 12/14/2011 03:23
Last Updated 12/13/2011 05:51

The server will be rebooted at 2.30am GMT to complete a kernal upgrade. This update is needed to resolve the performance issues you have been experiencing lately - one of the drives is being recognized as an IDE drive rather than SATA which is causing a large IO bottleneck, the kernal upgrade will fix this issue.

Server Reboot (Resolved) High

Affecting Server - Nash

12/12/2011 03:21 - 12/13/2011 05:51
Last Updated 12/12/2011 03:29

Server has become unresponsive and has been rebooted, updates to follow.

Update: Server is now back online, it'll take a few more minutes for sites to start loading whilst the services all restart

Server Slowdown (Resolved) Medium

Affecting Server - Nash

12/08/2011 04:04 - 12/08/2011 04:57
Last Updated 12/08/2011 04:31

The server nash.hyliahub.com is currently suffering performance issues, and this is causing the slow/non loading of websites. The server is up but running very slowly, we are working on it.

Update: The cPanel backup script was causing the load issues, we are manually killing off the proccesses

Update: Load is now coming down, if you give it 10 minutes or so everything should be back to normal.

As a sidenote, new hardware for a new-Nash server was shipped from the US this past Tuesday, so as soon as we get it set up in our UK DC, we'll be migrating all users away from the old (slow!) hardware, to bring it inline with our other UK and US servers.

Scheduled Rebbot (Resolved) High

Affecting Server - Nash

11/03/2011 00:00 - 11/03/2011 07:35
Last Updated 11/03/2011 07:34

We will be rebooting the server in the early hours of Thursday November 3rd to boot it into the CloudLinux kernal. Downtime will be 10 - 30 minutes.

Update:The server isn't booting into the new kernal so the DC techs are currently working on it

Update: It is still being worked on and we are awaiting updates from the DC

Update: Still awaiting more news, apologies we cannot provide you with more information, the DC are not being forthcoming with it which we find totally unacceptable, but that's something we will have to take up with management once the server is back online. We will try and get them to provide us with KVM access so we can work on the server ourselves.

Update: There was a problem with grub on the server. The reinstallation of grub resolved the part of the problem. Grub could not find the kernel files which was the second part of the problem which had to be fixed as well. As it was being done through KVM remotely, it took a bit longer to rectify the issue. Now the server is up but we will need to change the network settings to bring it back on the network as it was connected to PXE VLAN to get it loaded on rescue image.

Update: Server is now back online and sites accessible, please let us know if you continue to have any issues.

Failed Services (Resolved) High

Affecting Server - Nash

10/31/2011 10:28 - 11/02/2011 04:06
Last Updated 11/01/2011 12:02

Final Update: All sites should now be back online, if you are still having problems please can you open a support ticket and we will check it out for you. Apologies for the extended downtime, we will send an incident report to all affected Clients in the next couple of days.

�

Several services on Nash have failed, including httpd, cPanel and MySQL. We are aware of the issue and are working on it. We will update this report with more info as we get it.

Update: We have found an issue with one of the drives, so we are rebooting the server and forcing a FSCK. During this time the server will be offline, the FSCK can take anything from 30 mins to a number of hours depending how much data there is to check. We will update you with more info when available.

Update: The RAID card is failing to be detected, we are replacing it with a new one

Update: The main drive is completely dead and cannot be found by the server. We have replaced it and we are installing the OS on it. Once that is done we will install cPanel and we will restore the data from the backup drive.

Update: Server is back online, we are now proceeding to install cpanel on the server.Please hold on

Update: Data is being synced, please hold on.

Update: As there are around 468 accounts to be copied over and the size of data to be copied is high, it may take a few more hours. As of now 50GB of data in /home directory have been copied. We estimate there is around 100 GB left to be copied.

Update: 106 GB has now been copied, we estimate there is just under 50 GB left to go

Update: The sync is almost complete, accounts are being syncd in order and we are currently on the letter "Y". Once the sync is complete we will repair any permission errors and that should then see sites come back online.

Update: The sync has been completed and some sites back online. Others showing either an Internal Server error, or a 403 Forbidden message will need file permissions fixing. We are currently running a script that will do this, like the sync, it will run through the accounts in alphabetical order.

Update: All accounts with a cPanel username starting with A - D should now be back online

Update: Still having issues with the permissions, we are working on this and will update you soon.

Scheduled Network Maintenance- 03:00 EST thro (Resolved) Medium

Affecting System - Network - Atlanta

10/30/2011 00:00 - 10/31/2011 05:32
Last Updated 10/29/2011 04:52

We have been notified by the datacenter that there is a scheduled network maintenance that will be carried out 03:00 EST through 07:00am EST Sunday, Oct 30th.

Further details follow:

Maintenance Notification:
Facility: Atlanta
Time: 03:00 EST through 07:00am EST Sunday, Oct 30th
Anticipated impact: 30 minutes

Description:
Upstream scheduled maintenance

Impacted services:
Atlanta Cogent (Silver Network) upstream connections

Details of Action:
We have been informed of scheduled maintenance that will briefly impact our Cogent upstream connection on the Silver network in Atlanta on Sunday, October 30th. During this time period customers on the Silver network may experience some sub-optimal routing as traffic shifts to other providers.

IP's Not Resolving (Resolved) High

Affecting Server - Hermes

08/27/2011 04:17 - 08/27/2011 05:59
Last Updated 08/27/2011 04:39

Some of the IP's on hermes.hyliahub.com have gone down which is causing some sites to stop loading, we are currently working on the issue.

Update: All IP's are now back up apart from h209.217.246.141, we are working on it

Change switch port (Resolved) High

Affecting Server - Shaw

08/03/2011 00:00 - 08/04/2011 08:56
Last Updated 08/03/2011 12:56

We'll be taking the server down to put it onto a new switch port that can be managed remotely, during this time websites will be unavailable.

Downtime should be 30 mins - 1 hour.

The work will be done at off peak hours for minimum disruption.

Update: We've had to extend the downtime window as we've run into issues with the server after it's been hooked up in the new rack, we are working on it.

Update: The server is now back online. However, changes need to be made to the Network Switch and the Engineers will be carrying this out shortly

Hardware Check (Resolved) High

Affecting Server - Wilde

08/03/2011 01:00 - 08/04/2011 08:55
Last Updated 08/03/2011 05:42

We will be taking the server down at 1am EST on Wednesday August 3rd to perform a check on the RAM, and replace it if need be.

This should resolve the issues users have been facing over the past few days, downtime is expected to be 2 - 4 hours.

Update: We've had to extend the downtime window as we've run into issues with the server. The memtest failed to complete and the system shut itself down, we believe it's either due to the motherboard or faulty RAM slots. We will get the server back online and then do more checks at off peak hours.

Issue on Wilde Server (Resolved) Critical

Affecting Server - Wilde

08/01/2011 14:24 - 08/01/2011 17:04
Last Updated 08/01/2011 17:04

We are aware that there are issues on Wilde server and have reported this to the data center. We will update when we have more information

The server was not responding to SSH requests. It has been rebooted and is now responding

Performance Issues (Resolved) Medium

Affecting Server - Wilde

07/31/2011 04:59 - 08/01/2011 04:29
Last Updated 07/31/2011 05:00

We are aware of performance issues on the Wilde server, causing websites to load very slowy. We are aware of the issue and are checking it now.

Issue on Wilde Server (Resolved) Critical

Affecting Server - Wilde

07/30/2011 14:37 - 07/30/2011 16:02
Last Updated 07/30/2011 16:02

We are aware that there is an issue on Wilde server we are in touch with the data center to fix this. We wil update when we have more information

�

This issue is now resolved and the server has been rebooted.

Issue on Nash Server (Resolved) Medium

Affecting Server - Nash

07/25/2011 17:01
Last Updated 07/25/2011 17:13

We are aware that there is an issue on Nash server and have contacted the Data Center for more information and will up-date when we have more information.

Update: Server has been rebooted and is back online

Server Reboots (Resolved) Medium

Affecting Other - All shared servers

07/20/2011 05:32 - 07/30/2011 00:00
Last Updated 07/21/2011 03:25

We will be rebooted all of our shared servers onto a new Kernal in the coming days, so during the reboot websites will be unavailable. Downtime shouldn't be more than 30 mins - 1 hour.

We'll be doing the reboots at off peak hours to cause as little disruption as possible.

Update: This has now been completed

Scheduled Network Maintenance (Atlanta) (Resolved) Medium

Affecting Other - Network

07/21/2011 23:00 - 07/22/2011 13:52
Last Updated 07/19/2011 03:52

Maintenance will be performed on some of our core hardware on Thursday, July 21st, requiring a restart of this hardware. This will impact traffic passing through several providers causing traffic to reroute through other upstream connections. It will also impact traffic passing through nLayer network, causing rerouting on this network as well. Other upstream will not be impacted. Some disruption may be experienced during this maintenance as rerouting takes effect.

Work on this issue is scheduled between 2011-07-21 23:00:00 EST and 2011-07-22 01:00:00 EST.

Sites loading cgi-sys/defaultwebpage.cgi (Resolved) High

07/04/2011 10:16 - 07/04/2011 12:03
Last Updated 07/04/2011 12:02

We are aware that some sites are not loading correctly on Goron, and are instead loading cgi-sys/defaultwebpage.cgi

We are checking this issue now.

Update: This has been resolved, apologies for any inconvinience caused.

UK Server Nash (Resolved) Critical

Affecting Server - Nash

06/26/2011 10:17 - 06/26/2011 12:35
Last Updated 06/26/2011 12:35

We are aware that there are currently problems on the UK Server, Nash. We have contacted the data center about this for more information and will update as soon as we receive the info. Thank you for your patience

This issue is now resolved and all sites on Nash should be back up

cPanel Not Loading (Resolved) Medium

Affecting System - Multiple Servers

05/25/2011 14:28
Last Updated 05/25/2011 16:19

We are aware of cPanel loading issues on the following servers:

Bullseye
Goron
Hyrule
Hermes
Shaw
Wilde

This appears to be due to a NFS being down and as such cPanel is hanging when trying to connect to it. We have opened a ticket with the datacenter about this and should have it resolved soon.

IP Resolution Issue (Resolved) Critical

Affecting Server - Nash

05/11/2011 04:53
Last Updated 05/11/2011 05:33

The nameserver IP's for Nash are currently not responding and as such sites are not resolving, we are working on this.

Update: The datacenter will take a bit of time to figure out the resolution issue for the nameserver IP's, so to speed things up we've changed the IP's to new ones and your sites should now be back up. If you are still unable to access please clear your cache, or try accessing from a different browser. If you continue to have issues please open a support ticket.

Upcoming Network Downtime (Resolved) Medium

Affecting System - Network

03/14/2011 06:14 - 03/16/2011 04:37
Last Updated 03/14/2011 06:15

There will be around 5 minutes of downtime on the network later on today whilst we make some routing/subnetting adjustments, this will effect shared and reseller customers hosted on our US based servers.

Shaw (Resolved) High

Affecting Server - Shaw

02/26/2011 05:49
Last Updated 02/26/2011 13:25

The "Shaw" server hasn't come back up after last nights quota recalculation, we have found that it's currently in a "locked" state - we have contacted the DC about this.

�

Resolved at 17.30 GMT

Quota Recalculations (Resolved) Medium

Affecting Other - Servers: Bullseye, Hyrule, Shaw

02/25/2011 00:00 - 02/26/2011 13:27
Last Updated 02/25/2011 15:55

We need to recalculate the quota on the servers Bullseye, Goron, Hyrule and Shaw as they are showing incorrect disk usage data.

We are going to be doing this during off peak hours; 1am EST on Saturday 26th Feb, and downtime for each server should be no more than 30 minutes.

Nash Server Down (Resolved) High

Affecting Server - Nash

02/03/2011 18:22 - 02/04/2011 01:43
Last Updated 02/03/2011 18:49

It would seem that there is a problem with the server, Nash, currently. We are in touch with the data centre to try and ascertain the problem and to get an estimated time of resolution.

We will update this as regularly as we receive updates.

�

00.45 GMT

The Data Centre became aware of power issues affecting UK servers. The power has now been restored and the servers are now coming back online. We will keep you updated here when Nash is fully powered and retored.

Quota Rebuild (Resolved) Medium

11/03/2010 00:00 - 02/03/2011 18:26
Last Updated 11/03/2010 04:42

The quota on the Goronserver has become damaged and needs to be manually rebuilt.

During this time websites and email will be down.

We've scheduled this for 1am EST (6am GMT) on Wednesday 3rd November to cause minimum disruption for users.

Update: This has been completed

Quota Rebuild (Resolved) Medium

Affecting Server - Hermes

10/27/2010 00:00 - 10/27/2010 06:26
Last Updated 10/27/2010 06:26

The quota on the Hermes server has become damaged and needs to be manually rebuilt.

During this time websites and email will be down.

We've scheduled this for 1am EST (6am GMT) on Wednesday 27th October to cause minimum disruption for users.

Update: This has now been completed

Credit Card Gateway Issues (Resolved) Critical

Affecting Other - Card processing

10/01/2010 10:24 - 10/01/2010 13:21
Last Updated 10/01/2010 10:25

We've had reports from customers about credit card payments failing to complete, we've found that this is an issue with our merchant gateway and the payment module our system uses. We are currently working on fixing the issue.

Server Reboots (Resolved) Critical

Affecting Other - All Servers

09/21/2010 16:23 - 09/22/2010 03:19
Last Updated 09/21/2010 16:27

As any Linux geeks will already be aware, a new Kernel vulnerability has just been discovered which can cause big security issues if it isn't patched.

To stop this from happening to us (and you!) we are going to be patching the kernels on all affected servers right away, to complete the kernel upgrade/patch the servers will need to be rebooted, during this time your sites and email will appear offline.

The servers usually only take a couple of minutes to boot back up, but can take up to 1 hour in some cases.

We've got this scheduled to take place at 1am EST on Weds 22nd September to cause minimum disruption for you.

Intermittent Connections (Resolved) High

Affecting Other - Network

09/12/2010 14:56
Last Updated 09/13/2010 03:49

Some users have opened tickets reporting websites being unavailable and connections dropping out, this is only effecting certain users, so if this is effecting you, could we please ask you to do the following on your computer, and then send the results to our technical support department so we can track down the issue:

To perform a tracert on a Windows machine:

Start > Programs > Accessories > Command Prompt

Type the following command and hit the "enter/return" key:

tracert your_domain_here.com

Can you then copy/paste the results of the test and send them to us via a support ticket.

To copy the results to your clipboard, right click anywhere in the black space and choose "Select all", then push the enter/return key on your keyboard. You can then come back into this ticket, and push CTRL + V on your keyboard to paste in the results.

Update: This issue has now been resovled

Server Reboot (Resolved) Medium

Affecting Server - Wilde

09/09/2010 11:18 - 09/10/2010 04:28
Last Updated 09/08/2010 11:19

We are going to be rebooting this server at 1am EST to complete the setup of a NFS. During the reboot sites will be unavailable, it should be for more than 20 - 30 minutes.

Node Migration (Resolved) Medium

Affecting Server - Wilde

09/02/2010 00:00 - 09/02/2010 09:04
Last Updated 09/02/2010 04:09

We are going to be moving the server wilde.d9hosting.com onto a new hardware node to improve performance, we are going to do this at the same time as the VLAN reconfiguration.

There is expected to around 1 hour of downtime for sites and email on the server wilde.d9hosting.com as a final rsync takes place to ensure the integrity of the data.

This is scheduled to take place at 1am EST (6am GMT) on Thursday September 2nd to cause minimum disruption.

Update: This has now been completed, we are now going to tweak the firewall settings on the server

VLAN Reconfiguration (Resolved) Medium

Affecting Other - Network

09/02/2010 00:00 - 09/02/2010 09:04
Last Updated 09/01/2010 09:09

We are going to make some tweaks to our VLAN configuration in the early hours of Thursday morning. Some users may experience some downtime during this time, but we don't expect this to take more than 1 hour to complete.

This is scheduled to take place at 1am EST on Thurs 26th August to cause minimum disruption.

Update: This has been moved back to Thursday September 2nd, at 1am EST.

Issues with credit card processing (Resolved) High

Affecting Other - Payment Gateway

08/20/2010 03:16 - 08/23/2010 10:25
Last Updated 08/23/2010 10:25

We are currently having a few issues with our credit card processing system, so we are unable to process card payments. We are aware of the issue and are working with the gateway to get it resolved.

Update: This has now been resolved

cPanel & WHM Loading Issues (Resolved) High

Affecting Other - cPanel & WHM

08/18/2010 13:15
Last Updated 08/18/2010 16:50

We are aware that at around 7pm GMT, clients have been experiencing loading issues when logging into their cPanel, only half of the page will load, if that.

We are aware of the problem and are currently working to get this fixed. We will update this page when we have more information.

This has now been resolved, please contact us if you face any further issues.

Sites not resolving (Resolved) High

Affecting Server - Hyrule

08/09/2010 05:42 - 08/09/2010 08:17
Last Updated 08/09/2010 08:15

Sites are currently not resolving correctly, we are aware of this and are currently working on the server.

Update: There was a firewall misconfiguration on the server. We have fixed this and the DNS zones are currently reloading, once the zone for a domain has been reloaded the site will appear online, so if some of your sites appear online but others don't, this is the reason. Once all the zones have been re-loaded the sites will be back online.

Update: All sites are now back online, please let us know if you have any further issues.

Multiple Server Network Migration (Resolved) Medium

Affecting Other - Network Migration

07/13/2010 14:28
Last Updated 07/20/2010 09:13

**The migration has now been completed and all sites running from the new servers**

==========================

Status Update 1: A few UK users are having routing problems with the new IP's, we are aware of the problem and the DC are working on it

==========================

Status Update 2: We have moved all sites back to the old servers whilst we sort out the routing issues on the IP blocks, if you still see a "Site is being moved" page, you can either edit your .htaccess file and remove the redirect statement, or open up a support ticket and a member of our team will be happy to do this for you

==========================

Status Update 3: The IP routing issue has been resolved and we are now continuing with the migration, apologies for any inconvenience this has caused you so far

==========================

Status Update 4: A small number of scripts are having problems with blank pages and depreciated functions, this is due to the PHP version used on the new servers, we are currently manually compiling PHP and downgrading it back to the version we had on the old servers - During this time you may encounter a few "500 internal server error" messages when visiting your sites, this is temporary and will be fixed one the recompile is complete.

==========================

Status Update 5: The migration is now complete for all servers except Bullseye which is still transferring. If you find you are missing any files or websites then please let us know ASAP and we will re-sync everything for you. We are currently installing RVSitebuilder on all the servers and will then proceed with the Fantastico installation, so both of the addons should be installed today.

===========================

During the next few days, the following servers will be migrated to new hardware in a new data center:

wilde

bullseye

goron

hyrule

shaw

There should be no downtime during this move, however we like to keep you all informed on what's going on just incase.

The move to the new DC will also incorporate a new hardware approach that we've been researching for some time now, whilst there are many technical details that we wont bore you with, one of the main differences with the new hardware compared to the old is the Disk system we will be using. No longer do you have to put up with SATA drives that bottleneck at peak times, our new hardware includes enterprise SCSI drives in a RAID 10 array to bring you ultimate performance and reliability.

We are also utilizing an external backup system that will provide us not only with account level backups, but also full file system backups so we can get back up and running much quicker if any disaster occurs!

The geographical location of the new data center is in Florida, so any of you that are based in the Florida area may even be able to see the new machines from your bedroom window ;-)

We will send you all an email when the migration is complete, but until then, please let us know if you have any problems.

Kind Regards,
D9 Hosting

P.S. This server migration does not effect clients hosted on our UK servers or Dedicated server clients.

Sluggish Servers (Resolved) Medium

Affecting Other - Servers: Goron, Bullseye, Wilde, Shaw, Hyrule

07/20/2010 03:48 - 07/20/2010 09:12
Last Updated 07/20/2010 03:50

We are currently running the 1st backup on these machines, as it's the 1st backup all data on the server needs to be backed up, so this is quite a resource intensive process and uses quite a lot of resources.

When the backups finish, the load will be back to normal.

Future backups are incremental, so the high load is only due to it being the 1st backup run, rather than something that'll be an ongoing issue.

MySQL & Website problems on Bullseye (Resolved) High

Affecting Server - Bullseye

07/18/2010 05:00
Last Updated 07/18/2010 11:58

Update: The problem is now resolved, an abusive user was found on the server which was causing 100% disk usage and services to fail. We have suspended the user and freed up some space. The MySQL problems were due to databases becoming corrupt when there was no more disk space for them to work in, if you still face problems please let us know.

Update: We've got sites back up and running, but we are having problems bringing up the cPanel service so you'll be unable to log into your cPanel/WHM areas, we are working on this and will update you as and when.

A number of users are reporting "Cannot connect to Database" errors on their database driven websites, we are aware of the problem and are currently investigating.

We have also discovered some sites are not resolving correctly, we've escalated this issue to the DC and update you when we have more info.

httpd Service Not Running (Resolved) Critical

07/16/2010 12:04 - 07/16/2010 12:33
Last Updated 07/16/2010 12:30

After setting the PTR record for the server and tweaking the hostname, Apache is refusing to restart and as such sites are not resolving, we are working on the problem and will have everything back up ASAP.

We have resolved this problem, and all sites are now back up.

Server Down (Resolved) High

Affecting Server - Hyrule

07/12/2010 04:39
Last Updated 07/12/2010 05:04

The Hyrule server has become un-responsive, we are currently working on the server and will post status updates on this page.

Update: The server has been rebooted and is now back online

Network Issue (Resolved) High

Affecting Server - Nash

06/27/2010 04:18
Last Updated 06/27/2010 05:17

Sites on the server nash.hyliahub.com are taking longer to load than they should, this is down to a network problem at the Datacentre, we have opened a support request with them about the issue and will update this page when we have more info.

Update: This has been resolved, please let us know if you have any further issues

Server Not Responding (Resolved) High

Affecting Server - Hyrule

06/23/2010 04:35
Last Updated 06/23/2010 05:00

The server hyrule.hyliahub.com has become unresponsive. We are going to reboot the server and will update this thread when we have more information.

Update: The server is now back online after a reboot, we are investigating to see what caused the issue

Update: The /var partition got clogged up and became 100% full, we've cleared the partition and everything is running fine.

Please let us know if you face any issues.

Drive Replacement (Resolved) Critical

Affecting Server - Bronte

05/21/2010 09:34
Last Updated 05/21/2010 10:59

We are going to be taking the server down to replace a drive, during this time the D9 Hosting websites will be unavailable. (The server is used for the D9 client area, no customer websites will be effected)

This has now been completed

Files Hacked (Resolved) Critical

Affecting Server - Wilde

05/16/2010 04:39
Last Updated 05/16/2010 10:32

It has come to our attention that quite a lot of accounts on the Wilde server have been hacked, and had their index pages replaced. We are going to restore the accounts from our backups, we will update this page when it has been completed.

Update: After some investigating, we have discovered that the hack isn't as widespread as we 1st feared, so we don't need to restore every single account. What we are doing at present is running a report to list all files that were modified on the 16th May (when the hack took place) so we have a full list of every account that has been infected, we can then go ahead and restore those accounts.

Update: We've now located all of the hacked accounts, there are 8 in total. The hack occurred during the last backup, so we've got some clean backups from the last daily backup and some from the weekly. They are currently being restored, once restored we'll be changing the passwords on the accounts and notifying the owners via email.

Can we also take this time to remind users to ensure all scripts are kept up to date at all times.

Update: The restore is now complete, please let us know if you find any more defaced files.

Houston Network Issues (Resolved) High

05/03/2010 08:07
Last Updated 05/03/2010 09:05

We are currently facing a network issue which is effecting clients with VPS, and Hybrid servers in our Houston facility. The DC techs are aware of the problem and are currently working on it.

All of our servers in the DC are back online, please let us know if you continue to have problems.

Power Cable Switch (Resolved) Low

Affecting Server - Hyrule

04/28/2010 01:24 - 04/30/2010 03:45
Last Updated 04/28/2010 01:25

We are going to switch power cables on the rack, to connect the server to a new APC remote reboot device, during this time sites on the server will be down. Downtime shouldn't be more than a few minutes.

Email Issues (Resolved) Critical

Affecting Server - Hyrule

04/24/2010 14:18
Last Updated 04/25/2010 04:34

There are currently problems with the email service on the Hyrule server, inbound mails are not being received. We are currently looking into this.

Update: The issue has been resolved, an automatic Exim update caused the AV scanner to malfunction, you should now be able to receive mails. Please contact us if you face any issues.

Server Down (Resolved) High

Affecting Server - Wilde

04/13/2010 06:42 - 04/13/2010 09:28
Last Updated 04/13/2010 09:28

The server wilde.d9hosting.com has stopped responding. We are aware of the problem and are currently working on it.

Update: A file system check is currently taking place before the server can boot back up, as long as no errors are found during the check the server will continue to boot up and sites will be back online.

Update: The server is still fsck'ing.

Update: The fsck is now complete and the server is back online. Please accept our apologies for any inconvenience caused.

UK Datacentre Network Problem (Resolved) High

Affecting Server - Nash

04/09/2010 12:51 - 04/09/2010 13:13
Last Updated 04/09/2010 13:13

There is currently a network issue at the UK datacentre, which has been caused by the Cisco firmware on the switches. Sites were unable to resolve for a short period of time but are now back up and running and the issue has been corrected.

DNS problem (Resolved) High

Affecting Server - Bullseye

04/07/2010 04:34
Last Updated 04/07/2010 06:33

There is currently a DNS issue on the server bullseye.d9hosting.com which is causing sites not to resolve, we are aware of the issue and are currently working on it.

Update: There was an issue with the firewall on the server, this has been corrected and all sites are resolving again. If you continue to have problems, please clear your ISP and browser cache.

Server Upgrade (Resolved) Medium

Affecting Server - Bullseye

04/02/2010 09:56 - 04/06/2010 16:04
Last Updated 04/06/2010 16:04

We are in the process of upgrading the Bullseye server, we will be moving to a new, higher spec quad core machine, with double the RAM of the old server. The data is currently being sync'd between the 2 servers, so the load on the Bullseye server will be higher than normal at present due to the sync that is taking place.

No action is required on your part, once the data is sync'd we'll update the nameserver, and hostname IP's and sites will start to resolve from the new Bullseye server.

Update: We had added an additional drive to the server, and the upgrade is now complete, all sites now resolve from the new server. Please let us know if you have any problems.

HDD Replacement (Resolved) Medium

Affecting Server - Wilde

04/02/2010 09:55 - 04/06/2010 16:03
Last Updated 04/06/2010 16:03

The primary drive in the server is failing, so we are going to create a snapshot of the drive, and restore it to a new drive. We will then replace the old HDD with the new one, downtime will be 10 - 15 mins whilst the drives are swapped.

Update: This has now been completed

IP's Not Responding (Resolved) High

Affecting Server - Bullseye

04/04/2010 04:10 - 04/04/2010 04:48
Last Updated 04/04/2010 04:48

All of the IP's on the server have stopped responding, apart from the main server hostname IP. We are not yet sure if this is to do with the move to a new box we told you about earlier in the week. We are currently investigating and will let you know when we have any further information.

Update: The issue has now been resolved

RAM Upgrade + Drive Replacement (Resolved) Low

Affecting Server - Wilde

03/26/2010 15:35
Last Updated 03/31/2010 11:17

During the next few days we'll be upgrading the Wilde server, adding more RAM and an additional hard drive. The server will be taken offline to install the new drive and the additional RAM, down time shouldn't be more than 30 - 60 minutes.

Update: We have now started the upgrade. (31st March, 4.15pm GMT)

Update: This has now been completed, downtime was less than 15 minutes.

Drive Replacement (Resolved) Critical

Affecting Server - Bronte

03/30/2010 09:26
Last Updated 03/30/2010 12:16

We will be replacing a drive on the server bronte.d9hosting.com

During the replacement the site will be offline. No client sites are hosted on the server, only the D9 Hosting websites and client area, so clients sites will remain unaffected.

This has now been completed.

High Server Load (Resolved) High

Affecting Server - Bronte

03/23/2010 17:19 - 03/24/2010 14:17
Last Updated 03/24/2010 14:17

The server that hosts the D9 Hosting client area and websites (Bronte) is becoming overloaded, causing the D9 Hosting sites to load very slowly, if at all. We are aware of the issue and are investigating.

No other servers are effected by this, and all clients sites are loading as normal, the problem is isolated to the D9 Hosting websites.

Update: We've installed a new Kernal and this has rectified the issue, please let us know if you continue to have problems.

Rack Connectivity Issues (Resolved) Critical

Affecting Server - Wilde

03/18/2010 09:18
Last Updated 03/18/2010 14:11

Hello,

The rack that houses the server wilde.d9hosting.com is currently having connectivity issues, as such the server is unreachable. The datacentre are aware of the issue and are currently working to bring connectivity back to the rack.

Other servers, on different racks remain uneffected.

Update: There was a power issue with the rack, power has now been restored. We need to wait and see if there server boots gracefully or requires a file system check

Update: Powered is restored but the server hasn't come back online gracefully, so the onsite admins are currently working on the server to bring it back online.

Update: The server is still being worked on after failing to boot itself after the power was restored

Update: The server is currently fscking, updates will follow

Update: The file system check has now completed and the server is back online, thanks for your patience.

Server Down (Resolved) Critical

Affecting Server - Nash

03/13/2010 03:14 - 03/13/2010 03:14
Last Updated 03/13/2010 06:49

The server nash.hyliahub.com is currently unresponsive.

We have put in a reboot request with the DC and will update this thread when we have further info.

Thanks for your patience.

Update: The server is failing to come back online, the onsite NOC techs are currently checking over the server to diagnose the problem.

Update: 12.47 GMT: The server nash.hyliahub.com is up and running fine now

Drive Replacement (Resolved) High

Affecting Server - Hyrule

03/01/2010 00:00 - 03/03/2010 04:43
Last Updated 03/03/2010 04:44

One of the drives in the server is failing, therefore we must replace the drive ASAP.

To do this the server will be taken down to have the drive replaced, the Operating System and cPanel will then be installed, and the data copied over from the bad drive.

During this time your sites and email will be offline.

Downtime is expected to be around 1 hour.

We appreciate your patience during this time, and we will update this status when complete.

This has now been completed, downtime was around 20 minutes.

Changes to backup procedure (Resolved) Medium

Affecting Server - Bullseye

02/12/2010 04:11 - 03/01/2010 08:20
Last Updated 02/12/2010 04:14

We have found that the server is having problems keeping up with the daily backup schedule, each backup is taking longer than a day to complete, so a new backup is starting whilst the old backup is still going on, causing an endless loop of backups, which is causing the server load to remain high at all times.

To combat this, we have decided to set the backups to run every 3 days, rather than every day, to see how this effects the backups and the server performance.

Shaw server downtime (Resolved) Medium

Affecting Server - Shaw

01/30/2010 11:20
Last Updated 01/30/2010 12:21

There are currently issues with the Shaw server. Our level 3 tecnicians are working on this, so please watch this space for further information

Update:

The Shaw server is actually `up' but there have been problems with faulty IPs so the IP pool is being rebuilt. We are currently waiting for an estimated timescale and will update again when we hear from the datacentre.

Update: This issue is now resolved.

cPanel Updates (Resolved) Medium

Affecting System - All servers

12/17/2009 08:00 - 12/18/2009 16:10
Last Updated 12/17/2009 08:01

We are currently updating the cPanel version on all servers to the latest stable release, during this time you may have intermittent problems when trying to access your cPanel whilst the upgrade is still in progress.

Domain Name Registry Problem (Resolved) Medium

Affecting System - Domain Names

11/19/2009 03:42 - 11/19/2009 11:02
Last Updated 11/19/2009 03:44

There is currently a network issue with the registry that is causing the following actions to fail:

- Registering / renewing domain names

- Changing Name Servers

The registry is aware of the problem and is working on bringing the system back online.

Apologies for any inconvenience caused.

Network Issues - Sites Down (Resolved) High

Affecting System - Entire Atlanta Network

10/16/2009 12:11
Last Updated 10/16/2009 17:08

You may have noticed your sites and email have been unavailable for around 1 hour earlier today, this was a network issue and now looks to be resolved. Will post more info as soon as we get it!

Update: The problem was down to a router failure

Sites not resolving (Resolved) High

Affecting Server - Bullseye

08/26/2009 03:30 - 08/26/2009 09:53
Last Updated 08/26/2009 09:53

Sites have stopped resolving on the Bullseye server after an emergency Kernel upgrade. We are working on this issue and will have the sites back online for you ASAP.

Update: The problem is due to the re-allocation of IP's in the block, the IP's which have been incorrectly allocated by the DC include the nameserver IP's as well as the main shared IP. We've escalated this to DC level to get the issue resolved.

In the meantime you can still access your email either via our webmail service:

https://bullseye.d9hosting.com/webmail

Username is the email address, password is the password for that email account.

Of if using an email client (Outlook, Outlook Express, etc) you can change the incoming and outgoing server names to: bullseye.d9hosting.com

More updates to follow.

Update: The network settings for the server have now been restored and all IP's now resolve as they should. Please let us know if you are still having issues and apologies for the inconvinience this has caused.

Network Issues on Hyrule Server (Resolved) Medium

Affecting Server - Hyrule

08/06/2009 10:58
Last Updated 08/06/2009 11:02

16.59 GMT There are currently issues on Hyrule server and some sites on this server are not resolving. We are working on this issue and will update as soon as possible.

16.01 GMT This issue is now resolved.

DNS Issue on Wilde Server (Resolved) Critical

Affecting Server - Wilde

07/24/2009 05:30 - 07/24/2009 13:08
Last Updated 07/24/2009 13:08

There is currently a DNS issue on the Wilde server, we are aware of the problem and are will update once we have more info.

The server is still up and running but no sites will be resolving.

*Update*

The DNS server is up and running, but some sites are showing an Internal Server Error message and some are not resolving at all. We are continue to work on the problem.

Thanks for your patience, we hope to have the sites back up and running ASAP!

*Update*

All sites should now resolve from the server, rather than bringing back a generic error message from your browser. They will however show a 500 Internal server error message, we are aware of this and technicians are still working on the issue.

The problem was caused when a site transfer went wrong and "messed up" (technical term!) the ownership rights of a number of core files.

*Update*

All non-PHP files (.html, images, etc.) are now resolving as normal. Still working on the PHP config.

*Update*

The issue is now resolved, if you continue to face problems with your site or email please get in touch with us. Thankyou for your patience, it's appreciated!

*Update*

There are still problems with the email server failing to come up, we are aware of the issue and are working to get your email back up and running ASAP.

*Update*

The mail issue is now resolved. Please contact us if the problem persists.

Hard Drive Replacement (Resolved) High

Affecting Server - Wilde

07/10/2009 14:53
Last Updated 07/10/2009 16:49

The backup hard drive on the Wilde server is dying, we are going to take the server down and replace the bad hard drive with a new one. Sites and email hosted on the Wilde server will be offline during this period.

Update: This has been completed and backup started to the new drive.

RAM Upgrade (Resolved) Medium

Affecting Server - Bullseye

07/09/2009 13:56
Last Updated 07/09/2009 16:25

The Bullseye server will shortly be taken offline so we can add some more RAM to the server, downtime it likely to be between 10 - 20 minutes.

Server Offline After Reboot (Resolved) Critical

Affecting Server - Wilde

06/25/2009 06:15 - 06/25/2009 08:21
Last Updated 06/25/2009 08:13

After performing graceful server reboot the server isn't coming back online, the issue is being looked into.

Update: The server is now back online.

Bullsye Taken Down For Investigation (Resolved) Critical

Affecting Server - Bullseye

06/23/2009 12:40 - 06/25/2009 03:53
Last Updated 06/25/2009 03:53

The server Bullseye will be taken down in the next few minutes whilst we check the config. Only 1 CPU is being detected so we need to see if it's a BIOS config problem, or a problem with the CPU itself.

Updates will follow.

**The issue has now been resolved**

Network Upgrade (Resolved) Medium

06/19/2009 00:00 - 06/20/2009 03:00
Last Updated 06/18/2009 03:54

On Friday (19 June 2009), we are going to perform an upgrade of our core router. That will improve the following:

- redundancy by helping position to remove single points of failure
- network response times

The expected downtime will be 20-35 minutes.

Apache Update (Resolved) Medium

Affecting Server - Bullseye

06/05/2009 04:58
Last Updated 06/07/2009 06:59

We are currently performing an Apache update on the Bullseye server, during this time websites may become unresponsive. We will update this issue when complete.

Server Down (Resolved) Critical

Affecting Server - Bullseye

04/28/2009 04:52
Last Updated 04/28/2009 05:55

The Bullseye server has stopped responding, we are aware of the problem and are currently investigating.

The Apache server encountered problems due to tempfs (a memory based system), we have fixed it and the server is up and running again.

Hyrule Port 80 Issue (Resolved) High

Affecting Server - Hyrule

04/06/2009 17:14
Last Updated 04/08/2009 12:24

There is currently an issue on the hyrule server preventing site visitors from accessing the http port (80) and this is causing your websites to appear as offline despite the server being up.

Our admins are currently aware of the issue and are working on the server. We will let you know when the problem is resolved.

Update: The problem is because of the shared IP address no longer resolving. Working with the Datacentre to correct the issue. Apologies for any convinience caused.

Update: This issue is now resolved

Update: The problem has returned this morning and we are investigating

Update: Sites are now resolving from the server again, we will keep a close eye on the server for the next few hours to see if the problem returns.

Update: Sites have been resolving on and off all day now so we are taking the serer down whilst a full analysis is performed. (16.28 GMT)

Update: Sites are now resolving the problem was due to a network misconfiguration.

Possible Hard Drive Replacement (Resolved) Medium

Affecting Server - Wilde

04/03/2009 09:42
Last Updated 04/04/2009 04:56

We received a warning message stating that one of the drives on the server may be having problems. We are scanning the drives for errors and if any are found the drive will be replaced.

Both drives have been checked and no errors have been found.

File System Check (Resolved) High

Affecting Server - Wilde

03/26/2009 04:37 - 03/29/2009 03:04
Last Updated 03/29/2009 15:05

**Latest Updates Posted At Bottom Of Page**

The "Wilde" server is currently experiencing a much higher load than normal, we are aware of the problem and are working to resolve it. We will update the network issues page with more info when it becomes available.

Update: The load average is now back to normal and all services are up and running, if you have any problem accessing any sites on the server please open a support ticket.

Update: The problem is re-appearing daily at around 9am GMT when the daily backups are running. Whilst the backups are resource intensive they shouldn't be causing the problems they are, so we are continuing to look into the problem. Stopping the backups from running would solve the problem, but we feel that it's better to continue to let them run whilst we look into the issue, that way we always have a recent backup of all your sites.

Update: A file system check is now going to be run within the next few hours, this will cause the server to be unreachable for the time when the file system check is running. We will keep this page updated with information.

Update: We thought we had found the problem last night when we found too many IPCS processes running on the server, we killed them but the problem re-occured this morning so we are still investigating. No downtime has occured as yet.

Update: The file system on the backup drive was extremely "messed up" (it's a technical term!) so we have reformated it. This should now resolve the problem.

Bullseye Downtime - LAN Card Replaced (Resolved) Critical

Affecting Server - Bullseye

01/22/2009 11:45 - 01/22/2009 13:45
Last Updated 01/22/2009 13:49

Users on the "Bullseye" server experienced around 3 hours of downtime on January 22nd.

After issuing a reboot request the server failed to come back online. After inspection by our onsite admins we found a LAN card was causing the problems. This was replaced and the server was brought back online.

cPanel/WHM Update (Resolved) Medium

Affecting System - All Servers

12/30/2008 08:22 - 12/30/2008 08:41
Last Updated 12/30/2008 08:23

We are currently updating cPanel/WHM to the latest RELEASE build, during the update users on all servers may be unable to access their cPanel/WHM. We will update this notice when the update is completed.

Server Down (Resolved) Critical

Affecting Server - Bullseye

12/30/2008 00:00 - 12/30/2008 00:00
Last Updated 12/30/2008 08:20

Users on the "Bullseye" server were unable to connect to websites and email. The server was rebooted and is currently functioning as normal.

Network Status

View

Support

Server Status

Support

HOSTING LINKS

Domain Names

HELP & SUPPORT

Other Stuff

Network Status

View

Support

Server Status

Support

HOSTING LINKS

Domain Names

HELP & SUPPORT

Other Stuff

Generate Password