Site Issues…

A number of our sites are currently having issues, including our own. We are working to resolve the issues with our maintenance team and will update this post as soon as possible.

21/02/10 – Power Maintenance

Our data centre in the Kent Science Park is going to be undertaking a large amount of power maintenance on the night of the 21st of February 2010.

During December, a specialist electrical testing company were contracted to perform a comprehensive analysis of the electrical infrastructure within the KSP. Infrared, ultrasonic and traditional circuit analysis was performed across the supply and distribution network and a report compiled. This analysis was undertaken to confirm the integrity of the electrical design and to ensure that there were no latent issues within the power infrastructure. It also facilitated the planning of any remedial changes that may be necessary to eliminate the possibility of a repeat power outage event of a similar nature occurring in the future. The technical analysis concluded that no critical issues existed. Progress was therefore made to the next stage which was planning the removal and replacement of the damaged post-UPS bus bar infrastructure and implementation of a suitable replacement.

We only recently received final details of the revised design, and instructions have now been issued to the electrical contractors to carry out these works. The new design removes the remaining core 250A and 400A bus bar distribution elements that are currently utilised after the N+1 UPS stacks. The final 100A bus bar distribution system which feeds each row of racks will remain in place, but this will be fed directly from a new 400A master UPS distribution panel.

To complete these works it will be necessary to shut down the power infrastructure so that connections on the UPS and bus bars can be changed over to the new distribution system. To minimise disruption to customers, the shutdown is scheduled to start at 00:15 on Sunday 21st February. A team of 5 electricians will be deployed concurrently to ensure a rapid transfer to the new infrastructure. Because of the scale and complexity of the tasks to be completed the anticipated down-time is approximately 4 hours.

[Resolved] Server reboots

We are currently performing kernel compiles to fix the recent R1Soft problem. Once completed this will require a reboot and have scheduled this for midnight tonight (28th January 2010).

You may have also noticed that the server has been under high load and your webstats have not been working correctly. This is due to the log process being stuck which will also be fixed by the reboot.

The downtime will be around 5 minutes while the server is rebooted and services come back online. We have invested heavily in KVM units over the past few weeks so if a problem arises during the reboot we can quickly resolve the issue.

[Resolved] Apache Upgrade

We are currently upgrading Apache and PHP which may cause a few minutes of website downtime while the service is compiled and restarts. This is expected to last around 15 minutes.

Apache is being upgraded to 2.2 to provide better support for future Subversion and Trac features as well as improved compatibility with Litespeed 4.0.12

Update: The compile is taking longer than expected however PHP is now installing which is one of the final steps. Estimated time remaining is 10 minutes.

[Resolved] R1Soft Backup Issues…

We are currently experiencing issues with our R1Soft backup system. The backup agent that resides on each server is unable to connect to our backup server which means we are not able to take our usual backup four times a day.

We’re currently working with R1Soft to get this resolved and hope to have it working again soon. We will be backing up our servers to a remote location so your data is safe, this just means that individual files cannot be restored via the control panel during this time.

It is also recommended that you keep a full backup of your website on your local computer incase you need to restore any files.

[Resolved] Emergency Server Reboot…

A recent bug within the CentOS kernel has caused two of our servers to freeze due to the R1Soft backups. To stop this happening again we have compiled custom kernels which require a reboot to be applied.

We will start rebooting servers at midnight tonight, reboots will be staggered to avoid the DNS cluster going offline and each server will be down for around 10 minutes.

Some servers have not been affected by this kernel bug yet but rather than wait for each server to crash we’ve applied the patch to prevent extended periods of downtime in the future.

If you have any questions or queries, please feel free to contact us via the support help desk.

[Resolved] UK Servers down…

The data centres that our servers are located in his currently having a power outage, as a result all of the servers are currently down.

We’ll provide more feedback when it’s available.

Update @ 13:00:

We’ve been informed that the reason for the outage is exactly the same as last time i.e. due to a failure of power equipment resulting in a short of the UPS systems.

There are currently multiple specialist engineers onsite both repairing the issue whilst trying to get to the bottom of why this is happening.

More information to follow shortly.

Update @ 14:25:

Power to the UPS systems have been restored, however theres a software issue that needs to be resolved. We expect full service restoration shortly.

Update @ 14:59: Servers are now powered up and are coming back online.

[Resolved/Finished] HostThree.net – Urgent maintenance

We have taken the HostThree.net website down for urgent maintenance on our site. We appoligise for any inconvenience caused and request any support emails be sent to admin [at] scotmediagroup.net.

Thank you.
ScotMedia Group Admin.

[Resolved] Server Issues…

We seem to be having a couple of issues with MySQL/Database applications, we are currently looking into the issue and will update you as soon as possible.

Update: Servers have been having some issues, currently doing a FSCK – 35% so far

Update: Servers up and running, sorry for the issues

Brand new Blogging product!

The ScotMedia Group and HostThree is proud to release our brand new blogging product, BlogThree.co.uk!

For just £2.35 a month you can now have your very own blogging site that will be installed and maintained by our very own blogging experts.

Visit – www.blogthree.co.uk for more details