During the past couple of months the QuickServe Online website has experienced an unusually high amount of downtime. I would
like to give you an update on what has been going on, why we've had problems and improvement plans that are in progress.
QSOL Server Upgrade Project Launched in September, 2006
In September we launched phase 1 of a multi-phased project to completely upgrade all servers and software on QuickServe Online.
Our existing servers are going on 2 1/2 years old, and it is time to move to newer machines. Our web traffic has tripled
in the last two years, which is a very extreme growth in usage, and the current servers have been struggling to keep up in
the last 2 months.
This upgrade project is a major project. Below are the project elements and timeline:
|
Project Element
|
Jul |
Aug |
Sep |
Oct |
Nov |
Dec |
Jan |
Feb |
Mar |
Apr |
May |
| Test new Oracle software upgrades in development |
X
|
X
|
|
|
|
|
|
|
|
|
|
| Install new Oracle database cluster to production |
|
X
|
X
|
|
|
|
|
|
|
|
|
| Move remaining cluster applications from old CIS cluster to new cluster |
|
|
|
X
|
X
|
|
|
|
|
|
|
| Test webserver software upgrades in development |
|
|
|
|
|
X
|
X
|
|
|
|
|
| Launch 6 new production web servers to production |
|
|
|
|
|
|
X
|
X
|
|
|
|
| Test and install new staging servers |
|
|
|
|
|
|
|
X
|
X
|
|
|
| Test and install new load balancer |
|
|
|
|
|
|
|
|
X
|
X
|
|
| Test and install new firewall application |
|
|
|
|
|
|
|
|
|
X
|
X
|
In September we completed the installation of the Oracle cluster database. Since then we have experienced intermittent problems
related to the database. In each case the problem has been identified and a permanent fix implemented.
The most recent problem experienced this week on Nov 14 and 15 was a software bug with the Oracle database cluster. A bug
was discovered in the new version of Oracle, causing the database to shut down. We are in the process of installing software
patches to correct this bug.
There are basically two high level causes for the recent downtimes we have experienced.
- Start up problems with the new Oracle cluster.
- Intermittent slowness and problems with webservers due to the heavy traffic load.
To summarize,
- Website usage has grown much faster than we had anticipated in the last year.
- We have started a major project to upgrade all hardware / software which will greatly improve performance and capacity of
our servers. This project was planned back in Q4-05, and was launched on schedule in the fall of 2006.
- We have completed the Oracle database upgrades.
- We are experiencing some problems with Oracle right now and are putting solutions in place to permanently correct the problems.
- In December we will be preparing for the webserver upgrades that will take place in January. Once the new webservers are
in place, we expect to have much greater capacity for website traffic.
Rest assured that our group is monitoring website performance 24/7. We have an excellent monitoring system in place that
lets us know when servers are beginning to have problems. Through our monitoring tools, we know about the problems well before
the servers actually go down, and before customers see the problem. Our expectation is to have 98% uptime worldwide and we
are working very hard to meet that objective. We are planning these server upgrades in very deliberate phases so that there
is no impact to customers. We will not cut any corners or rush the project - we are doing it right so as to minimize any performance
risk during the transitions. Unfortunately right now we are experiencing growing pains as more and more users register for
QSOL across the world.
We apologize for the inconvenience the QSOL issues have caused you and your business. We have very capable people on the
job who are working very hard to bring you the best website performance possible.
Best regards,
Kathy Oren
Business Manager, eBusiness
