Upgrading
the Blackboard Academic Suite in "15 hours"
By
Maria Chin
|
|
|
The Blackboard Academic Suite (Blackboard)
[1] has been our campus e-Learning
platform and University Portal since September 2005. It is the
learning and communication hub for 30,000 energetic and creative
students and staff, and provides integrations to essential administrative,
messaging and specialized learning facilities. It is also a meeting
place for organizations and groups, and a gathering spot for alumni
and guests. With its popularity, more and more features and functions
are being added to it, and it has become our mission critical
system where performance and stability are expected.
The University is mindful of the user expectation and demand
on the system and has been monitoring its usage pattern and performance
since its inception. In 2006, a Blackboard server hardware capacity
planning was carried out. The findings suggested that the Blackboard
Database Server will be stretched out when our Application Servers
take on more loads. These findings were reinforced by the results
of the Performance Audit Exercise performed by the software vendor,
Blackboard, in 2007 that the Database Server will reach its limit
when the Application Severs are 75% utilized.
Thus, the discussion on upgrading the Blackboard Database Server
hardware was initiated and there were various proposals which
range from simply adding extra CPU power to the current hardware
to deploying the latest-and-greatest engine. We focused our effort
on finding the reliable technology with an adequate configuration
that can integrate with the existing Blackboard hardware components
(so as to secure our investment and optimize our technical expertise)
as well as sustain the anticipated usage growth for the five years
to come. We finally decided on the SUN Sparc Enterprise M9000
(M9000). The order was placed in September 2007 and the upgrade
was planned for Christmas or the Chinese New Year break in February
2008 depending on its delivery and installation.
The decision to purchase the new database server hardware, the
M9000, opened up enhancement opportunities for related Blackboard
server software and hardware components including the upgrade
of the operating system (OS) from Solaris 9 to Solaris 10
[2] , the upgrade of the High
Availability System (HA) for the Database Server and the File
Server from SUN Cluster 3.1 to SUN Cluster 3.2, the upgrade of
the Database Software from Oracle 9i to Oracle 10g
[3] , the provision of newer
and more disk storage for users, and the upgrade of the Blackboard
Academic Suite from Release 7.1 to Release 7.3. With all of the
above in place, our Blackboard server configuration will be as
a whole being upgraded from a vendor’s compliant server configuration
to a certified server configuration with Level 1 Support (full
support) from Blackboard. Moreover, it will be ready for future
upgrades e.g. the Blackboard Academic Suite Release 8, where Release
7.3 is the prerequisite.
Our upfront challenge was the need to acquaint the user support
staff and the system technical staff with the aforesaid new environment
in three months, interesting enough even for the software and
hardware vendors, some components were put together the first
time. For the upgrade of the OS, server hardware and storage hardware,
we worked closely with Sun Microsystems and EMC2 [4]
. The process of upgrading the OS from Solaris 9 to Solaris 10
for the Application Servers started in December 2007. Each of
the four servers was taken off-line in turn for four days for
upgrade and then being added back to the load-balanced pool. In
this way we maintained three Application Servers in production
and did not affect user service at all. We had even put a hold
on the upgrade as such all four Application Servers were in service
at peak usage periods during Course Registration and Grade Release.
In the meantime the M9000 was delivered in mid December 2007 and
the service downtime for system upgrade was confirmed for the
Chinese New Year break on February 8th. It was a critical
and time pressing period. The M9000 was set up and installed with
Solaris 10, Sun Cluster 3.2 and Oracle 10g and we had to speed
learn the new systems and tools and then revise the existing ones,
otherwise, the environment would not be operational.
Another important task was to test and clock the time needed
for importing the 8GB of Oracle 9i Blackboard core database to
Oracle 10g speedily and accurately. From our previous upgrade
experience, it was the lengthiest process and its duration impacted
the entire upgrade schedule and service downtime. The new EMC
disk volume came later with the new data synchronization software,
the Networker sym-clone, and we worked out how and how long it
took to migrate the 600GB of user data files from the old disks
to the new ones. There were lots of parallel tasks going on and
eventually we had to put them together into a workable environment,
for instance, we had to work through the connections from the
Application Server to the new Database Server and File Server.
To do so, one of the four production Application Servers was taken
out from the load- balanced pool. This Application Server was
reconfigured to connect to the new Database Cluster and new disk
volume, followed by a substantial read/write test to ensure the
data can be retrieved and stored correctly.
Alongside, our Blackboard development environment which has the
same footprint as our production environment but less powerful
was upgraded to Solaris 10, Oracle 10g and Blackboard Release
7.3. Every step was carefully thought through as such similar
steps could be applied in the actual upgrade of the production
environment. We even cloned the production data to the development
environment to better understand the new features, system functions,
and did system stress tests. Based on the results of the stress
tests, the Consultants from Blackboard performed a configuration
audit on the production environment and suggested some fine tunings
that could better utilize Release 7.3 and the server hardware.
At this end we had help from the Blackboard Professional Service
and their local representative, BEENET [5]
. Regular conference calls were conducted amongst the two
parties and CityU (comprised the user support team from the Education
Development Office and the system support team from the Computing
Services Centre). It was a strenuous process for all parties and
as we work through the problems, we found with Release 7.3
[6] some, unfortunately, were
uncommon or trivial for other institutions but not for ours; an
example of which would be the extensive use of Chinese characters
in our courses.
With much effort and persistency, we re-did some critical tasks
more than once to get the steps and timing right, adding checkpoints
and contingencies. An upgrade schedule requiring 14 hours of service
downtime was finally produced in which a test of the new HA system
was included as it could only be proved functioning when all the
pieces were connected and running.
On February 8th, the Blackboard service was shutdown
at 6:00am. The migration of the 8.3GB of core database data and
the 600GB of user data files were done in parallel and completed
in less than 3 hours. There was a hiccup with the new File Server
which was not a problem during our earlier test. The engineers
from SUN Microsystems and EMC2 were called in to fixed
the problem which had delayed the upgrade schedule by an hour.
Thereafter, the Application Servers were connected to the new
Database Server and File Sever clusters to proceed with the Blackboard
Release 7.3 upgrade which ran smoothly. The upgrade was successful
and normal service was resumed at 9:00pm the same day. An apparently
lengthy upgrade, which took 15 hours from the end user point of
view was, in fact, the result of months of careful planning, unfailing
cooperation and dedicated hard work of the parties involved. Nevertheless,
we are pleased with the good experience gained and the valuable
lessons learned.
Appendices
Appendix A -
Highlights on upgrading to Solaris 10
The High-Performance Networking Stack in Solaris 10 increases
scalability, supporting more connections and enabling server network
throughput. The Predictive Self-Healing can predict and detect
faults on both hardware and software. The Process Rights Management
provides a highly secured environment. Dynamic Tracing can show
the details of the processes running and resources being used
at any moment. Overall, Solaris 10 improves system performance,
security and administration.
Appendix B -
Highlights on upgrading to Oracle 10g
Oracle Database 10g provides new and enhanced tools for easier
and more efficient monitoring, managing and tuning of a database.
For instance, the Automatic Shared Memory Management speeds up
server response. The Automatic Workload Repository (AWR) saves
information about the utilization of system resources by various
Oracle processes which are important for performance tuning. The
Automatic Storage Management (ASM) simplifies the storage and
administration of Oracle files.
Appendix C –
A summary of Blackboard Academic Suite issues found by CityU
There were problems running the Blackboard 7.3 updater:
