The Blackboard Academic Suite (Blackboard) ^[1] has been our campus e-Learning platform and University Portal since September 2005. It is the learning and communication hub for 30,000 energetic and creative students and staff, and provides integrations to essential administrative, messaging and specialized learning facilities. It is also a meeting place for organizations and groups, and a gathering spot for alumni and guests. With its popularity, more and more features and functions are being added to it, and it has become our mission critical system where performance and stability are expected.

The University is mindful of the user expectation and demand on the system and has been monitoring its usage pattern and performance since its inception. In 2006, a Blackboard server hardware capacity planning was carried out. The findings suggested that the Blackboard Database Server will be stretched out when our Application Servers take on more loads. These findings were reinforced by the results of the Performance Audit Exercise performed by the software vendor, Blackboard, in 2007 that the Database Server will reach its limit when the Application Severs are 75% utilized.

Thus, the discussion on upgrading the Blackboard Database Server hardware was initiated and there were various proposals which range from simply adding extra CPU power to the current hardware to deploying the latest-and-greatest engine. We focused our effort on finding the reliable technology with an adequate configuration that can integrate with the existing Blackboard hardware components (so as to secure our investment and optimize our technical expertise) as well as sustain the anticipated usage growth for the five years to come. We finally decided on the SUN Sparc Enterprise M9000 (M9000). The order was placed in September 2007 and the upgrade was planned for Christmas or the Chinese New Year break in February 2008 depending on its delivery and installation.

The decision to purchase the new database server hardware, the M9000, opened up enhancement opportunities for related Blackboard server software and hardware components including the upgrade of the operating system (OS) from Solaris 9 to Solaris 10 ^[2] , the upgrade of the High Availability System (HA) for the Database Server and the File Server from SUN Cluster 3.1 to SUN Cluster 3.2, the upgrade of the Database Software from Oracle 9i to Oracle 10g ^[3] , the provision of newer and more disk storage for users, and the upgrade of the Blackboard Academic Suite from Release 7.1 to Release 7.3. With all of the above in place, our Blackboard server configuration will be as a whole being upgraded from a vendor’s compliant server configuration to a certified server configuration with Level 1 Support (full support) from Blackboard. Moreover, it will be ready for future upgrades e.g. the Blackboard Academic Suite Release 8, where Release 7.3 is the prerequisite.

Our upfront challenge was the need to acquaint the user support staff and the system technical staff with the aforesaid new environment in three months, interesting enough even for the software and hardware vendors, some components were put together the first time. For the upgrade of the OS, server hardware and storage hardware, we worked closely with Sun Microsystems and EMC^{2 [4]} . The process of upgrading the OS from Solaris 9 to Solaris 10 for the Application Servers started in December 2007. Each of the four servers was taken off-line in turn for four days for upgrade and then being added back to the load-balanced pool. In this way we maintained three Application Servers in production and did not affect user service at all. We had even put a hold on the upgrade as such all four Application Servers were in service at peak usage periods during Course Registration and Grade Release. In the meantime the M9000 was delivered in mid December 2007 and the service downtime for system upgrade was confirmed for the Chinese New Year break on February 8^th. It was a critical and time pressing period. The M9000 was set up and installed with Solaris 10, Sun Cluster 3.2 and Oracle 10g and we had to speed learn the new systems and tools and then revise the existing ones, otherwise, the environment would not be operational.

Another important task was to test and clock the time needed for importing the 8GB of Oracle 9i Blackboard core database to Oracle 10g speedily and accurately. From our previous upgrade experience, it was the lengthiest process and its duration impacted the entire upgrade schedule and service downtime. The new EMC disk volume came later with the new data synchronization software, the Networker sym-clone, and we worked out how and how long it took to migrate the 600GB of user data files from the old disks to the new ones. There were lots of parallel tasks going on and eventually we had to put them together into a workable environment, for instance, we had to work through the connections from the Application Server to the new Database Server and File Server. To do so, one of the four production Application Servers was taken out from the load- balanced pool. This Application Server was reconfigured to connect to the new Database Cluster and new disk volume, followed by a substantial read/write test to ensure the data can be retrieved and stored correctly.

Alongside, our Blackboard development environment which has the same footprint as our production environment but less powerful was upgraded to Solaris 10, Oracle 10g and Blackboard Release 7.3. Every step was carefully thought through as such similar steps could be applied in the actual upgrade of the production environment. We even cloned the production data to the development environment to better understand the new features, system functions, and did system stress tests. Based on the results of the stress tests, the Consultants from Blackboard performed a configuration audit on the production environment and suggested some fine tunings that could better utilize Release 7.3 and the server hardware. At this end we had help from the Blackboard Professional Service and their local representative, BEENET ^[5] . Regular conference calls were conducted amongst the two parties and CityU (comprised the user support team from the Education Development Office and the system support team from the Computing Services Centre). It was a strenuous process for all parties and as we work through the problems, we found with Release 7.3 ^[6] some, unfortunately, were uncommon or trivial for other institutions but not for ours; an example of which would be the extensive use of Chinese characters in our courses.

With much effort and persistency, we re-did some critical tasks more than once to get the steps and timing right, adding checkpoints and contingencies. An upgrade schedule requiring 14 hours of service downtime was finally produced in which a test of the new HA system was included as it could only be proved functioning when all the pieces were connected and running.

On February 8^th, the Blackboard service was shutdown at 6:00am. The migration of the 8.3GB of core database data and the 600GB of user data files were done in parallel and completed in less than 3 hours. There was a hiccup with the new File Server which was not a problem during our earlier test. The engineers from SUN Microsystems and EMC² were called in to fixed the problem which had delayed the upgrade schedule by an hour. Thereafter, the Application Servers were connected to the new Database Server and File Sever clusters to proceed with the Blackboard Release 7.3 upgrade which ran smoothly. The upgrade was successful and normal service was resumed at 9:00pm the same day. An apparently lengthy upgrade, which took 15 hours from the end user point of view was, in fact, the result of months of careful planning, unfailing cooperation and dedicated hard work of the parties involved. Nevertheless, we are pleased with the good experience gained and the valuable lessons learned.

Appendices

Appendix A - Highlights on upgrading to Solaris 10

The High-Performance Networking Stack in Solaris 10 increases scalability, supporting more connections and enabling server network throughput. The Predictive Self-Healing can predict and detect faults on both hardware and software. The Process Rights Management provides a highly secured environment. Dynamic Tracing can show the details of the processes running and resources being used at any moment. Overall, Solaris 10 improves system performance, security and administration.

Appendix B - Highlights on upgrading to Oracle 10g

Oracle Database 10g provides new and enhanced tools for easier and more efficient monitoring, managing and tuning of a database. For instance, the Automatic Shared Memory Management speeds up server response. The Automatic Workload Repository (AWR) saves information about the utilization of system resources by various Oracle processes which are important for performance tuning. The Automatic Storage Management (ASM) simplifies the storage and administration of Oracle files.

Appendix C – A summary of Blackboard Academic Suite issues found by CityU

There were problems running the Blackboard 7.3 updater:

Problems with the Unique ID conversion program

Building Blocks compatibility with Release 7.3

CityU self-developed building blocks crash

Process ID location changed

Collaboration server cannot start

Minor bugs

--------------------------------------------------------------------------------

[1] The Blackboard Academic Suite is a product of Blackboard Inc., http://www.blackboard.com.

[2] See Appendix A for highlights on the advantages for upgrading to Solaris 10.

[3] See Appendix B for highlights on the advantages for upgrading to Oracle 10g.

[4] EMC² , http://www.emc.com.

[5] BEENET, http://www.bee-net.com

[6] See Appendix C for a summary of Blackboard Academic Suite Release 7.3 issues found.