Upgrading the Blackboard Academic Suite in "15 hours"
The Blackboard Academic Suite (Blackboard)  has been our campus e-Learning platform and University Portal since September 2005. It is the learning and communication hub for 30,000 energetic and creative students and staff, and provides integrations to essential administrative, messaging and specialized learning facilities. It is also a meeting place for organizations and groups, and a gathering spot for alumni and guests. With its popularity, more and more features and functions are being added to it, and it has become our mission critical system where performance and stability are expected.
The University is mindful of the user expectation and demand on the system and has been monitoring its usage pattern and performance since its inception. In 2006, a Blackboard server hardware capacity planning was carried out. The findings suggested that the Blackboard Database Server will be stretched out when our Application Servers take on more loads. These findings were reinforced by the results of the Performance Audit Exercise performed by the software vendor, Blackboard, in 2007 that the Database Server will reach its limit when the Application Severs are 75% utilized.
Thus, the discussion on upgrading the Blackboard Database Server hardware was initiated and there were various proposals which range from simply adding extra CPU power to the current hardware to deploying the latest-and-greatest engine. We focused our effort on finding the reliable technology with an adequate configuration that can integrate with the existing Blackboard hardware components (so as to secure our investment and optimize our technical expertise) as well as sustain the anticipated usage growth for the five years to come. We finally decided on the SUN Sparc Enterprise M9000 (M9000). The order was placed in September 2007 and the upgrade was planned for Christmas or the Chinese New Year break in February 2008 depending on its delivery and installation.
The decision to purchase the new database server hardware, the M9000, opened up enhancement opportunities for related Blackboard server software and hardware components including the upgrade of the operating system (OS) from Solaris 9 to Solaris 10  , the upgrade of the High Availability System (HA) for the Database Server and the File Server from SUN Cluster 3.1 to SUN Cluster 3.2, the upgrade of the Database Software from Oracle 9i to Oracle 10g  , the provision of newer and more disk storage for users, and the upgrade of the Blackboard Academic Suite from Release 7.1 to Release 7.3. With all of the above in place, our Blackboard server configuration will be as a whole being upgraded from a vendor’s compliant server configuration to a certified server configuration with Level 1 Support (full support) from Blackboard. Moreover, it will be ready for future upgrades e.g. the Blackboard Academic Suite Release 8, where Release 7.3 is the prerequisite.
Our upfront challenge was the need to acquaint the user support staff and the system technical staff with the aforesaid new environment in three months, interesting enough even for the software and hardware vendors, some components were put together the first time. For the upgrade of the OS, server hardware and storage hardware, we worked closely with Sun Microsystems and EMC2  . The process of upgrading the OS from Solaris 9 to Solaris 10 for the Application Servers started in December 2007. Each of the four servers was taken off-line in turn for four days for upgrade and then being added back to the load-balanced pool. In this way we maintained three Application Servers in production and did not affect user service at all. We had even put a hold on the upgrade as such all four Application Servers were in service at peak usage periods during Course Registration and Grade Release. In the meantime the M9000 was delivered in mid December 2007 and the service downtime for system upgrade was confirmed for the Chinese New Year break on February 8th. It was a critical and time pressing period. The M9000 was set up and installed with Solaris 10, Sun Cluster 3.2 and Oracle 10g and we had to speed learn the new systems and tools and then revise the existing ones, otherwise, the environment would not be operational.
Another important task was to test and clock the time needed for importing the 8GB of Oracle 9i Blackboard core database to Oracle 10g speedily and accurately. From our previous upgrade experience, it was the lengthiest process and its duration impacted the entire upgrade schedule and service downtime. The new EMC disk volume came later with the new data synchronization software, the Networker sym-clone, and we worked out how and how long it took to migrate the 600GB of user data files from the old disks to the new ones. There were lots of parallel tasks going on and eventually we had to put them together into a workable environment, for instance, we had to work through the connections from the Application Server to the new Database Server and File Server. To do so, one of the four production Application Servers was taken out from the load- balanced pool. This Application Server was reconfigured to connect to the new Database Cluster and new disk volume, followed by a substantial read/write test to ensure the data can be retrieved and stored correctly.
Alongside, our Blackboard development environment which has the same footprint as our production environment but less powerful was upgraded to Solaris 10, Oracle 10g and Blackboard Release 7.3. Every step was carefully thought through as such similar steps could be applied in the actual upgrade of the production environment. We even cloned the production data to the development environment to better understand the new features, system functions, and did system stress tests. Based on the results of the stress tests, the Consultants from Blackboard performed a configuration audit on the production environment and suggested some fine tunings that could better utilize Release 7.3 and the server hardware. At this end we had help from the Blackboard Professional Service and their local representative, BEENET  . Regular conference calls were conducted amongst the two parties and CityU (comprised the user support team from the Education Development Office and the system support team from the Computing Services Centre). It was a strenuous process for all parties and as we work through the problems, we found with Release 7.3  some, unfortunately, were uncommon or trivial for other institutions but not for ours; an example of which would be the extensive use of Chinese characters in our courses.
With much effort and persistency, we re-did some critical tasks more than once to get the steps and timing right, adding checkpoints and contingencies. An upgrade schedule requiring 14 hours of service downtime was finally produced in which a test of the new HA system was included as it could only be proved functioning when all the pieces were connected and running.
On February 8th, the Blackboard service was shutdown at 6:00am. The migration of the 8.3GB of core database data and the 600GB of user data files were done in parallel and completed in less than 3 hours. There was a hiccup with the new File Server which was not a problem during our earlier test. The engineers from SUN Microsystems and EMC2 were called in to fixed the problem which had delayed the upgrade schedule by an hour. Thereafter, the Application Servers were connected to the new Database Server and File Sever clusters to proceed with the Blackboard Release 7.3 upgrade which ran smoothly. The upgrade was successful and normal service was resumed at 9:00pm the same day. An apparently lengthy upgrade, which took 15 hours from the end user point of view was, in fact, the result of months of careful planning, unfailing cooperation and dedicated hard work of the parties involved. Nevertheless, we are pleased with the good experience gained and the valuable lessons learned.
The High-Performance Networking Stack in Solaris 10 increases scalability, supporting more connections and enabling server network throughput. The Predictive Self-Healing can predict and detect faults on both hardware and software. The Process Rights Management provides a highly secured environment. Dynamic Tracing can show the details of the processes running and resources being used at any moment. Overall, Solaris 10 improves system performance, security and administration.
Oracle Database 10g provides new and enhanced tools for easier and more efficient monitoring, managing and tuning of a database. For instance, the Automatic Shared Memory Management speeds up server response. The Automatic Workload Repository (AWR) saves information about the utilization of system resources by various Oracle processes which are important for performance tuning. The Automatic Storage Management (ASM) simplifies the storage and administration of Oracle files.
There were problems running the Blackboard 7.3 updater:
The updater failed to run if Java 1.6 is used.
The required database Character Set changed from UTF-8 to UTF-16. Conversion of Oracle 9i data to Oracle 10g data had to be re-run.
New database table could not be created because of Oracle 10g allows less privilege by default.
The Oracle 10g must be installed under the same account as the old Oracle 9i, otherwise a permission denied error will occur.
Problems with the Unique ID conversion program
- Unique ID is a new feature in Release 7.3. For each item in the Content System, a unique ID is assigned by which the item can be linked up independent of where the item is stored. The advantage is that items are free to move to other folders and the link by unique ID still works which cannot be done in earlier Releases. A conversion tool is provided by Blackboard to create unique ID for each old item and modify existing links to use the new unique ID instead of the old virtual path. However, the conversion tool cannot convert properly when the item name or path contains special characters like space, bracket and Chinese. After running the conversion tool, there will be a mixture of items with and without unique ID. After discussion among Blackboard consultants, the EDO and the CSC colleagues, it was decided not to run the unique ID conversion tool, because it is most clear to the end users that all items before the upgrade do not have unique IDs, only items newly uploaded after the upgrade will have unique IDs.
Building Blocks compatibility with Release 7.3
- The building blocks of Pronto, Wimba, RefWorks, SafeAssign, Health Check and Campus Pack being used in Blackboard 7.1 do not work with Blackboard 7.3. Newer versions of those building blocks have to be re-installed.
CityU self-developed building blocks crash
- There were basic changes in Blackboard 7.3 API. CityU developers had to re-test their written building blocks. Some were found crashing with Blackboard and had to be re-written.
Process ID location changed
- The locations for storing Process IDs of the Apache web, Tomcat java and Virtual Classroom/Chat java have been changed. Scripts for monitoring those processes have to be revised.
Collaboration server cannot start
- The Collaboration server cannot be started after upgrading to Release 7.3. It was found that configuring SSL for the collaboration server is different from Release 7.1. The problem was fixed after applying the new configuration method.
- Some minor bugs with discussion board, backpack, course import, batch group were found and reported to Blackboard. Service Pack 1 for Blackboard Release 7.3 was announced on 21 December 2007. It was immediately applied to the development environment for re-testing.