EDT 8140 - LMS and Evaluation
In this page I will examine the following considerations to provide an evaluation of the two learning management systems, D2L and Blackboard Learn.
To prepare this report, my team interviewed LMS coordinators and administrators at a number of midsize to large universities in the Midwest. These individuals provided information and insights regarding their particular experiences. In some cases, the institutions had recently changed to the current LMS and so also could comment on the reasons they choose the system that they are currently using over other systems they considered. In addition I searched the internet for documents from institutions who used the D2L or Blackboard systems that were relevant to the five criteria we use for our evaluation. Information is provided in a side-by-side comparison for each category.
- Customer Satisfaction Level
- Training and Support
- Technical Support
- Costs and Hidden Costs
- Analytical Data Provided by LMS
To prepare this report, my team interviewed LMS coordinators and administrators at a number of midsize to large universities in the Midwest. These individuals provided information and insights regarding their particular experiences. In some cases, the institutions had recently changed to the current LMS and so also could comment on the reasons they choose the system that they are currently using over other systems they considered. In addition I searched the internet for documents from institutions who used the D2L or Blackboard systems that were relevant to the five criteria we use for our evaluation. Information is provided in a side-by-side comparison for each category.
1. Customer Satisfaction Level
Desire to Learn
The satisfaction level indicated by the LMS administrator at the Midwest university we interviewed was generally positive. Reviews posted on the Web also supported this view of strong customer satisfaction for the D2L LMS. Particular emphasis was placed on the smooth and well thought out migration steps from their previous LMS, and the gradebook facility. However there were several negative comments made. These included a Discussion board that was perceived to be hard to navigate, hidden costs needed to add on some features, and a lack of integration with third party vendors. |
Blackboard Learn
In general the LMS administrator whom we interviewed indicated that customer satisfaction at the institution was poor. This mirrors my impression of the system in the evaluations that I have performed working with the LMS to prepare this report. The system was not intuitive and while many features were available, they were not easily identified in the command structure. Many institutions currently with BB are moving to Canvas and the current institution interviewed for this report is considering a similar move. One independent review of Blackboard Learn indicated that the product was a leader in customer satisfaction. Another Web publication indicated 95% user satisfaction. However, this must be compared with the ratings for other LMS of 98% to 100% customer satisfaction ratings. |
2. Training and Support
Desire to Learn
One of university LMS administrators we interviewed mentioned that they had transitioned to D2L a few years ago from WebCT. At the time of the transition, D2L sent in training people to instruct the support staff in CTL on the new system. Since then, the support for faculty and students is delivered by the institution’s personnel who have received the original D2L training or were in turn, trained by others on the institution’s staff. The university administrator interviewed for this evaluation indicated the support was adequate for their current needs. |
Blackboard Learn
Training programs are available from Blackboard for organizations and students. These features come at additional costs. The majority of training on Blackboard seems to come from the individual institutions that use the LMS rather than from the company itself. We learned from one of our LMS administrator interviewees, that faculty at the university require approximately 8 hours each semester to keep up with changes to the system. |
3. Technical Support
Desire to Learn
The Midwest university using D2L had made some choices when installing their LMS which impacted their need for technical support. This institution hosts the LMS on site which means they are responsible for some of the technical support needed to maintain the system. This can have its benefits such as having more control of the data, assuring local backup of data, and a lower per annum cost (but see #4 below). However, the administrator also noted that upgrades had been a bad experience since they require three steps of evaluation (testing, development, and production) prior to incorporating the changes system wide. There also was an additional cost for the upgrades. A technical benefit for the D2L LMS is its ease of implementation with third party products, although as noted above this was not a universal attitude towards D2L. As the future for LMS systems may rely on an integrated system that incorporates features from many application providers, this technical feature can be viewed as making the LMS a versatile platform for customization as the needs of the institution change or as new functions are developed. |
Blackboard Learn
The LMS administrator we interviewed indicated technical support for the Blackboard LMS is a challenge for students and faculty. There are three levels of support; Student support (help desk) knowledge base, and Blackboard chat. My work with Blackboard Learn echoed this frustration with technical support. The online help tutorials or information pages were either not existent for the things I needed to know or were themselves difficult to follow. Most of the technical support that was available was derived from information pages posted by other institutions which use Blackboard Learn as their LMS. This limited the utility of the information since in some cases the steps were site specific. |
4. Costs and Hidden Costs
Desire to Learn
The Midwest university using the D2L system, enrolls approximately 15,000 students. With a three-year license for D2L, the annual cost is approximately $125,000. As upgrades become available, these cost an additional $5,000. There also was a one-time transition cost of approximately $10,000 when the university’s LMS was moved from WebCT to D2L. Note that this institution hosts the D2L servers onsite which, as noted above, can be a cost benefit. However, as the needs of the institution change and there are new developments in LMS capabilities, these may require significant upgrades in hardware. Thus the year-to-year savings which accrue from hosting the LMS onsite may be erased when significant cash outlays are necessary at the times upgrades become necessary. |
Blackboard Learn
The institutional LMS administrator whom we interviewed about Blackboard Learn hosts about 7,000 courses on the LMS. In addition the data are hosted offsite by Blackboard Learn on the cloud. For these services the institution pays $300,000 per year. Hosting the LMS offsite entails considerably greater costs, but may have an advantage for smaller institutions since there also are considerable in-house costs for personnel as well as the server and storage hardware needed to maintain and backup the data. In addition, as upgrades are developed for the software and hardware, there will be significant "one time" costs periodically for an in-house system which would not be required for a system hosted by the LMS vendor. |
5. Analytical Data Provided by LMS
Desire to Learn
A shortcoming of D2L which was mentioned by administrators and faculty was a lack of adequate analytical information provided by the D2L LMS. This feeling was driven in some instances by the high level of test analytical information that was provided by the former system used at the institution, WebCT. However, D2L is also lacking in metadata analytical information on student and faculty use of the LMS. |
Blackboard Learn
Analytical data are available on BB via a separate add-on product Blackboard analytics. One published study of this system suggested the program is in an early stage of development and in need of development in it's sophistication and features. Although there were ways to track student and faculty involvement with the LMS these features were not apparent to most users and required a separate in-house workshop to educate the faculty users of the system. |
LMS Weekly Recommendation
1. D2L gets the point over Blackboard Learn for this one. While each system has its own problem and neither is perfect, on measure, D2L is a step above Blackboard Learn on reported customer satisfaction. I share this judgement having worked through the various tasks to create this overall report.
Score: Blackboard – 0 : D2L – 1
2. The evaluation is about equal here. Support and training are available from either vendor for an additional cost at the time the system is installed on the host computer system. After then some of the training is handled in house for both systems.
Score : Blackboard – 1 : D2L – 1
3. D2L gets the extra point here. This is not because D2L has an excellent technical support program, but rather because the Blackboard Learn support is considerably lacking. This is the consensus of the majority of online comments made regarding Blackboard Learn, and is echoed by my own experience with developing the steps needed to perform the tasks outlined in this report.
Score : Blackboard – 0 : D2L – 1
4. The two systems are about equal in this measure of quality. Each provides a basic package but has many add-ons and upgrades that can add additional numbers to the bottom line. Getting the best value will depend on having a good negotiator to work with the LMS providers sales personnel.
Score: Blackboard – 1 : D2L – 1
5. Both systems do equally poorly in this measure. There are only the barest of information made available from the standard LMS installation. Blackboard Learn provides some analytical data using an additional program, but this requires an additional outlay of funds.
Score: Blackboard – 0 : D2L – 0
Final score:
Blackboard – 2 : D2L – 4
Overall, the D2L LMS outperformed Blackboard Learn on the "behind the scenes" assessments of system quality. Each could improve their score by introducing better analytical data capabilities. In addition the technical support could be significantly improved by both LMS companies.
1. D2L gets the point over Blackboard Learn for this one. While each system has its own problem and neither is perfect, on measure, D2L is a step above Blackboard Learn on reported customer satisfaction. I share this judgement having worked through the various tasks to create this overall report.
Score: Blackboard – 0 : D2L – 1
2. The evaluation is about equal here. Support and training are available from either vendor for an additional cost at the time the system is installed on the host computer system. After then some of the training is handled in house for both systems.
Score : Blackboard – 1 : D2L – 1
3. D2L gets the extra point here. This is not because D2L has an excellent technical support program, but rather because the Blackboard Learn support is considerably lacking. This is the consensus of the majority of online comments made regarding Blackboard Learn, and is echoed by my own experience with developing the steps needed to perform the tasks outlined in this report.
Score : Blackboard – 0 : D2L – 1
4. The two systems are about equal in this measure of quality. Each provides a basic package but has many add-ons and upgrades that can add additional numbers to the bottom line. Getting the best value will depend on having a good negotiator to work with the LMS providers sales personnel.
Score: Blackboard – 1 : D2L – 1
5. Both systems do equally poorly in this measure. There are only the barest of information made available from the standard LMS installation. Blackboard Learn provides some analytical data using an additional program, but this requires an additional outlay of funds.
Score: Blackboard – 0 : D2L – 0
Final score:
Blackboard – 2 : D2L – 4
Overall, the D2L LMS outperformed Blackboard Learn on the "behind the scenes" assessments of system quality. Each could improve their score by introducing better analytical data capabilities. In addition the technical support could be significantly improved by both LMS companies.