It is the cache of ${baseHref}. It is a snapshot of the page. The current page could have changed in the meantime.
Tip: To quickly find your search term on this page, press Ctrl+F or ⌘-F (Mac) and use the find bar.

Open Source Software: Fully featured vs. "the devil you know"? | Hotrum | The International Review of Research in Open and Distance Learning
IRRODL: 43. Open source software: fully featured vs. “the devil you know”

March - 2005

Technical Evaluation Report

43. Open Source Software: Fully featured vs. “the devil you know”

Michael Hotrum and Brian Ludwig
Masters of Distance Education Programme 
Athabasca University – Canada’s Open University

Jon Baggaley, PhD 
Athabasca University – Canada’s Open University

Abstract

The ILIAS learning management system (LMS) was evaluated, following its favourable rating in an independent evaluation study of open source software (OSS) products. The current review found ILIAS to have numerous features of value to distance education (DE) students and teachers, as well as problems for consideration in the system’s ongoing development. The current findings were compared with DE students’ reactions to a similar LMS product, ATutor, also rated highly in the independent OSS evaluation. In comparing an ATutor course website with a simple HTML-based version of the same site, the ten students voted unanimously to retain the simpler site. This result is consistent with previous evaluation findings in the current series of reports, and indicates that increasing integration of product features does not necessarily improve a product’s ease of use or educational effectiveness.

Introduction

As the current series of reports has indicated, the number of online learning management systems (LMS) has proliferated, and at least a hundred are currently available. Leslie (2004) has identified over 50 such systems based on open source software (OSS) alone. In evaluating 35 OSS systems, Farrell (2003) has rated ATutor and ILIAS as the best, with Moodle close behind. The current evaluation studies have so far evaluated Moodle (Report #25) and ATutor (Report #37), rating both products favourably with respect to their potential value in distance education (DE). The current report completes this trilogy of reviews by examining the DE merits of Ilias. In deciding the criteria for its review, the evaluation team reviewed the LMS selection standards used by (1) the American Society for Training & Development (ASTD: see previous reports in this series); (2) the Commonwealth of Learning evaluation study by Farrell (2003); and (3) the software comparison service, Edutools.info (2003). The criteria considered the most appropriate were combined and some new ones added. The final set of criteria is considered to represent the judgments that administrators, facilitators, designers and end users need to make in determining the value and usefulness of an LMS in DE. In addition, previous online opinions of ILIAS were considered in the light of the version currently reviewed (v. 3.2.0.). Following the ILIAS review, the report also examines the trends that have evolved in the review series so far, with respect to DE students’ reactions to complex, fully-featured LMS platforms, in comparison with those described in Report #8 as “lean and mean”. In keeping with the convention adopted earlier in this series (Report #25), the LMS acronym is adopted in order to avoid the confusion of terms currently arising in the educational software literature.

Evaluation of Ilias

ILIAS is a free, open source (OS) Web-based learning management system (LMS) developed at the University of Cologne (2004). Originally released in July 2001, with early versions dating from mid-2000, ILIAS stands for Integrated Learning, Information and Co-Operative Work System. It has been designed as a collaborative, hypermedia working/ learning environment, flexible for use by teachers with different pedagogical approaches, and allowing the reuse and exchange of materials based on course-sharing standards (e.g., IMS, The Aviation Industry CBT [Computer-Based Training] Committee (AICC) and ARIADNE Foundation for the European Knowledge Pool). As the previous report in the series (Report # 42) has indicated, all OSS is by definition a work in progress, constantly enhanced and modified as a result of user input and collaboration. ILIAS is no exception. Its website currently reports its use on over 100 servers, mainly at Universities in Europe and Asia. In the OSS sharing spirit, therefore, the current evaluation team contacted the four listed North American users with inquiries about their ILIAS experience (though none responded).

Particular aspects of ILIAS that support the moderate constructivism philosophy are:

  • Team based authoring: ILIAS allows for the collaborative development of course materials.

  • Personal annotations: ILIAS offers a personal context-related annotation function that permits a learner to add summaries, comments and annotations to text elements, graphics or images. These annotations can be retrieved at any time within the original context, printed out in a structured form, and exchanged with members of a learning group.

  • Personal bookmark folder: learners can build and administer content-related resources available from the Web.

  • Learning as a self-directed process: users can choose the learning materials to review, when to review them, and the sequence of materials. The materials are accessed from a central database which only displays the content relevant for the learner’s current concerns and focus. A search engine allows the user to seek specific content within the materials.

  • Learning is a social process: facilitators and learners can create groups of users and define access privileges.

  • Text elements are assumed to achieve the best educational effect for differently targeted learning groups: the text elements form the ‘backbone’ of an ILIAS course. Other elements (e.g., animations, interactive elements, co-operative elements, graphic, audio, and video) support this basic structure. Once the elements have developed and integrated into an ILIAS-based course, the author is free to associate and link them based on a specific pedagogical approach.

ILIAS is a cross-browser system, requiring Internet Explorer 6.0, Mozilla 1.4, Safari 1.0, and/ or Lynx 2.8.4 (or higher), and runs with JavaScript disabled. The system is frames-based, cannot support text-only navigation, and is not usable with large fonts. It requires MySQL 4.0.16 or higher. A digest of the product’s features is as follows:

  • Text discussions (viewable in threads and associated with course content)

  • Integrated real-time text chat (java-based Babylon)

  • File transfer available between users

  • Small-group discussion environments (created by teachers and students)

  • Each group can have its own discussion forum, chat room, and file exchange

  • Web-based mailing system with usermail, groupmail, and integrated send mail functions

  • Personal student homepage (desktop) linking to courses, groups, bookmarks, and email

  • Students can attach notes to displays, and can search and compile them for the course as a whole

  • Course content is downloadable for offline work and local printing/ storage

  • A printable study guide can be generated

  • Polls can be administered with online and print-based statistical reports

  • Teachers can create self-assessment tests (multiple-choice and cloze) with automatic scoring

  • Teachers can control a student’s progress through course modules based on test results

  • Students can use specified criteria to compile a collection of exercises from an item test bank

  • Student exercises can yield a feedback report of correct and incorrect responses

  • Administrators can set courses to be publicly accessible or can restrict access by ID and password (author, learner, administrator, and guest levels)

  • Specific tasks can be assigned at each access level

  • Users can maintain their own passwords

  • Detailed reports tracking access to course materials, though not for individual users

  • Integration with institutional IT architectures via LDAP/ RADIUS authentication

  • Third party hosted services are available (in German only)

  • No whiteboard (scheduled for next version)

ILIAS currently suffers from limitations common to many OSS products. These include: lack of interoperability and integration; lack of support for audio/ video-conferencing, time-zone or date localization; poor accessibility standards; and an unclear estimate of the true cost of ownership. Currently, many OS LMS platforms lack integration with the applications typically used for registration and employee management in universities, colleges, and industry (Bailetti, 2004). The same writer also notes that OSS systems tend to lack good user, developer and administration documentation; and this is true of ILIAS. Although the software contains a 13-language translation facility, up-to-date documentation for it is available only in German; non-German users have little chance of influencing the product’s development via participation in the various help forums. (On the other hand, Farrell (2003) notes a growing following for the product in China.) While ILIAS is classified as freeware, its use is not actually cost-free. The time spent in installing and maintaining the system must be added to the costs of the hardware, technical support and training; and the current evaluators (both experienced server administrators) found the logistical problems of ILIAS installation to be considerable. They spent several days trying to install and configure it on Linux, in conjunction with its dozen or more required third party applications. Poor English documentation contained conflicting instructions. As a last resort, the messages in the German-language ILIAS user forums were scoured, until an English description of a Windows-based installation alternative was found. With this, ILIAS was working on a Windows/2000 server within 30 minutes. Despite its many current shortcomings, it has an admirable range of features, pedagogical integrity, and good potential for DE users. OSS applications such as this have come a long way within the three years of the current evaluation series, and they are continuing to advance quickly.

Previous reports in this series have indicated, however, that fully-featured online tools do not necessarily find favour with DE users. Students can find them difficult to master, and DE teachers do not necessarily share a need for the many features they offer. In addition, their integration with existing institutional systems can be expensive, and the overall costs of proprietary systems especially, may be excessive in the relation to the actual uses to which they are subjected. In Report #8 in this series, a direct comparison was made between DE students reactions to a fully-featured conferencing product and a simple ‘bare-bones’ alternative. Unexpectedly, a majority of students preferred the simple application, and regarded a transfer to the more complex option as not worth the effort. In the two years since that study, online collaborative tools have converged within ambitious integrated products such as WebCT Vista, and in OSS applications (e.g., ATutor, ILIAS, and Moodle). Having now established that each of these products has a wide range of features of use to DE teachers and students, the next section of this report replicates the earlier study, by asking DE students to compare the comprehensive open-source ATutor product with the more limited HTML-based site of their current online course.

Comparison of Course Website Formats

Ten students in a graduate Masters in Distance Education (MDE) course at Athabasca University – Canada’s Open University, were asked to compare the simple, home-grown, frames-based format of their current course website (see Figure 1) with that of the OSS LMS ATutor. The content of the course site was replicated in ATutor (see Figure 2), using the free trial and hosting facilities of the product’s website. ATutor was selected for this purpose in view of its top rating by Farrell (2003), and its favourable evaluation in Report #37 of this series. The current study fulfilled a promise stated in that report, that ATutor would be adopted for the MDE course depending on feedback from its current students as a whole.

Figure 1. The frames-based format of an online Master’s in Distance Education course website

 

Figure 2. A sample configuration of ATutor (same course as Figure 1)

 

A representative selection of the students’ responses to the two course platforms is as follows.

Student A:

  • I am very impressed with ATutor. Based on the available tools and the initial appearance, it appears to have all of the features of WebCT . . . plus much more customization.

  • For the most part, navigation seems very intuitive, particularly at the higher levels. Once I get into the lower menu levels such as at this step of posting a message, the green and red circles confused me . . .

  • I did not like the built-in email feature in WebCT, nor in this program either. I prefer using my regular email program. Having this feature just means I have another email box to check.

  • The chat is a nice tool, but I would rather use something like Yahoo Messenger. This chat only works if you are logged into the course. Yahoo works whether you are in the course, another course, or doing something completely unrelated to the course. It is much easier to stay in contact with individuals with Yahoo rather than using a built-in tool such as this that has only one specific use.


Student B:

  • There are only so many things that can be done with a text based conferencing format.

  • I do not like to use an additional email . . . just adds to my daily chores.

  • I don’t personally like the “I’m online” feature . . . we can just be so over-solicited and sometimes it’s good to just go about one’s business in peace.

  • I love variety, but I want simple, consistent, predictable access to my studies. Getting to them, I already have enough obstacles.


Student C:

  • Too many tools makes for a cluttered toolbox with more time spent sorting than working.

  • With so many tools and so many changes, one (cannot) get comfortable with an application before having to move on to the next version. This is an example of productivity paradox.

  • My initial impression of ATutor was that it was busy. Icons are nice, but they take up screen real estate.

  • I’ve never been a big fan of help files. I prefer to jump in and get my hands dirty. A truly intuitive application makes limited use of help files.



Student D:

  • ATutor interface is extremely cluttered, although I like the icon/ text option - meets accessibility needs. At least as a user you can use preferences – maybe too much opportunity to change.

  • (The many features are) like the result of a progressive build – someone wants something added, so it’s added.

  • I find the navigation is not intuitive, and the help files too lengthy and require scrolling – no search feature. I think screen clutter impacts upon my comfort with the navigation.

  • I like the embedded links to our MDE course, but they do clutter the screen.



Student E:

  • I have been reflecting on how my rural, barely computer literate students will react to this software. I believe I would need to send detailed written instructions on how to logon and navigate. Online help is distracting and hard to use if you are uncomfortable in this environment.



Student F:

  • In terms of interaction usability, at first I didn’t find it that intuitive . . . wasn’t sure what some of the circle icon meant. However, the user interface was for the most part fairly predictable.

  • Not impressed with some of the “bells and whistles” like the internal email . . . just one more thing to worry about.

  • I do like some of the site tools. Changing the preferences made the overall appearance more palatable for me. Also, the search and site-map features made navigating more purposeful.

 

Despite ATutor’s clear technical superiority in comparison with the existing HTML-based course site, the majority of students voted to retain the existing frames-based site shown in Figure 1. None voted to move to the ATutor platform. This result was disappointing to the course instructor, who had hoped for student feedback justifying a move to the superior support that would be available for ATutor in the University. A common theme in the students’ responses is their reaction to the ATutor display as too “cluttered/ busy” with too many “obstacles/ bells and whistles.” These responses are essentially the same as those reported in the similar study reported earlier in the series (Report #8). The clearest explanation for this preference for a simple course delivery format was provided in the present study by a student who stated: “I see advantages to both, but nothing compelling enough to make a change.”

Conclusions

The recurring finding of these studies brings into question the wisdom of providing DE students with ever more complex online learning systems; and it suggests the need to balance comprehensiveness of product features against clarity of usage. In the absence of unequivocal justification for moving to a new learning platform, both teachers and students are apt to select “the devil they know”. If new systems have definite advantages, these should be carefully pointed out to them. It is inevitable that the developers of online software (OSS as well as proprietary) will continually offer “bigger and better mousetraps” on the assumption that the more features their products contain, the more competitive they will be. OSS as well as proprietary software developers are moving in this direction; and the increasing comprehensiveness of LMS software such as ILIAS and ATutor is evident from the current product reviews and those provided by Farrell (2003) and edutools.info (2003). Reviews based solely on the volume of a product’s features, however, do not necessarily predict users’ reactions to the product or its educational effectiveness; and it is clear that simple preferences for proprietary or OSS approaches do not predict the educational merits of a product either. DE students’ reactions to the highly rated ILIAS and ATutor products, as expressed in the present report, suggest that too much product integration can be counter-productive in the online learning process, and that teachers and students may both benefit from a simple, modular-based approach that allows them to expand their online tools and skills as and when they need to do so.

Acknowledgement

The authors thank the members of Athabasca University’s MDDE662 course (Fall 2004), for their comments quoted in this study.

The next report in the series will continue the current discussion of attitudes to LMS methods.

N.B. Owing to the speed with which Web addresses are changed, the online references cited in this report may be outdated. They are available, together with updates to the current report, at the Athabasca University software evaluation site: http://cde.athabascau.ca/softeval/. Italicized product names in this report can be assumed to be registered industrial or trademarks.

 JPB. Series Editor, Technical Evaluation Reports

References

Bailetti, T. (2004). Improving Open Source Learning Content Management Systems. Article found at Eduforge.org website. Retrieved November 2, 2004 from: http://eduforge.org/wiki/wiki/eduforge/wiki?pagename=Improving%20Open%20Source

Edutools.info (2003). ILIAS Review. Retrieved November 2, 2004 from: http://www.edutools.info/course/productinfo/detail.jsp?id=203

Farrell, G. (2003). COL LMS Open Source. Paper found at the Commonwealth of Learning website. Retrieved November 2, 2004 from: http://www.col.org/Consultancies/03LMSOpenSource.pdf

Leslie, S. (2004). Open Source Course Management Systems. EdTechPost Blog. Retrieved November 2, 2004 from: http://www.edtechpost.ca/pmwiki/pmwiki.php/EdTechPost/Open SourceCourse
ManagementSystems

University of Cologne (2004). Didactical concept of ILIAS. Retrieved November 2, 2004 from: http://www.homer.ilias.uni-koeln.de/iliasdoc/doc/html/6_1.html






ISSN: 1492-3831