Group assessment:
alternative forms to evaluate student skills

Luís Manuel Borges Gouveia
University Fernando Pessoa Journal, May 1998

ABSTRACT

This paper presents a proposal for an environment designed to assess students knowledge in an Information Systems class. The environment was based on the university local LAN infrastructure and in the fact that almost all the students have their own laptop computer

In order to innovate the conventional exam, it was implemented a challenge to groups of three students. In each group, one member must be in the classroom, where normal examination took place, another at a social location where he can interact with others in a free form and the last one, at the library, where he can use local resources to answer the questions.

The three students can interact between them with their notebook computers and different services provided by their ability to use the potential of a Ethernet LAN, the services provided by Windows95 operating system and local university servers (WWW, email, news, IRC, ftp, Unix).

The paper describes the experience, their results and the way students lived them.

Keywords

computer supported cooperative work, collective problem solving in workplaces, educacional applications, using the WWW, group assessment.

1. The local environment

In 1995 University Fernando Pessoa set, as a requirement for admission, that every first year student should have a laptop computer. The University offers courses in areas such as social sciences, psychology, management, literature, advertising, and engineering, and wanted its students to acquire the basic skills to use, interact and survive in a information sea similar to the one that they will encounter in a information society environment.

Fall 1997 figures of the project show that about 2100 students and 25% of the staff have their own notebook computers, and that there are 400 network entry points to support them. The total population of the University is 4300 students and this is now a three years project that covers more then 50% of the total university students (from third, second and first years of all the courses).

Off-campus access to the University network services is also possible, but just via an Internet Service Provider. This "laptop for all" project has as a major objective to bring students to University with their computers and promote their use locally.

All the students can access the network via Web services at the university from Classrooms, Labs, and Library by using their Ethernet 10 Mbps LAN card and one of the existent network entry points. The environment created by intensive LAN use lead the way to different forms of interaction mostly between students that include hard disk share, email, IRC, chat and Web pages from personal to interest groups ones.

University Intranet services include WWW, ftp, news and Internet access for them and as a complement to the content created by professors, departments and development units. The main goal is to move from the traditional change in paper (and electronic) work presentations from students, to increase the way students from different courses interact with each other and "trade" information among them.

2. The experiment: setup and associated rules

The student’s evaluation must be a process accepted by the students and provide a practical situation, similar to the ones they will encounter in their professional lives. The assessment itself can be seen as a powerful learning environment where student’s skills, communication and tecnhology use, are tested. As future professionals they will have to meet some requirements like the ability to work together and contribute to a common goal (in this case represented by exam completation). Conventional evaluation systems tend to fail test these types of skills in students because they are mostly individual ones.

The experiment conducted tries to introduce a "real world" situation where several professionals must gather information in co-operation trying to reach common goals. The group assessment was designed based on assumption that the evaluation can act as a strong group motivation. In this group assessment the marks obtained are equal for the entire group, resulting in a clear statement that the group success is the individual success of each of the group members.

Each group has three students, gathered freely from the class. Each student has a different task and they must act as a whole to obtain the best results. One is at the classroom answering the paper examination, reporting the exams questions and receives the answers in his notebook. The second student is in a social location where he can interact with others. His objective is to try gather the maximum information possible to give to the rest of the group (this one has access to other expertise not in the group, based in his ability to have near him the right people, simulate the network knowledge that every professional must need to have in these days). The third element is in the library where he can only use the library resources to help the group.

All the three students can use their laptops and all the available software resources to communicate between them. The exam have six questions that are different for each group. These questions are part of a list made public a week in advance (the list has the 20 topics related with major discipline themes). In the beginning of the exam the theme for examination is distributed in random manner for each of the students in the classroom. After that, they fill in the exam paper the identification of the group elements and the working theme. The total time to answer the examination is two hours.

3. The action

The class selected for the experiment was Information Systems and was chosen in order to test the potential of the Intranet facilities to soften the theoretical and descriptive bias of this content subject and act as "a put in practice" of the discipline content. Also, the selected course belongs to non-technical related student type; the course is Economics and Finance. The students from this course have little knowledge in the use of networks or Internet/intranet services prior to these subjects (just one term to introduce their laptop, the computer concepts, the operating system, a word processor, a spreadsheet and a presentation software).

Information Systems is a one term class that introduces the system side of computer science, with a mixture of concepts, new technologies, introduction to data modelling and data flow diagrams and some case studies to integrate concepts. This is a general introduction class to computers and systems. The new technologies section gives the opportunity to introduce Internet and the Web in order to let students create their own homepages, learn how to use email, news and the web itself to search, share and publish information about specific subjects.

After the introduction of the new technology section, students have two and a half complete work months and the ability to answer a group assessment (the basic collaboration tecnhiques and some practice using the Internet to communicate and gather information). The course has two groups of 16 and 22 students each (the first group make the evaluation in May, 16 and the second in May, 19). With a week advance, the rules and complete description of the experiment was made public. This action prepares students guaranteeing that they understand this new form of evaluation, their criteria, the avaiable forms to gather information, the importance to communicate between them and the fight against time as a normal part of the evaluation. The themes list to work is also given at the same time and some discussion on where they can possibly gather information about, takes place (the list comprises 20 different theme proposals).

This experiment has several goals in mind: promote the use of the intranet services and Internet potential; training students to be more active and test their capacity to collect information; force students to manage their relations within a group and introduce group mediated technology as a workplace environment.

The experience set-up has been completed with the help of the librarian that managed to have the space needed by all the students at the library and controlled the time between student arrival and assessment completation. Also, in the social location some arrangements are made to assure that the necessary entry points to local LAN are available when needed. At last, a normal classroom with entry point was used.

Physically all the action is in the classroom where the group is identified and communications tested by each group leader. After a small period of time (15 minutes) the random choice of themes is made and the examination starts with each leader receiving a paper examination and an answer sheet to put the results of the group examination. They have two working hours where they test their abilities to interact between them, using computers as the communication channel.

4. The results

The experiment has been repeated in two different classes EF_A and EF_B; the first class with five groups (16 students, one missed the exam) and the second with eight groups (22 students, forced two of the groups to have just two elements – in fact they have recruit the third element from the other class).

The marks obtained by all the groups and their respective connect time is shown in figure 1. It is possible to see that the median results are inferior to the ones obtained in normal exams (58,31% against 72,32%) and even its dispersion is great in the group assessment (20,42% against 18,92%). The different between the best and the worst result is also significative (80-12=68).

Figure 1: marks and respective connection time for the group assessment

There is a relation between marks performance and time where the 12% marks represent a 184 minutes connection. Its possible to say that most of the groups drop out connection when there is no more time to be used. This connection time was controlled by librarian that enter the start time and the time students go away from reserved area in library; anyway, the complete time given to the exam does not exceed the given 120 minutes. So the extra time counted represent resilient communication between students after the exam delivery.

After public announce of the results from the group assessment, the students are asked to fill a query with eight questions. For the first seven the answer must be a number in a five point scale and for the last question, students are asked to give their opinion on how to improve this kind of assessment and what are the needs to support them.

When asked if they have understood clearly the rules for the group assessment, students respond yes (4.5 in 5 for yes). That result gives a good starting point for analysing the other results because students have understood the particularly characteristics of the assessment (even the marks are worst than in normal assessments).

When asked if they have learned new ways of using their notebooks computers, students respond yes (4.6 in 5 for yes). That result stated clearly some value for the group assessment experiment, at least as a novel living situation for students.

When asked if they use external expertise (the help of other people) in the assessment, students respond a "half-way" yes (3.8 in 5 for yes). Here we see a situation where seven students state no or little help from other people. An interesting situation is that all these students use just one type of service between the above mentioned.

When asked if the time for the group assessment is enough, students respond it was medium (3 in 5). This can be a strong indication that group assessment activities can be shorter than equivalent standard two hour exams. When make the same question for the exam, students say that the time is not enough (1.2 of 5, where 5 means enough time).

When asked if group assessment can train them to future professional activities, students respond yes (4.3 in 5). This means that they are open minded to innovative forms of evaluation their knowledge (even this means some loss in their marks) and that they are aware of the collaborative character of their future work environment.

The students have access to some Web pages specially organised for the Information System classes. When asked if they used them in the group assessment, students respond just sometimes (3.4 in 5). This means that the search for information and its replication was just part of the capabilities needed to make the group assessment (what is in fact one of the design goals for the experiment).

When asked for the services used, students report just five different services; they are WWW, email, irc, ftp, and chat. However students use these services in different percentages as seen in figure 2.

Figure 2: use of communication services in group assessment

We can regroup these services as search and gather information (www and ftp; note that students usage of these services is not as publishers, for the situation reported); asynchronous communication facilities (email) and synchronous communication facilities (irc and chat) as represented in figure 3.

Figure 3: services usage by groups of functionality

Based on the values presented it possible to say that students prefer synchronous relationship between them. Also, the information gathering activity requires a significative percentage of their time and effort and it must be classified as an individual activity. At last, it is not clear that synchronous communication facilities will suffice and that there use need not to be completed with asynchronous communication facilities.

The open question to students in how to improve group assessment gives some answer that can open some future work in these types of experiments. First, some suggested that the third element can be eliminated (it seems that students are more confortable just interacting in pairs). Second, some appointed more time to do it as the big issue to improve the group assessment (most of students report that they need a great amount of the available time to "get used" to collaborate). Third, the most mentioned improvement was practice and more frequent group assessment situations, in the words of one of the students "…to increase team work."

When asked about the needs to support group assessment, students point out more integration between the available tools, their use in normal classes and labs and a digital support library where they can found reference material for their answers. It seems they can set the basic requirements for CSCW applications as fresh starters of computer mediated collaboration situations.

5. Future work

The group assessment experience was the result of the availability of an environment where all students have their own notebook computers and the LAN infrastructure is everywhere in the campus, waiting for each one to plug in and get connected to university services or to someone’s computer. The question is information and communication but it can be too collaboration.

So, this is just an empirical first experience where a lot of work can be done to develop a support application for group assessment. It can also be a real lab to test some CSCW software already available like Habanero (http://www.ncsa.uiuc.edu/SDG/Software/Habanero/) and BSCW (http://bscw.gmd.de).

Even in the use of normal software there is some work to be done studying how people use the available time in activities and how they use the different available services. Also, how can be people prepared for working together and how to improve the rules for group assessment. This paper intends to be just an introduction report to future improvements on how to use collaboration facilities to assess student’s skills and evaluate them in a context more close to their future work environment.

6. References

Gouveia, F.R. and Gouveia, L.B. Active education: a pedagogical attitude. 2º Conferência sociedade de informação interactiva, reinventar a educação, Funchal, December, 1996.

Gouveia, L.B. NetLab, work at Fernando Pessoa. Away Day, CSEG Group, Lancaster University, UK, 14 November, 1997.

Gouveia, L.B. Use of notebook computers in higher education: perspectives. Simpósio de Investigação e Desenvolvimento de Software Educativo, Universidade Nova de Lisboa, Lisbon, October, 1996.

Hughes, J., King, V., Rodden, T., Andersen, H. (1994) Moving out from the control room: ethnography in system design. Lancaster University, Centre for Research in CSCW. Research Report CSCW/9/1994, Lancaster, 1994.

Oravec, J.A. (1996) Virtual individuals, virtual groups: human dimensions of groupware and computer networking. Cambridge Series on HCI: Cambridge University Press.

Plowman, L., Rogers, Y., Ramage, M. (1995) What are workplace studies for? Lancaster University, Cooperative Systems Engineering Group. Technical Report CSCW/12/1995, Lancaster, 1995.

Página criada em 26/5/98 por Luís Gouveia
Última alteração: 26/5/98