In doctoral programmes that don't use a viva or oral exam, a PhD thesis is assessed by one or more external examiners, who write a report recommending either a pass, amendments, or failure. These reports are usually confidential, so I was fascinated to come across a 1997 paper by the late Sue Johnston of the University of Canberra that analysed the common themes and concerns of 51 of them. Some highlights:

• Examiners put a lot of work into their reports: they were up to 16 pages long. Generally they were positive and supportive attempts to help the student see what changes could be made to enable a pass—far from the hatchet jobs that candidates fear.
• There was often significant variation between examiners of the same thesis: in one case, two recommended a pass as is, but the third failed it. Differences of opinion seems mostly to do with the expertise of the examiners in the technical details of the research, as well as some conceptual or methodological bias.
• Examiners were unclear in practice about the difference between a pass-with-corrections and a revise-and-resubmit. Very different standards were applied. In some cases, theses were passed [as is], accompanied by a list of recommended substantive changes!
• Writing quality matters. Writing was the number one concern: sentence structure, proof reading, referencing errors, clear arguments, even clear presentation. As an examiner put it, “One has to keep in mind that there is often a relationship between the quality of presentation and quality of scientific results.”
• An important criterion for a pass was that the research was “publishable”, but examiners interpreted this widely, some thinking this meant the thesis could be published as is, others that the results could be worked up and submitted.
• Attitudes towards supervisors ranged from blame for not picking up on issues of argument or presentation, to praise for collaborating on the thesis with the candidate.
• And the worst thing? Something I wouldn’t wish on any candidate: to have their thesis described, in the words of one examiner, as “benumbingly boring”.

Johnston, S. (1997). Examining the examiners: An analysis of examiners’ reports on doctoral theses. Studies In Higher Education, 22(3), 333–347.

Permalink

One interesting feature that emerged from the recent Wikipedia in Higher Education summit was the time required to teach students how to edit. This isn't surprising: like setting up a blog, or learning the etiquette of tweeting, Wikipedia’s code is a novelty to students. Professors, however, were unhappy that they needed to devote one or two class periods to the mechanics of editing, rather than to actual course content. This is where IT departments, university libraries, or learning support centres need to step up. We can’t expect academics to teach these basic digital literacy skills, and as Wikipedia becomes more widely used in the classroom we run the risk of duplicated effort, where students are hearing again and again how to log in, how to create a link, how to stop your article being deleted… A central training resource (and, indeed, mandatory Wikipedia accounts for every student) that all students can access would be the solution: a mixture of links, reading material, workshops, and hand holding.

The Inside Higher Ed article also noted that Wikipedia editing demands a certain amount of personal resilience and negotiation. These are skills not common in university courses; I regularly meet students whose group work falls apart because they don't have the maturity and social skills to deal with freeloaders. It would be fascinating to see how online collaboration that involves editors outside the academy would develop student confidence and ability to justify their research.

Permalink

What does digital literacy even mean? I seem to be at odds with most people using the word, who are keen to push Microsoft Word training at schoolkids and the unemployed. This hearkens back to an older era of schooling, when the classroom was a space for preparing good employees who could write neatly and keep accounts. But we all agree the purpose of education has moved a bit beyond that.

I deal with plenty of 18-year-olds at a university, the cream of the New Zealand education system, potential voters and workers. They’ve never known a world without the Web. What has over a decade’s education in that environment done for them? Well, I find they cannot reliably distinguish between:

• A PDF of a journal article linked from a webpage
• The text of the article on the page
• A government or university press release about the findings
• An opinionated blog posting summarising the article

They think these are all the same thing. All sources are cited with equal authority, research is primarily through Google searches (rather than Google Scholar or specialised databases), and students are too naive to distinguish between content on .edu, .org, and .com domains. Most don’t realise that Wikipedia is editable by anyone, let alone that they should correct mistakes where they see them: it’s gospel. I’ve yet to meet a student that displays any skepticism about the possible agenda of a website author: if it’s on the web, it’s true.

In one recent case, a pithy quote in an essay was attributed to “essays4u.com” (not its real name), a commercial US term-paper mill for cheaters. The student had obviously found one of the site’s sample essay via Google, liked the ideas expressed there, and carefully cited the source as instructed by their lecturer. All in good faith. That takes a pretty breathtaking lack of understanding of the internet.

Parents and some teachers grossly overestimate the computer competencies of teenagers. So, indeed, do those teenagers. But I’ve met very few “digital natives”: most have very restricted areas of expertise, do not grasp the underpinnings of the technology they use, and flounder when required to learn something new. These are not the “life-long learners” of the NZ curriculum—their computer education has been through workshops and formal classes (just what most digital literacy fans advocate), so they passively expect to be taught the skills they need. It’s a rude shock when I expain they now have to teach themselves new software packages, every year for the rest of their lives. 

The digital illiteracy I see is not just a problem for university students. We want those 18-year-old school leavers to be able after a few years to set up a business, research a medical condition, make purchasing decisions in an office, or deal with EQC and CERA so they’re not left destitute after an earthquake. If they don’t learn these skills during their time in the education system, they’re going to find it much harder to teach themselves other skills later. Advocates of digital literacy sometimes forget that, like all education, it's learning how to learn.

So I don’t think digital literacy means “computer skills” classes—certainly not programming skills as some die-hard geeks seem to think (you don’t need to know how to build a car to drive one, or to write a novel before you can read one). 

I don’t think digital literacy is about learning this commercial software package rather than that one: they'll probably both be obsolete in a few years. Cheap or free software’s obviously better because everyone can own it; beyond that, the argument’s a waste of time. General principles are the goal, so making students switch to a new browser or word processor every week teaches flexibility.

I don’t think digital literacy is a self-contained subject, either, tacked onto the curriculum—it should be a seamless underpinning of the way we teach, because schools have enough to do already. 

I don’t think it’s about “employment skills”, because we live our lives on computers, not just our jobs. It’s not touch-typing and pivot tables.

Digital literacy is about being an informed, sceptical citizen that can find and filter information. 

Permalink

A study in British Columbia found that students took writing assignments more seriously when they were appearing in the public space of Wikipedia. No surprise there. But the students also saw wiki editing as a transferable skill (which is true: we'll all be editing wikis and collaborating with distant authors soon). Consequently, they were keener to learn Wikipedia’s conventions and code than they were to learn the course management system Blackboard and other specialised tools which universities want students to use, but which aren't found in the real world.

Permalink

How to get lecturers to stop whining about Wikipedia and start helping improve it? The Association for Psychological Science (the main professional body for academic psych departments) has teamed up with the Wikimedia Foundation, to form the APS Wikipedia Initiative, aimed at encouraging psychological researchers to correct inaccurate articles. All very worthy, but a bit tedious for academics, like getting them to eat their vegetables, as the Chronicle of Higher Education reporter puts it. The secret seems to be to work with postgrads and advanced undergrads and use article-editing as a writing assignment—just what Wikipedia has also been advocating. This really does seem to be the start of a change in attitudes towards Wikipedia in the universities; academics are realising that, inaccurate or not, its articles are the public face of their discipline, so it’s in their best interest to make sure they're fixed.

Permalink

One of the advantages of having students writing for Wikipedia is that it’s actually read by a lot of people; unlike student essays or reports, which have a readership of two. Jon Beasley-Murray, whose class project on Latin American magic realism I've mentioned elsewhere in this blog, checked out the page view stats for the article on Vargas Llosa his students rewrote and brought up to feature-article status. He found their work was being read by 500 people a day, every day. But the day Vargas Llosa was award a Nobel, 116,700 people read the page. That article's page views are up a million a year now, which is more readers than a New Zealand journalist is going to get for a feature article in the biggest-selling paper, or an academic for almost anything they write in their life.

Permalink

The Wikimedia Foundation have started working with US professors to enourage their students to create content for Wikipedia, as this Inside HigherEd article outlines. The way they're organising it is interesting. Wikimedia are recruiting student “ambassadors” (or evangelists, I suppose) to act as troubleshooters; one of the problems of teaching with Wikipedia is the editing conventions take a while to learn, and professors are not necessarily experienced enough at Wikipedia editing to handle disputes or disasters in their students’ work. It reads like Wikimedia are training them up from scratch in three-day workshops, but I’m hoping the ambassadors will be largely drawn from the ranks of the active Wikipedia editors already at the school. There’s already a Wikipedia culture of experienced editors volunteering to help newcomers learn the ropes, and this will just be formalising it by shifting it offline. I think the institution should at least provide free pizza for all these volunteer teachers. Also good to see that Wikipedians from around the world will also act as mentors and emergency helpers, to handle deadline crunches when the lecturers aren’t available.

The article gives examples of projects, and it’s clear some of the teachers quoted (and commenting) aren’t very familiar with Wikipedia—they talk about getting to students to write entries offline and submit them, which is not a bad way to get the entire article deleted; far better to get the students involved with editing and gradual collaboration on Wikipedia topics, as they learn the ropes and the writing conventions expected.

Still, this is great news, and even better to hear that Wikimedia are trying to develop a resource kit of guidelines and tools to help teachers use Wikipedia as a teaching tool.

Permalink

When you're learning to use the Net better, it can be helpful to know where you stand in relation to everyone else. There have been numerous attempts to divide internet users into marketing group or tribes: one recent attempt has been the Digital Anthropology Report, which ranks people from Digital Extroverts to Timid Technophobes. The digital divide is not a crude division between the tech-savvy and the luddites: one noticable trend is the increasing number of people who use the Net all the time through mobile devices, but don't, for example, blog or even send much email. You can take the quiz to see where you fit.

The Pew Research Center's internet project has another more nuanced online survey, with categories like Digital Collaborator, Roving Node, and Technology Indifferent. Pew are interested in whether people are creating or just passing on information, and whether they use their phone for voice only or as a window onto the Net. I'd be interested to see whether the availability of broadband, and the cost of cellphone web surfing, affect the usage patterns of New Zealanders compared with the results of these UK and US-based surveys.

If you feel like taking both quizzes, do let me know whether the results agree with one another, and if you think they reflect how you use the internet.

Permalink

For some time, I've been worried about who in academia teaches students how to use information technology. I gave a presentation on the subject at the Association of Tertiary Learning Advisors of Aotearoa/New Zealand (ATLAANZ), on Massey's Albany campus, 20 Nov 2009. This is an outline of what I said; you can download a PDF of my slides as well and follow along.

Students enrolling now will, as soon as they graduate, be using indispensible tools that haven't been invented yet. Facebook was invented in 2004, and opened to the general public in 2006; Twitter came out in 2006 too. These two tools are hot stuff at the moment, but it's likely that last year something was invented that will be just as big in 2011. And next year they'll invent the hot website of 2013. How do we prepare students for this?

Universities almost all mention information technology skills somewhere in their list of graduate attributes. The usual phrasing is "information literacy" or "appropriate use of technology". What does this mean in practice?

I suggest that "appropriate" IT skills, for both undergraduates and postgraduates,

• start from a high-school baseline of Microsoft Word, saving, printing, file management, web use, and email
• are ones that will crop up both during study and immediately on employment
• exclude mandatory (and non-transferable) use of educational software like Moodle or Blackboard
• are the skills graduates might need in 2015 (that is, the freshmen of 2010 + 3 years on + the 2 years it takes a university to organise anything.)

Here is a suggested list for the year 2015, in order from most essential to least, with which the audience concurred.

• Manage passwords
• Assess the reliability of a web source
• Manage multiple email addresses including forwarding
• Manage dozens of emails a day
• Crop and change the resolution of a photo
• Format a long document
• Use presentation software professionally
• Create better-than-default charts from a spreadsheet
• Edit a collaborative online document
• Use [Twitter/whatever] to maintain a social network
• Create and edit a wiki
• Create a blog
• Register a domain name
• Create a simple website
• Use bibliographic management software
• Record and edit sound
• Shoot and edit video

IT topics can be sorted along two axes. Some are tools, for accomplishing any number of things, like Wordpress. Others are specific tasks that need to be completed, like creating a class web page. Another way of sorting is by contrasting generic topics that useful for multiple different contexts with course-specific specialised (or embedded) content. Here's an example.

We can then arrange all IT topics in a space, from general tools to course-specific tasks; I've put some examples below.

OK. So how is this stuff actually taught in universities and polytechnics? There are four key players: the library, the IT department, individual lecturers, and the Learning Skills/Academic Support unit.

When I surveyed individual institutions and the audience, I found a wide variety of strategies for covering the bases. No institution taught all, or even most, of the list of IT topics we all agreed were important. The best case was where both the library and Learning Skills collaborated in covering main topics. IT gave generic workshops, and lecturers or teaching assistants taught whatever was necessary for students to pass their course.

In some institutions, though, the IT department had absolved itself from training students, and concentrated on staff. In others, Learning Skills did not touch on IT at all, and the library had to pick up the slack in both general and specific topics. Which of these matches your own institution best?

Three questions arise from this exercise.

1. How much do we assume about the abilities of "Digital Natives"? (see Prensky’s papers and book on this.) My concern is that lecturers will assume these students arte generally tech savvy and can pick things up on their own, but the evidence suggests their abilities are very narrow and practical—Facebook and txting, essentially.
2. Most topics on the essential-skills list are not being taught. Who is able to teach all this? It's essential to have IT staff on board, but they aren't usually trained teachers, and may have little experience with the curriculum and current student needs, so we can't just pass the buck.
3. The conversation on IT training is centred around workshops, but the workshop model might be the wrong one. Given we all have to learn new IT skills every year for the rest of our lives, how do help students move from being passive learners, waiting for the institution to supply training, to active learners that teach themselves new technology as they need it?

Permalink

Anne Cunningham, lecturer at Cardiff University, tells how a student subverted a classroom demonstration:

The session aimed to show the inaccuracies of Wikipedia and how it could not be trusted. But this student sabotaged the exercise. He demonstrated that the essential quality of Wikipedia is that it can be edited. Before most students had got round to the piece of work, he went into the Wikipedia article and improved its quality by updating the content and referencing the article!

It would be nice if half as much effort was spent improving Wikipedia as vilifying it.

Permalink

UC Geography lecturer Deirdre Hart has been getting her students to create of Wikipedia pages with some success, an about-turn after her initial ban on the use of the site. There’s a nice write-up in the June 4th UC Chronicle (PDF); Deirdre points out the peer-review aspect of Wikipedia is very useful (and it's rather like the way academic research works), and that the students took it quite seriously:

They actually saw it as a more real-world assignment than most of the ones they’d done at University, and told me so. They loved that their work would not just be gathering dust on my shelf, that it would be out there in the real world being read and reviewed by people.

The NZ Herald reported on Deirdre’s experiment, and got some good comments from other academics. Kerry Shephard at Otago pointed out that Wikipedia might be unreliable, but likewise “the teachers might be wrong, the textbooks might be wrong.” John Roder at Auckland noted that students should be thinking critically about the provenance of their information, and Wikipedia at least possesses the virtue of transparency.

Permalink

Nigel Robertson, eLearning Designer at Waikato, kindly shared with me the results of an anonymous survey of staff and students on, among other things, Wikipedia use. Tthey ranked their usage on a 5-point scale, but, as Nigel points out, “I think the way we asked the question [was] flawed. Our 5 point scale of use included ‘to learn’ (or ‘to teach’) to indicate the highest level of use; however it is quite possible to use something for learning but not to use it regularly. We’ll be looking to tweak that question format in the future.” 

Permalink

Here's a very typical extract from the essay-writing guidelines of a department at UC:

Sources such as Wikipedia are not subject to rigorous checking, and can be added to by anyone, whether qualified in the subject or not. Wikipedia should NOT be used in the preparation of an academic essay.

Note the ban is not on citing Wikipedia, but on using it as part of the research process. Most students nevertheless use Wikipedia in their preliminary research, so one assumes they lie when lecturers ask them how they wrote their essay. Since quite a few professors use Wikipedia too (how else do you write all those lectures?), students must assume their lecturers are hypocrites. This is not healthy. Perhaps we need to elevate the discourse beyond these sort of blanket bans.

Permalink

The idea of Wikipedia article creation as a teaching tool is becoming more popular; for example, getting a class to create 120 articles on Roman civilisation. There are short proposals here and here, a link to an interesting email discussion which mentions student resistance to becoming an author, and a longer introduction to the process by Piotr Konieczny.

But actual reports from the front lines are always the most valuable. Jon Beasley-Murray, a fellow Duke alum, describes having his SPAN312 class write articles on Latin American literature. A lovely walk-through of the tribulations and successes, and the unexpected help of the Feature Article team. He compares it to “the usual essays and exams that we assign our students, [which] really are rather pointless busywork.” Jon also has a some helpful tips for lecturers contemplating trying this themselves—recommended reading.

Permalink

Many academics aren’t happy with Wikipedia. They have two basic beefs:

1. Some if it is wrong. (When an academic tells me they found a mistake on a Wikipedia page, I ask them “so, did you fix it?”)

Wikipedia gets more accurate the more people use it, it’s more accurate than the Encyclopedia Britannica, more up to date than the students’ textbooks, and any biases or discussion are openly documented on the Talk pages. Yes, students have trouble telling good from bad, in all things, not just Wikipedia. It’s our job to help develop their information literacy.

Nigel Robertson at Waikato has a good response to the inaccuracy claims, and suggests some student assessments that use Wikipedia as a starting point. Robert Cummings, author of Lazy Virtues, does the same, and defends its value in teaching writing.

2. Students use it as their primary research tool.

In fact, students don’t. Actual research at the University of Washington suggests students use Wikipedia as a starting point, and then move on to peer-reviewed published sources, via the library or database searches (as do most academics, by the way). Wikipedia is a fabulous beginning, not a primary source—but then neither are encyclopedias or dictionaries, both also occasionally cited in student essays. Perhaps we should concentrate on teaching students what qualifies as a good reference in academic writing.

I find that many objections to Wikipedia are from academics who have not really explored it, or whose opinion was formed some time ago. For example, this objection dates back to 2005, when Wikipedia was only four years old, with ‘only’ 860,000 articles (some interesting comments at 7 and 12, by the way). A lot has changed in the last five years—there are now 3,200,000-odd articles, for one—and a lot more will change in another five.

Permalink

The Teaching with Wikipedia project is interested in how Wikipedia is being used in the University classroom, by both teachers and students. We are seeking to raise the level of discourse about students using Wikipedia for research, encouraging faculty to move beyond banning it outright. We also want to promote assigning students Wikipedia articles to create or edit, as an assessment tool alongside traditional essays or reports.

If you have an opinion on or interest in Wikipedia in the classroom, come along to an informal discussion and showcase of successful projects.

Friday 23 April, 10–11am, Learning Skills Centre, South Bank
Contact: Mike Dickison, LSC, mike.dickison@canterbury.ac.nz
Further information and notes from the meeting will be posted to this site

Permalink

Robin Heyden mounts a vigorous defense of students using Wikipedia for research. Don’t blame Wikipedia for students not consulting enough sources—that’s a teaching problem.

 She points out that “these history and discussion pages could be a good classroom tool. Isn’t this precisely what we’re trying to get our students to do? To think critically about information, to question, to dig deep?”

Permalink

Permalink

Wikipedia maintains a list of educational institutions that use Wikipedia as a teaching tool, and tips on how to run a good wiki-making assignment, at the School and Universities Projects page; the shortcut is WP:SUP. Lots of good advice here: try editing articles yourself for a while before you make students do it, so you can see what sort of negative edits they might encounter; don’t use vandalism to make a teachable moment, as you could end up getting your school blocked; and double-check the copyright status of all the images that students might upload.

Permalink