EdTechResearcher

The Wiki Quality Instrument

• Section 1: Summary Questions
• Section 2: Definitions
• Section 3: Decision Rules

Section 1: Summary Questions

We present these summary questions to introduce new educators and researchers to the WQI. These summary questions were not used by our trained researchers. In piloting the WQI, we found that we achieved the highest interrater agreement among trained coders when we used short, declarative sentences as decision rules. However, in presenting our findings at conferences, lab meetings, and in print, we have found that a list of summary questions is an effective way to communicate the items of the Wiki Quality Instruments to new audiences.
 
Summary questions of coding categories used in the Wiki Quality Instrument

Section 2: Definitions

The following key terms are referenced in the WQI decision rules:
Educator: A person acting in the capacity as a professional educator: including teachers, librarians, school administrators, IT staff, homeschool teachers, coaches, etc.
Student: Students are young people enrolled in a learning experience: classroom students, homeschool students, athletes on a team, members of a club, etc.
Others: Persons who do not fit in one of the above two categories including parents, families, and community members not employed by some kind of educational institution.
Creator: The primary content-creator or facilitator of the wiki environment. Usually the person with the most edits and the earliest edits. The person who “owns” and exerts the primarily editorial control over the content, structure, and presentation of the wiki. Each wiki should be coded as either “Student”, “Educator”, or “Other”, but not in multiple categories.
Participants: A person or persons who are not the creator who contribute to the wiki. These are people who made some kind of direct change to the wiki (page edit, comment post, document upload, etc.), but are not the creator.  Each wiki can be coded as Student, Educator, or Other, and can be coded in multiple categories.
Audience: A person or persons who are the intended viewers of a given wiki. Each wiki can be coded as Student, Educator, or Other, and can be coded in multiple categories.  By default, we assume that student-created wikis have an educator audience unless we find evidence to the contrary (such as a student who creates a wiki with study guides for all of her classes intended to be used just by other students and not evaluated by an educator). We also assume that educator-created wikis with “student-facing” materials—such as naming the wiki for a course or class, posting syllabi, instructions, assignments, etc.—have a student audience. For all other audience designations, we look for specific in text or contextual references to a specific audience. For instance, to code a student-created wiki as having a student-audience, we would need evidence that the student creator intends for other students to view the wiki (“Hey guys, I hope this study guide for chapter 4 is helpful; let me know if you have suggestions!). To code an educator-created wiki as having an educator audience, we look for similar kinds of evidence (such as a page of links to lesson plans for teaching about the area of a circle).

Section 3: Decision Rules

Note: I’m having some trouble rendering the table correctly below. In the meanwhile, here it is as a word document.
WQI For Web
 

Demographic Questions

Category	Indicator	Score as “0” if there is	Score as “1” if there is
Meets Criteria
	Viewable	When the URL is entered, it returns an error because the wiki is private, deleted, or unchanged	The wiki can be viewed in a browser.
	US Based?	No evidence that wiki supports learning in U.S. based schools	Evidence that the wiki supports learning in U.S. based schools, including Department of Defense schools overseas
	K-12	No evidence that the wiki supports teaching and learning in the K-12 grades.	Evidence that the wiki supports teaching and learning in the K-12 grades.
Grade Level
	K-5	No evidence that the wiki supports teaching and learning in the K-5 grades.	Evidence that the wiki supports teaching and learning in the K-5 grades.
	6-8	No evidence that the wiki supports teaching and learning in the 6-8 grades.	Evidence that the wiki supports teaching and learning in the 6-8 grades.
	9-12	No evidence that the wiki supports teaching and learning in the 9-12 grades.	Evidence that the wiki supports teaching and learning in the 9-12 grades.
	University	No evidence that the wiki supports teaching and learning in a post-secondary setting.	Evidence that the wiki supports teaching and learning in a post-secondary setting.

 

Category		Indicator	Code as “0” if…	If not “0”, code as text with the following information
Narrative	Narrative		The wiki has only been barely started with no information about the users or planned activities. (Note: sometimes even a URL is enough to begin to recognize probable users and activities)	Write a few phrases describing the usage and purpose of the wiki
Create Date	Create Date		Never. If a wiki is viewable, the create date is viewable.	Enter the date of the first version of the Front Page, which is automatically generated by PBworks.
Host	School Name		Cannot identify the hosting school or no school is involved	Write the name of the primary hosting school
	District Name		Cannot identify the hosting district or no district is involved	Write the name of the primary hosting district
	Site Name		Cannot identify another hosting institution (supra-district organization, public library, homeschool organization, etc.) or no other institution is involved	Write the name of the primary hosting institution
	School (2) Name		Cannot identify the secondary hosting school or no secondary school is involved	Write the name of the secondary hosting school
	District (2) Name		Cannot identify the secondary hosting district or no secondary district is involved	Write the name of the secondary hosting district
	Site (2) Name		Cannot identify another secondary hosting institution or no other secondary institution is involved	Write the name of the secondary hosting institution
Category	Indicator		Score as “0” if there is	Score as “1” if there is
Subject Area
	Contained Elementary		No evidence that the wiki supports teaching and learning in a contained elementary classroom. Contained elementary refers to wikis used in elementary school classrooms where a single teacher teaches all subjects to a classroom.	Evidence that the wiki supports teaching and learning in a contained elementary classroom. Contained elementary refers to wikis used in elementary school classrooms where a single teacher teaches all subjects to a classroom. In contained elementary wikis, it’s likely that you will give a 1 for “Contained Elementary” as well as a 1 in several subject areas.
	English/ Language Arts		No evidence that the wiki supports teaching and learning in English or Language Arts subjects. ELA classes are devoted to the study of English, reading, writing, literature, poetry, prose, textual analysis of drama, spelling, and so forth.	Evidence that the wiki supports teaching and learning in English or Language Arts subjects. ELA classes are devoted to the study of English, reading, writing, literature, poetry, prose, textual analysis of drama, spelling, and so forth.
	Math		No evidence that the wiki supports teaching and learning in Mathematics.	Evidence that the wiki supports teaching and learning in Mathematics.
	Science and Engineering		No evidence that the wiki supports teaching and learning in the natural and physical sciences- biology, chemistry, physics, geology, etc—or engineering courses.	Evidence that the wiki supports teaching and learning in the natural and physical sciences- biology, chemistry, physics, geology, etc—or engineering courses..
	English as a Second Language		No evidence that the wiki supports teaching and learning of English to non-native English speakers. (Sometimes also called English as a Foreign Language or English Language Learners).	Evidence that the wiki supports teaching and learning of English to non-native English speakers. (Sometimes also called English as a Foreign Language or English Language Learners).
	Social Studies		No evidence that the wiki supports teaching and learning in the Social Studies, including history, religion, psychology, economics, geography, and other social sciences.	Evidence that the wiki supports teaching and learning in the Social Studies, including history, religion, psychology, economics, geography, and other social sciences.
	Computer Science and Technology		No evidence that the wiki supports teaching and learning in Computer, Computer Science, Programming (Java, C++, etc), Technology Applications, or Typing Classes, or teaching and learning about educational Web/technology applications independent of an academic subject area.	Evidence that the wiki supports teaching and learning in Computer, Computer Science, Programming (Java, C++, etc), Technology Applications, or Typing Classes. Also includes wikis that support teaching and learning about educational Web/technology applications independent of an academic subject area.
	Modern Foreign Language		No evidence that the wiki supports teaching and learning in Modern Foreign Languages such as Spanish, French, German, Chinese, etc.	Evidence that the wiki supports teaching and learning in Modern Foreign Languages such as Spanish, French, German, Chinese, etc.
	Classic Languages		No evidence that the wiki supports teaching and learning in Classical Languages such as Latin and Greek	Evidence that the wiki supports teaching and learning in Classical Languages such as Latin and Greek
	Art		No evidence that the wiki supports teaching and learning in any form of visual, studio, or performing arts.	Evidence that the wiki supports teaching and learning in any form of visual, studio, or performing arts.
	Business		No evidence that the wiki supports teaching and learning in business classes.	Evidence that the wiki supports teaching and learning in business classes.
	Health/Physical Education		No evidence that the wiki supports teaching and learning in health classes, sexual education, physical education, or extra-curricular sports teams.	Evidence that the wiki supports teaching and learning in health classes, sexual education, physical education, or extra-curricular sports teams.
	Education		No evidence that the wiki supports teaching and learning in education classes for teachers or pre-service teachers.	Evidence that the wiki supports teaching and learning in education classes for teachers or pre-service teachers.
	Library		No evidence that the wiki supports a library site or teaching and learning in library or media studies courses.	Evidence that the wiki supports a library site or teaching and learning in library or media studies courses.
	Other		No evidence that the wiki supports teaching and learning in a subject area not defined above.	Evidence that the wiki supports teaching and learning in a subject area not defined above.
Creator
	Student		No evidence that a student is the primary content-creator or facilitator of the wiki	Evidence that a student is the primary content-creator of facilitator of the wiki. Common evidence in this category includes (but is not limited too) the creator identifying themselves as a student; academic work which is in response to a particular prompt; naming or identifying the wiki as in service of a student or a working group; grammar, spelling and syntax characteristic of young people.
	Educator		No evidence that an educator is the primary content-creator or facilitator of the wiki.	Evidence that an educator is the primary content-creator or facilitator of the wiki. Common evidence includes: self-identifying as a teacher, posting instructional materials, posting instructions for academic work, naming or identifying a wiki as in service of a course or class rather than for a student or working group.
	Other		No evidence that an educator is the primary content-creator or facilitator of the wiki.	Evidence that a non-student, non-educator is the primary content-creator or facilitator of the wiki. Rare.
Participant
	Student		No evidence that any student (not counting the creator) contributes to the wiki by contributing to a page edit, leaving a comment, or uploading a document.	Evidence that at least one student (not counting the creator) contributes to the wiki by contributing to a page edit, leaving a comment, or uploading a document.
	Educator		No evidence that any educator (not counting the creator) contributes to the wiki by contributing to a page edit, leaving a comment, or uploading a document.	Evidence that at least one educator (not counting the creator) contributes to the wiki by contributing to a page edit, leaving a comment, or uploading a document.
	Other		No evidence that any non-student, non-educator (not counting the creator) contributes to the wiki by contributing to a page edit, leaving a comment, or uploading a document.	Evidence that at least one non-student, non-educator (not counting the creator) contributes to the wiki by making a page edit, leaving a comment, or uploading a document.

 

Audience
	Student	No evidence that students are meant to be the audience of the wiki.	Evidence that students are meant to be the audience of the wiki. Most educator-created wikis have students as their audience. By default, if an educator is posting academic materials or instructions for academic work, we assume the audience of the wiki is students. If a student is the wiki creator, only code 1 here is there is evidence that the wiki is intended for viewing by other students. Evidence would include a specific reference to a student audience, such as a reference to how other students can use the presented material.
	Educator	No evidence that educators are meant to be the audience of the wiki.	Evidence that educators are meant to be the audience of the wiki. Unless there is clear evidence to the contrary, we assume that all sites created by students have educators as an audience. If an educator is creating a resource site for other educators, code this category as a 1.
	Other	No evidence that others are meant to be the audience of the wiki.	Evidence that others are meant to be the audience of the wiki. Only code 1 if there is explicit evidence that parents, experts or others are intended to be an audience for the site—such as an invitation to parents to view, messages from teachers and students to parents, or communication from the wiki creator and/or participants to others.

 

Quality Questions

 

Category	Indicator	Score as “0” if there is	Score as “1” if there is
Information Gateway
	Course Materials	No evidence that instructional or content materials are posted on the wiki.	Evidence that instructional or content materials are posted on the wiki by teachers for viewing by students or posted by students for viewing by other students (i.e. do not code 1 if a student posts instructional or content materials for teacher viewing, other viewing, or for the student’s own benefit)
	Information Gateway	No evidence that links to resources that are beyond the wiki subdomain are posted on the wiki by teachers for viewing by students or posted by students for viewing by other students.	Evidence that links are posted on the wiki to resources that are beyond the wiki subdomain by teachers for viewing by students or posted by students for viewing by other students (i.e. do not code 1 if a student posts links to external sites for teacher viewing, other viewing, or for the student’s own benefit)

 

 
 

Category	Indicator	Score as “0” if there is	Score as “1” if there is
Participation
	Contribution	No evidence that students contribute by making edits or changes (including but not limited to adding text, uploading files, etc.) to the wiki	Evidence that students contribute by making edits or changes (including but not limited to adding text, posting comments, uploading files, etc.) to the wiki
	Ownership	No evidence that a student (or students) owns the wiki. Wiki “owners” are the wiki’s creator, primary facilitator or primary content producer.	Evidence that a student (or students) owns the wiki. Wiki “owners” are the creator, primary facilitator or primary content producer for the wiki.
	Individual Pages	No evidence that at least one student owns an individual page. Wiki page “owners” are the creator, primary facilitator or content producer.	Evidence that at least one student owns an individual page. Wiki page “owners” are the creator, primary facilitator or content producer for the page.
	Shared Pages	No evidence that at least two students co-own a single shared page. Wiki page “owners” are the creator, primary facilitator or content producer for the page.	Evidence that at least two students co-own a single shared page. Wiki page “owners” are the creator, primary facilitator or content producer for the page.

 

Category		Indicator	Score as “0” if there is	Score as “1” if there is
Expert Thinking			EVIDENCE IN EXPERT THINKING CATEGORIES CAN COME FROM UPLOADED DOCUMENTS
	Content Required		No evidence that students complete tasks requiring academic content knowledge, or information and skills that are taught as part of the school curriculum.	Evidence that at least one student attempts to complete a task on the wiki requiring academic content knowledge, or information and skills that are taught as part of the school curriculum. Simple reading and writing do not count here, although specialized forms of reading (such as grammatical parsing) or writing (such as writing poetry or drama) do count.
	Knowledge organization		No evidence that students complete tasks requiring analyzing, organizing, or synthesizing information.	Evidence that at least one student attempts to complete tasks on the wiki requiring analyzing, organizing, or synthesizing information.
	Self-Reflection		No evidence that students use the wiki to reflect on their thinking process or work products.	Evidence that at least one student uses the wiki to reflect on his/her thinking process or work product. The reflective exercise must occur on the wiki, but the object of reflection does not have to be on the wiki.
	Crediting		No evidence that students credit the supports of their work products.	Evidence that at least one student attempts to credit the supports of his/her work product on the wiki.
	Teacher Feedback		No evidence that there is teacher feedback on the wiki to comment on the thinking process or work products of students.	Evidence that that there is teacher feedback on the wiki that comments on the thinking process or work products of students.

 

 

Category		Indicator	Score as “0” if there is	Score as “1” if there is
Complex Communication
	Commenting		No evidence that students comment upon the work of another wiki user (Student or non-student).	Evidence that at least one student comments upon the work of another wiki user (Student or non-student).
	Discussion		No evidence that students sustain a conversation of at least four turns with other users (student or non-student)	Evidence that at least one student engages in a discussion with at least four conversational turns with another user
	Concatenation		No evidence that more than one student contributes discrete text or content to a page.	Evidence that at least two students contribute discrete text or content to a page. Evidence of concatenation can come from bylines attributed to discrete text or from edit histories.
	Copyediting		No evidence that one student corrects the grammar, spelling, punctuation, or syntax of another user’s discretely owned text or content.	Evidence that at least one student corrects the grammar, spelling, punctuation, or syntax of another user’s discretely owned text or content. Evidence generally must come from edit histories with clear identities.
	Co-Construction		No evidence that a student substantively edits the work of another user working on the same page such that the text is no longer discretely owned.	At least one student substantively edits the work of another user working on the same page such that the text is no longer “discretely” owned. Evidence generally must come from edit histories with clear identities.
	Scheduling		No evidence that a student participates in a scheduling activity on the wiki, such as posting their name to a list of times or responsibilities.	Evidence that a student participates in a scheduling activity on the wiki, such as posting their name to a list of times or responsibilities.
	Student planning		No evidence that students discuss and develop strategies on the wiki for completing a work product (on or off wiki) with other students.	Evidence that at least one student discusses and develops strategies on the wiki for completing a work product (on or off wiki) with at least one other student.

 

Category		Indicator	Score as “0” if there is	Score as “1” if there is
New Media Literacy
	Formatting		No evidence that students use formatting such as colored text, boldface, italics, and so forth on a wiki page. (Formatting on attached documents does not count).	Evidence that at least one student uses formatting such as colored text, boldface, italics, on a wiki page. (Formatting on attached documents does not count).
	Links		No evidence that students add Web link to a wiki page.	Evidence that at least one student adds Web links to a wiki page. The links can be to other wiki pages, external Web sites, or to uploaded documents. (Links within attached documents do not count). For example: http://www.google.com.
	Hyperlinking		No evidence that students add links to non-URL text or images on a wiki page.	Evidence that at least one student adds a link to non-URL text or images on a wiki page.  For example: Google
	Images		No evidence that students embed images to a wiki page.	Evidence that at least one student embeds an image into a wiki page. You must be able to see the image on a page. Merely uploading a file should be coded under “Upload”. Simple links should be coded under “Links.”
	Multimedia		No evidence that students embed sounds, videos, or other multimedia files to a wiki page.	Evidence that at least one student embeds sounds, videos, or other multimedia files to a wiki page. (Multimedia files within attached documents do not count. The image or multimedia application must appear on the page as an embed frame. Merely uploading a file should be coded under “Upload”. Simple links should be coded under “Links.” ).
	Upload		No evidence that students upload files to the wiki.	Evidence that at least one student uploads a file to the wiki.

 

Final Overall Rating Questions

For the final categories, answer the following questions on a 1-7 scale (7 being highest)
Overall Participation
Overall, to what extent do students participate in this wiki?

No participation 1

Minimal participation 2

A little participation 3

Some participation 4

A fair amount of  participation 5

Considerable participation 6

Extensive participation 7

Overall Expert Thinking
Overall, to what extent does this wiki promote the development of students’ expert thinking skills—such as ill-structured problem solving, critical thinking, creative problem solving, metacognition, initiative taking?

No expert thinking 1

Minimal expert thinking 2

A little expert thinking 3

Some expert thinking 4

A fair amount of expert thinking 5

Considerable expert thinking 6

Extensive expert thinking 7

Overall Complex Communication
Overall, to what extent does this wiki promote the development of students’ complex communication skills—such as communication and collaboration?

No complex communication 1

Minimal complex communication 2

A little complex communication 3

Some complex communication 4

A fair amount of complex communication 5

Considerable complex communication 6

Extensive complex communication 7

Overall New Media Literacy
Overall, to what extent does this wiki promote the development of students’ new media literacy skills- the skills necessary to critically consume and produce multimedia content?

No new media literacy 1

Minimal new media literacy 2

A little new media literacy 3

Some new media literacy 4

A fair amount of new media literacy 5

Considerable new media literacy 6

Extensive new media literacy 7

Wiki Quality Instrument and Protocol by Justin Reich is licensed under a Creative Commons Attribution 3.0 Unported License.
Based on a work at www.edtechresearcher.com.