Guidebook: Designing an Alternative Assessment Where Learners Use GenAI

1. Introduction

The following guidebook will take you through the steps of thinking through your alternative assessment where students are invited to use GenAI to complete the task. There are a few more steps to consider than when designing an assessment that does not make use of GenAI. This guidebook will introduce you to these steps, pose questions to help you consider how you will integrate them into your assessment, and provide resources to help you.

You will note that you are not invited to write the student instructions for the assignment until every piece has been planned – writing the instructions is the second to last step.

2. Check Policies

As a first step, explore your institution’s policies governing the use of GenAI in the classroom. Many have rapidly developed new policies since ChatGPT was launched in November 2022 to respond to this new technology. Sometimes the policy is embedded within an academic integrity policy (i.e., the academic integrity policy was modified to include language about the use of GenAI), and other times a brand-new policy was crafted to address the use of GenAI. You can also look up the HESA AI Observatory to see the list of AI-related policies at different Canadian and International institutions to survey how institutions are responding to GenAI.

While most post-secondary institution policies allow its use in the classroom, some outright forbid it. Even when GenAI is allowed, there are guidelines about how to do so, particularly to ensure student privacy while respecting intellectual ownership. You will want to be aware of these before you proceed, so that you can follow these policies.

If your institution does not allow the use of GenAI in assignments, you may decide to pursue one of the following paths:

• You can continue to design an alternative assessment where students use GenAI as an exercise (even though you will not be able to deploy this assignment in your current courses, but you will gain practice at doing so);
• You can modify the following activity to use GenAI in the creation of an alternative assessment for your students (one that does not require them to use GenAI). The purpose of this exercise then becomes allowing you to explore the use of GenAI in your work, rather than educating students about its use.

3. Consider your L.O.

Course Learning Objective

The first step is to align your assessment to your courses’ learning objectives. Which of your course objectives will this assignment evaluate? Once you have identified this objective, keep it in mind as you continue the design work – this is the skill (or knowledge or attitude) that the assessment must evaluate.

Digital Literacy Learning Objective

Since GenAI is new, most courses will not have had the time to include language pertaining to it in their course learning objectives. However, the use of GenAI may address an existing goal – for example, helping students learn how to research information, how to cite information, how to avoid plagiarism, or broader digital literacy goals.

You may wish to refine the wording of your course learning objective to apply directly to GenAI as you work through this assessment. This is for your own purpose and it will help you stay focused. For example, you may want your students to:

• Explore the applications and implications of GenAI for society;
• Develop critical thinking, digital literacy, and AI literacy skills to evaluate and use GenAI tools effectively and ethically;
• Demonstrate creativity, innovation, and problem-solving skills by using GenAI tools to create original and relevant content for various purposes.

If you cannot tie the use of GenAI to one of your course objectives, consider why you wish to include it in this assessment. Should it be added? Or, do the course learning objectives need to be modified to include it?

Questions to Consider

Cornell University provides a very thoughtful guide to how and when and why to include GenAI in an assessment activity. It presents excellent philosophical considerations when engaging in this type of assessment. Reading through this short document is highly recommended at this stage.

4. AA - Overview

In this course, we will assume that you are familiar with assessments: what they are, the types that exist, and their purposes. Rather, we will zero-in on alternative assessments.

Alternative assessments are, in a way, defined by what they are not. They are not forms of assessments traditionally used in colleges and universities like quizzes and exams. They respond to the critique that while traditional forms of assessments may be able to capture what a learner knows about a topic, they often fail to capture the learner's ability to apply that knowledge in diverse contexts, to synthesize it with other knowledge, to create new ways to use that knowledge, and to evaluate it. You may have picked up that those abilities - Apply, Evaluate, Create - are those at the highest levels of Bloom's taxonomy of learning and they demonstrate that the learner has integrated, deeply, the learning into their understanding.

Alternative assessments go by many names. They are sometimes called non-traditional assessments and include sub-categories, such as

• Authentic assessments that mirror activities that take place in the workplace. If you have an interest, we recommend Dianne Conrad and Jason Openo's excellent open book Assessment Strategies for Online Learning.
• Performance (or performance-based) assessment that focus on demonstrating the ability to use knowledge.

You might recognize alternative assessments through specific assignment types such as concept maps, case studies, two-stage exams, conference papers, grant writing, simulations, serious games, escape rooms, public consultations, debates, virtual laboratories, inquiry-based laboratories, to name just a few.

Alternative Assessments Characteristics

One way to define them is to list their characteristics. Alternative assessments are assessment where:

• The instructions are open-ended. There is rich information provided as part of the instructions and there is not one right way to go about completing the assignment and assessment. Students must take ownership of formulating a plan for how they will address the assignment prompt.
• The assessment doesn't have one right answer. The learners will propose solutions to the assignment prompt that vary depending on what they prioritize as being important and what other knowledge they can integrate towards a response. There is an element of creativity in completing this assessment. You can expect that each student's solution will be unique.
• Learners are required to apply, integrate, create, synthesize, knowledge and skills. Alternative assessments do not usually assess content knowledge and memorization. They assess whether a student can use knowledge and the assessment helps them demonstrate the full range of knowledge, skills, and abilities they can bring to bear in using that knowledge. It's a fuller depiction of a student's abilities.
• Performance expectations are transparent and provided to learners ahead of the assessment.
• Usually, alternative assessments are used as part of scaffolded learning strategy where learners receive feedback about progress. While the end product may be used in a summative evaluation, there is ample formative feedback provided along the way.

Alternative Assessments and Equity

In addition to allowing students to showcase their learning in a way that is more authentic, alternative assessments address issues of diversity and inclusion in assessment. They are more equitable and culturally responsive.

Consider the many (many!) studies that show that underrepresented and underserved populations of students are at a disadvantage when taking exams compared to those in the dominant culture (here are just some studies that explore how tests disadvantage some learners: BIPOC and women and differently abled). For example, in a timed exam, a student who is using English as an additional language may be at a disadvantage compared to a student who is a native speaker. Or, an exam question may use a sports or cultural analogy that some students are not familiar with.

Compare that to the open-endedness of an alternative assessment, where learners use and integrate what they know and choose how they demonstrate their knowledge. They can use their strengths to demonstrate their learning rather than be hamstrung because their identity does not match their instructor's. It can be a more inclusive format of assessment that leverages the diversity in the classroom, rather than penalize it

You may wish to review OntarioTech University's webpage on Alternative Assessments, which provides a table comparing the characteristics of traditional and alternative forms of assessment.

5. AA - Pick One

Tour of Alternative Assessments

The assessment that you will develop this week should be an alternative assessment. To help you find the right one, consult A Guide to Alternative Assessments [PDF], written by Eliana. This guide provides a comprehensive list of over 35 alternative assessments formats. Each one is summarized in a brief one-pager that includes links to examples, rubrics to evaluate the student products, and additional resources.
(Note: This document is a revision of the one available on the York University website, and the broken links have been fixed).

If you didn’t find anything suitable, consider perusing the Champions of Reimagined Assessment, a YouTube playlist. [In the video, click on the 3 bars at the top of the video to see the playlist and choose the video you wish to watch; alternatively, click on YouTube at the bottom right of the video to open a YouTube page where you can see the playlist]. Eliana solicited alternative assessment practitioners from around the world to submit a short video where they describe an alternative assessment they use in their classroom. Hear about their experience first-hand.

GenAI to Brainstorm AA Ideas

Still didn’t find a format that would be suitable for your classroom? Consider asking a GenAI to assist. Use ChatGPT or Bing in Creative Mode to solicit ideas of alternative assessments you could use in your course context.

6. Define Your Assessment

Type of Assessment

In defining your assessment, one thing you should have in mind is the sort of assessment you are planning for. Here are three possibilities:

• A formative assessment is used to provide ongoing feedback and guidance to your students as they are learning a concept. They may or may not be graded.
• A summative assessment is used at the end of a unit or course to evaluate whether students achieved the desired objectives. These are usually graded.
• A combination of both formative and summative assessments that balance the process and product of learning. This is usually scaffolded, meaning that students are guided to complete the assessment in steps and provided low stakes feedback along the way so that their final submission can respond to the feedback and capture their performance.

Your Assessment

It’s time to combine the previous things you have explored: the course learning objective(s) that you want the assessment to evaluate, the alternative assessments that caught your interest, and how GenAI may assist students in completing the task to define your assessment.

You can think of those three components in the following way:

• The learning objective is the WHY (the goal);
• The alternative assessment is the WHAT (the deliverable format);
• The GenAI integration is the HOW (the task or data that must be integrated in the deliverable).

At this point, you want a general overview of the assessment – a description that is a couple of sentences. We will flesh out each component in later steps.

For example, consider a situation where your learning objective is for students to be able to tell apart what is science from other approaches to knowledge. The alternative assessment that caught your eye was a concept map because you want students to create links between concepts as they consider the characteristics of science knowledge. How might you integrate GenAI into this assignment? Here are a few options to consider:

• Students give a GenAI chatbot the following prompt: “Give me a list of 25 words or concepts associated with scientific knowledge and process.” Students then must use the 25 words/concepts they are provided and arrange them into a coherent concept map that links concepts appropriately (to show their relationship).
• Students create a concept map of what science is and how science works. For each concept, students must add at least two examples. Students are invited to use a GenAI tool to research and evaluate examples. They are responsible for choosing illustrative ones that showcase the concept clearly and succinctly.
• Students have to create a flowchart that another person could use to interrogate themselves as to whether a form of knowledge is scientific. The flowchart will ask Yes/No questions and, depending on the response, the flowchart helps the person identify the type of knowledge that they are starting with. Students can use ChatGPT or other GenAI tools to formulate the questions in their flowchart, but they are responsible for ensuring the integrity and accuracy of the whole flowchart.

Step 1: Divergent Thinking. Assembling those three components – learning objective(s), alternative assessment, and GenAI requires creativity. Begin your exercise by brainstorming as many assessment ideas as possible. Don’t shut down any ideas at this stage, just capture them, even if they are imperfect. If you are struggling, don’t forget that you can ask GenAI for help brainstorming ideas.

Step 2: Convergent Thinking. Once you have engaged in a good brainstorming session, you need to refine your list of ideas. Here are some criteria to use:

• Review your list with an eye to the learning objective(s). In fact, consider the alignment of the learning objective(s), the alternative assessment, and the use of GenAi in this assessment. Some of your ideas will more directly align with your learning objective(s) than others. These are the ones you should continue to work with.
• Consider the practicality of each idea. Perhaps they require tools, materials, or time that you do not have.
• Review the ethics of using the GenAI for your idea. Does it require students to input personal information into the tool? Then maybe this isn’t the right activity. Does it teach students to cite sources appropriately? Then maybe this meets a digital literacy objective in addition to your target learning objective and is appropriate.

Resources

This is probably the hardest part of the process. If you want assistance, consider posting in the Forum: Brainstorming Sandbox. This is your sandbox to use how you see fit – consult your peers, brainstorm ideas, support one another. Your peers and your facilitators may have suggestions to help you.

If you need to consult them, remember the following resources:

• A Guide to Alternative Assessments (35 ideas of alternative assessments)
• Assessment Ideas for an AI-Enabled World (~40 cards of assessments that use GenAI)
  
• 101 Creative Ideas to Use AI in Education
• Assigning AI: Seven Ways of Using AI in Class

7. Consider Scaffolding

Consider whether there is any way to scaffold your assessment. This means breaking it up into smaller and more manageable tasks that allow students to receive and incorporate feedback along the way. Scaffolding assessment can help students improve their learning outcomes, develop their metacognitive skills, and reduce their anxiety and cognitive load. Through scaffolded assessments, students learn while completing the assessment – learning the learning objectives and also learning how to learn.

For example, in an un-scaffolded assessment, you would assign “an essay,” where students have to figure out how to break down the task on their own and receive no feedback along the way. If they make a mistake at one of the stages in the production of the final essay (e.g., if the sources they find are not primary research papers and this was a requirement for their work), they will fail (or do poorly on) the assignment, even though they may understand how to do the other tasks.

In a scaffolded assignment, the task is broken down into stages, and students obtain feedback at each stage. They then incorporate the feedback to correct any mistake and ensure they meet the expectations for the assignment. In the example of an essay, you could break up the assignment into the submission of a bibliography, then an outline for the essay, then the introductory paragraph, then the citations in the correct citation style, and finally the whole essay. Each of the submission can be graded with low stakes grades to ensure they are done diligently (though often students take them seriously since they realize the opportunity this gives them to do well on the final assessment).

If you want to scaffold your alternative assessment, consider the steps required to complete the whole project, and make it a requirement for students to submit a product at the end of each of these steps. You can assign a small grade for submitting the work to encourage students to take it seriously. This will also spread out the grades for the assessment and reduce anxiety.

Consider the sort of feedback that would be helpful to improve student work at each step. For example, in the essay example provided above, the feedback on the outline may focus on the structure and strength of the argument, and the introductory paragraph may be a good place to make comments on the quality of the writing.

Sources of Feedback

Some instructors may be legitimately concerned about increasing their workload by scaffolding their assessment. Consider that you are not the only source of feedback for students. Here are possible sources of feedback:

• Instructor Feedback: Instructor feedback is the process of providing guidance, suggestions, or evaluation from the teacher on a specific task or assignment. Instructor feedback can help students clarify their expectations, identify their strengths and weaknesses, and monitor their progress. It can also provide students with expert knowledge and support. Instructor can use timely, specific, and actionable feedback to help students improve their work.
• Peer Feedback: Peer feedback is the process of giving and receiving constructive comments from other students on a specific task or assignment. Peer feedback can enhance students’ motivation, collaboration, and critical thinking skills. It can also provide students with multiple perspectives and insights on their work. To scaffold peer feedback, you can use strategies such as peer review, peer editing, or peer assessment. You can also use rubrics, checklists, or guidelines to help students give and receive quality feedback. In seeing other examples and assessing other students’ work, students learn what constitutes good, and a not so good, work.
• Institutional Support Resources: Some institutions offer library workshops, and support in the form of writing centres and peer tutors to help learners achieve their academic goals. An assignment could make use of such academic support systems to provide guidance and feedback on a step toward the final assessment submission.
• GenAI Tool Feedback: GenAI tools can act as personal tutors for each student. GenAI can suggest sentence structures, alternative arguments or perspectives, or sources. Instructors can suggest prompts, queries, or challenges to help students make use of GenAI tools to receive feedback.

The Steps

If you decide to scaffold your assessment, consider the following steps:

1. Identify the steps that students must perform in order to complete the assignment.
2. Identify a deliverable that students could submit at the end of each step that would allow someone else to provide feedback on the project’s development.
3. Identify the goal of each step. For example, the goal of creating an outline in an essay is to map out the main ideas and get a bird’s eye view of the structure of the argument. Aim to provide feedback on that aspect of the deliverable.
4. Determine who will provide the feedback and how they will directed to provide the appropriate feedback at this stage. For example, if using peer feedback, a worksheet that each peer has to complete, that directs their feedback to certain aspects of the work, can help to ensure the feedback is focused. If you plan to ask students to use GenAI to provide feedback on their writing, design an effective prompt for this task (more on this in the next section of this book).
5. Determine whether low stakes grades will be assigned to each (or some) of the deliverables.

Resources

If you are interested in learning more about scaffolding an assessment, consider the following resources.

• This article provides a good overview of the rationale and strategies for scaffolding assessments. (Shepard, L. A. (2005). Linking formative assessment to scaffolding. Educational leadership, 63(3), 66-70.)
• University of Melbourne guide to creating scaffolded assessments;
• The pressbook Encouraging Academic Integrity Through Intentional Assessment Design has a chapter on scaffolded assessments with an example for a health science class;
• The pressbook NC Course Re-Design, Renewal, and Development Guide_Alpha has a short chapter on scaffolded assessments that describes its components such as feedback;
• The Edutopia article Getting Students to Engage with Feedback provides some suggestions for ensuring that students take feedback seriously… The nature of scaffolded assessments makes students inherently care about the feedback, and this article proposes strategies to ensure that every student understands its importance.

Note: Images created using Bing Image Creator (September 2023)

8. Select a GenAI Tool

Your next task is to select a GenAI tool that is best suited for the students’ task. Even if you give students the option of selecting a tool of their choice, you may still want to suggest one and provide instructions for using it, so that students who are unfamiliar with GenAI have some support.

In selecting an appropriate AI tool, consider the following:

• TASK. What is the nature of the task that the GenAI will be doing to assist students? For example, if the task is to generate an image, select a GenAI tool designed to accomplish create images, such as DALL-E or Bing Image Creator accessed through Bing in Creative Mode). Other tools, such as ChatGPT, would not be able to accomplish this task.
• ACCESSIBILITY. Consider whether the tool meets accessibility requirements. For example, you probably want to select a tool that is freely available to all students, and one that is available in Canada. You may want to consider other criteria for students who have accessibility needs, such as whether the GenAI tool is available to students who use a screen reader to access the internet.
• PRIVACY. Even when they are freely available, most GenAI tool require registration to create a login, which means providing an email address. This could violate your institution’s privacy policies. However, there may be ways around it, like using a dummy email account to access the tool or providing a shared class login credential.
• RELIABILITY. Some GenAI tools are more accurate and reliable than others. Consider researching whether the tool you have chosen will be sufficiently robust and trustworthy to meet the needs of the assignment.

If you are new to GenAI and would like some help finding suitable tools, begin by reviewing the chapter GenAI Tools in the book Introduction to Generative AI in Assessments for some ideas. You may also consider asking a GenAI chatbot, like ChatGPT or Bing in Creative Mode, for a list of GenAI tools that could achieve your desired task.

Always try the tool’s ability to support the assignment before choosing it. Some tools will provide better responses than others for most prompts or requests.

9. Engineer a Prompt (optional)

Why Engineer a Prompt

Writing a good prompt, one that will cue a GenAI to give you the type of response you want, is fast becoming an essential digital literacy skill. This skill is called prompt engineering.

For your assignment, there are at least a couple of reasons to investigate what a good prompt might look-like:

• As a learning opportunity, where you impart that knowledge to students, to teach them how to write a good prompt (which will help them in this assignment, and beyond);
• As an assignment instruction, where students have to input the prompt you give them to use the GenAI in the way that you intended for the assignment (e.g., if the GenAI will work as a tutor, providing feedback on their essay, and behaving in the way that you intended).

Example of a Prompt

Before looking at how to write a good prompt, it is worth examining the following example, written by Dr. Ethan Mollick, an expert at using GenAI in education. This will convey the complexity of writing a good prompt. The following prompt could be given to students to input into the GenAI chatbot, and the GenAI would then function as a tutor, questioning students, providing contextually appropriate examples and analogies, and interacting in a conversation with them over several steps.

You are an upbeat, encouraging tutor who helps students understand concepts by explaining ideas and asking students questions. Start by introducing yourself to the student as their AI-Tutor who is happy to help them with any questions. Only ask one question at a time. First, ask them what they would like to learn about. Wait for the response. Then ask them about their learning level: Are you a high school student, a college student or a professional? Wait for their response. Then ask them what they know already about the topic they have chosen. Wait for a response. Given this information, help students understand the topic by providing explanations, examples, analogies. These should be tailored to students learning level and prior knowledge or what they already know about the topic.

Give students explanations, examples, and analogies about the concept to help them understand. You should guide students in an open-ended way. Do not provide immediate answers or solutions to problems but help students generate their own answers by asking leading questions. Ask students to explain their thinking. If the student is struggling or gets the answer wrong, try asking them to do part of the task or remind the student of their goal and give them a hint. If students improve, then praise them and show excitement. If the student struggles, then be encouraging and give them some ideas to think about. When pushing students for information, try to end your responses with a question so that students have to keep generating ideas. Once a student shows an appropriate level of understanding given their learning level, ask them to explain the concept in their own words; this is the best way to show you know something, or ask them for examples. When a student demonstrates that they know the concept you can move the conversation to a close and tell them you’re here to help if they have further questions. (Mollick & Mollick, 2023)

Consider entering this prompt into a GenAI chatbot to see the results of using this prompt. You will note, for example, how some of the words used in the prompt get re-used by the GenAI as it interacts with students. You may also notice that it sometimes gets stuck in a loop, or that it goes off on a tangent. These are important things to note about the behaviour of a GenAI – something we will return to in the next chapter.

Guidelines for Writing a Prompt

WRITING

When writing a prompt for a GenAI chatbot, consider including the following (adapted from Mollick & Mollick, 2023):

• ROLE. Tell the GenAI about the role they will play in the conversation. Who is it? Is it acting as a peer, a teacher, a mentor, an evaluator. You might consider also including some “personality characteristics,” such as friendly, or polite, or critical, to help the GenAI behave in the way you intend.
• GOAL. Tell the GenAI what its purpose is in the interaction. For example, if the purpose of the interaction is to provide feedback on student-written work, specify whether the goal is to provide feedback on the structure of the argument or on the grammar and spelling of the writing.
• STEPS. You may have noticed in the prompt above that the GenAI is told about how to behave at each step. This is done not only to provide instructions for a conversation that lasts several steps, but also to control the GenAI’s analytic response. For example, if you did not tell the GenAI to wait for a response, it might come up with the entire strategy for responding in this conversation and might therefore make the response less appropriate to the student’s context. Asking the GenAi to wait for a response means that it uses input provided at different stages in the conversation and uses it to come up with a revised plan at each step.
• PERSONALIZE. You may either input information about the context of the students (e.g., the students are in their first year of university and are studying biology), or you may direct the GenAI to request this information as part of the interaction and to customize its response accordingly.
• CONSTRAINTS. These are “guardrails,” boundaries you set on the GenAI’s responses. For example, you may not want it to provide two or three suggestions on ways to revise a student’s essay, but never tell the student which one to use and revise the work.
• PEDAGOGY. You might want to provide some suggestions to the GenAI about common student misconceptions or challenging concepts and direct the GenAI to ensure that these are addressed.
• REVIEW. Try the prompt yourself and examine the sorts of responses you get. Based on this, you may need to tweak the prompt. Note that some prompts work better on some chatbot than others, so you might want to try it on different GenAI tools (e.g., ChatGPT and Bing in Creative Mode) to see which one gives you the best output.

GRAPHIC

When writing a prompt for a GenAI image generator, consider the following:

• VISUAL DETAILS. Be visually specific about what you want the GenAI to draw. For example, writing “one person is the other’s mentor” does not tell the GenAI what you have in mind. You may consider something like “one person stands at a blackboard and is writing; the other is older and stands behind them and is pointing at the writing on the board.”
• STYLE. Provide instructions about the style of drawing that you would like. “Digital Art” often gets good results. There are other styles, such as oil painting, water colour, impressionist, graphic design, art deco, graffiti, cartoon, manga, etc. If you can’t think of more options, consider asking a GenAI chatbot to provide ideas.
• ITERATE. Think iteratively. Once the GenAI gives you a few options, consider directing it to modify the images in a way that better suits your expectations (e.g., now include a mountain in the background). Note that when the GenAI will do this, it will generate completely new images, not take the already generated images and add the new request into it.

Guides (More Resources)

Curious about how to write a good GenAI prompt? The following resources can help you learn more. (Note: the ones with *** in front of the author(s)’s name (and bold)  are particularly recommended)

Liu, D. (2023). Prompt Engineering for Educators – Making Generative AI Work for You. The University of Sydney. https://educational-innovation.sydney.edu.au/teaching@sydney/prompt-engineering-for-educators-making-generative-ai-work-for-you/

Mollick, E. (2023). A Guide to Prompting AI (For What it is Worth). One Useful Thing. https://www.oneusefulthing.org/p/a-guide-to-prompting-ai-for-what

Mollick, E. (2023). Power and Weirdness: How to Use Bing AI. One Useful Thing. https://www.oneusefulthing.org/p/power-and-weirdness-how-to-use-bing

*** Mollick, E.R., & Mollick, L. (2023). Assigning AI: Seven Approaches for Students, with Prompts. arXiv preprint arXiv:2306.10052. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4475995

Parsons, G. (2022). The DALL-E 2 Prompt Book. DALL-E. https://dallery.gallery/the-dalle-2-prompt-book/

Shieh, J. (2023). Best Practices for prompt Engineering with OpenAI API. OpenAI. https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api

*** Wright, L., Tape, R., Dias, M., Zerdin, J., Krbavac, M., Rogic, N., Chan, J., & Kolhatkar, V. (2023, July 31). Promptathon Event [Blog entry with Slides]. University of British Columbia. https://blogs.ubc.ca/promptathon/
[Note: Open the Google slides from this presentation for an excellent explanation of what large language models are, how to construct effective prompts (with several suggestions not mentioned above), and several suggested uses of GenAI chatbots – for example, to generate a glossary from a text or to create educational games.]

The following article provides prompt ideas to support the type of tasks commonly performed in the work place, which could inspire your own prompts.

Drapkin, A. (2023). 40 Best ChatGPT Prompts to Save You Time at Work. Tech.co. https://tech.co/news/best-chatgpt-prompts

What To Do for Your Assignment

Now that you have examined how to write a good prompt, design a good prompt for your assignment. Remember that you might be designing this prompt to test out how a student will use the GenAI to get assistance on their assignment (in other words, you are testing to make sure that the GenAI can assist students and you are investigating some of the best prompts to use… you may then share these tips with students in the assignment instructions). Alternatively, perhaps students will be required to use a prompt you provide them in the assignment, and this is your opportunity to craft the best one.

In addition to trying the prompt and tinkering to improve it, consider sharing it on the Brainstorming Sandbox to solicit ideas from peers that you might not have thought about. Don’t forget to share your goal and context for the prompt, so that others can provide feedback that is on target.

Note: Images created using Bing Image Creator (September 2023)

10. Introducing GenAI to Learners

One of the goals of integrating GenAI into any assignment is to educate students about its use – how to use it effectively, responsibly, and making students aware of the tool’s limitations. A step in developing your assessment is therefore to plan how the GenAI tool will be introduced to your learners.

Some of the topics you may wish to raise with your students include:

• NATURE of GenAI. Since some students will encounter GenAI for the first time through the assignment, it is worth preparing the experience to temper the students’ expectations. Notably, it’s worth noting that while a GenAI chatbot can seem like a conscious entity, it is not. The tool is a language model, predicting what word should follow another. The GenAI does not understand what it is saying, and it has no experiential knowledge of the concepts it is talking about. It’s also worth noting that the GenAI’s responses are unpredictable and that the same prompt can generate different responses. This information may help students set their expectations for the interaction and may help some students who are distraught by the experience of interacting with an entity that seems sentient.
• PLAGIARISM. Have a discussion with students about what constitutes academic integrity, why it is important, and how the rise of GenAI challenges traditional notions of academic integrity (e.g., while there may be clearcut cases of academic dishonesty, such as submitting an essay that was written by a GenAI, is it considered plagiarism if a student consults a GenAI tool while brainstorming an outline for an essay?). You should address how the tool may be used in the assignment so that it does not infringe on your course’s academic integrity policy and provide examples of what’s allowed and not allowed in using GenAI in your course. You may also want to make your students aware that policies for the use of GenAI may vary in each course and that they are responsible for checking with the syllabus and their instructor to ensure they do not inadvertently violate academic integrity policies.
• RESPONSIBILITY. Make students aware that they are responsible for the work they submit. A GenAI tool is a tool, just like a calculator or a spreadsheet software are a tool. The tool can be used to accomplish a task, but ultimately, they are the ones directing its use to solve a problem or accomplish a task. GenAI tools can be unpredictable and generate inaccurate information or hallucinate (e.g., GenAI makes up sources on a regular basis when asked to cite its work, and while these sources seem legitimate, they do not exist). Students should also be made aware that, if challenged, a GenAI tool may argue, maintaining that it is accurate when it is not. Students are responsible for verifying any information they include into their projects, verifying its sources and accuracy.
• DIRECTING THE GenAI. Let students know that when they use a GenAI tool, they are the ones in charge. They should not let the GenAI lead the conversation; rather they should be the ones to do so. The GenAI tool can sometimes become trapped in a loop and will keep providing responses that do not advance the conversation. It’s okay to tell the GenAI to move on. Similarly, sometimes the GenAI will go off on a tangent, seeming to answer a different prompt than the one that it was asked. It’s okay to re-focus the GenAI by telling it that it is off base.
• ETHICAL CONSIDERATIONS. Educate students about some of the ethical issues surrounding the use of GenAI. Tell them about some of the privacy and security concerns and provide clear guidelines about how to use the tool so that they are not compromising their own security and privacy. You may also wish to tackle other issues such as accessibility, intellectual property, and laws and regulations as part of a digital literary objective.
• PROMPT ENGINEERING. You may wish to provide instructions, tips and tools, or resources to help students use the GenAI tool effectively. This can include a step-by-step guide (or short video tutorial) to show students how to access and use the tool generally, or more focused instructions on how to construct a good prompt. You may want to tell them that a conversation with a GenAI is iterative, so they can continue to refine the GenAI’s response towards an answer that is closer to their target output. Finally, you may want to make students aware that different GenAI tools respond differently to the same prompt, and that students may therefore wish to explore several GenAI tools to determine which one provides the best responses for their task.

Consider consulting Mollick & Mollick (2023) for other ideas about how to introduce GenAI to students and what to make them aware of as they work with this new tool. You may also wish to review some of the items brought up in this FLO course as part of the discussion How to Prepare Students to Use GenAI.

To Do

Create a plan about how you will introduce the GenAI tool to students. Make a list of the topics that you would like to cover, and for each one, make a decision about how students will explore this topic.

For example:

1. Introduction to GenAI tools [Lecture]
1. What it is
2. Why we will use it in this assessment
3. What it can and cannot do
4. How to use it so that it is still considered the students’ own work (i.e., avoiding academic dishonesty)
2. Ethics of using GenAI to support academic work [Class discussion]
3. Instructions on how to use it
1. Tutorial for using Bing in Creative Mode [Video]
2. Prompt engineering [Written instructions]

Once you have crafted your plan, go ahead and develop the materials needed to carry it out. In the example above, it would mean:

Step 1: Developing the lecture notes and accompanying slides

Step 2: Thinking through the questions to start the discussion and the format of the discussion (e.g., think-pair-share)

Step 3: Producing the video and writing the instructions.

Note: Images created using Bing Image Creator (September 2023)

11. Learner Meta-Cognition

Supporting the development of students’ meta-cognitive skills (learning how to learn) is a goal of many post-secondary educators. Indeed, the ability to gauge where one stands with regards to learning a new task and how much more needs to be learned is a key skill that professionals must master (Lockyer et al., 2004).  

Teaching students how to reflect becomes particularly important when introducing a new tool, such as GenAI, that seems to be able to do many of the same tasks that a student has historically done in their academic work.

After using the GenAI tool, you may ask students to reflect upon some of the following:

• TOOL EFFECTIVENESS. How “good” was the GenAI tool? Were its responses accurate? Precise or general? Contextualized? Did it hallucinate? Given these observations, how useful would the use of such a tool be for different purposes? (e.g., in developing new medicines, in supporting students for learning, in helping computer programmers, in helping them with their next writing assignment, etc.)
• CRITICAL EVALUATION. What approaches did the student use to sift through the GenAI’s responses and determine the ones to trust from those that needed to be investigated further? Under what circumstances is the tool most accurate? Should users have a priori knowledge of a topic before using the tool? How would you recommend that a user work with a GenAI tool on a class project (which steps should a learner use to conduct research and when in the process should they use a GenAI tool?)
• USAGE. When using the GenAI tools, which prompts resulted in the best/worst responses? Why was this a good/bad response? Did students discover any tricks to write an effective prompt?  When did the GenAI tool get trapped in a loop or when was it more likely to hallucinate? How did students do when they encountered this situation? How did they make the most of the iterative nature of the interaction with the tool? Were some GenAI tools better than others and why? What recommendations would they make to a new user of the tool? (i.e., to students doing this assignment next year)

To Do

Review your assignment’s learning objectives and consider whether including a meta-cognitive component (i.e., a student reflection on the use of GenAI and what they learned from it) makes sense. If this would help learners achieve the learning objective of the assignment, consider what you will ask learners to reflect upon and how they will share their thoughts.

Resources

Aronson, L. (2011). Twelve tips for teaching reflection at all levels of medical education. Medical Teacher, 33(3), 200-205. https://doi.org/10.3109/0142159X.2010.507714

(One of the fields where reflection has been well-integrated into education and professional practice is medicine. This article provides 12 concrete suggestions for teaching students how to use reflection to improve their practice.)

Mollick, E.R., & Mollick, L. (2023). Assigning AI: Seven Approaches for Students, with Prompts. arXiv preprint arXiv:2306.10052. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4475995
(This article has been referenced many times in this course. In it, the authors provide suggested assignment instructions including instructions for a reflection at the end of the assignment.)

Tanner, K. D. (2012). Promoting student metacognition. CBE—Life Sciences Education, 11(2), 113-120. https://doi.org/10.1187/cbe.12-03-0033
(Note: this article is written for an audience of biology educators but can easily be applied to other disciplines and indeed is a staple article in the field of metacognition in undergraduate learning)

Centre for Innovation and Excellence in Learning (n.d.). Metacognition [compilation of blog posts]. Vancouver Island University. https://wordpress.viu.ca/ciel/category/metacognition/
(Series of blog posts on the topic of reflection and student meta-cognition. The post Metacognition: Valuable Resources on Thinking about Thinking shares many resources on the theory and practice of meta-cognition in the classroom.).

Note: Images created using Bing Image Creator (September 2023)

12. Alternatives for Learners

An important aspect of developing an assessment that invites students to use GenAI tools is to make it “GenAI-optional.” Most (all?) GenAI tools do not comply with the BC Freedom of Information and Protection of Privacy Act (FIPPA) (and therefore your institution’s policies) because they ask users to provide personal information as they register to create an account. Students can provide their personal information if they choose to, but they should not be required to do so to complete their coursework.

There are some ways around this. Here are a few options:

• TOOLS To ACCESS GenAI ANONYMOUSLY. There are a couple of tools at your disposal that allow users to access GenAI tools without requesting that they login.
  

  • TalkAI. This is a website that taps into the GPT model but does not require a login. Note that it is a bit “spammy,” showcasing ads on their website. However, it does not require any registration.
  • Perplexity.ai. This website is another large language model (i.e., it is comparable to, but not, GPT). The answers you get might be a little different to those you would get on ChatGPT. Sometimes that can be a good thing! And, it does not require a registration.
• SHARED GenAI ACCOUNT. Consider creating a shared class account on the GenAI tool website. You can do this by first creating a “dummy” email account (one that you can access, since many registration processes require you to retrieve an activation email message). Then, share the login information with the class. Everyone will use the same login information. Note that this means that students will be able to view every user’s search history. You will still want to educate students about how to protect their private information when using this shared account (e.g., not entering any personally identifiable or sensitive information). Some GenAI tools will suspend an account after it has placed too many requests that they deems inappropriate, so you may want to preface the use of this shared resource with instructions about how to use it responsibly.
• SAMPLE CONVERSATIONS. Another option is to play the role of a student as they work with the GenAI on their assignment and to share a printout (a PDF) of the resulting conversation. You may want to make a few versions available where you ask slightly different questions, so that students do not all use the same data. Providing “access” to the GenAI in this way will meet a learning objective of familiarizing students with this tool (they will observe how a typical interaction with a GenAI tool unfolds), yet it will preserve the privacy of each student.
• EQUIVALENT ALTERNATIVES. You may also want to give students an equivalent alternative for the assignment. This means providing instructions for completing the assignment in a way that does not make use of the GenAI tool, yet still achieves the same learning outcomes and requires the same level of student engagement. Students who opt for this option should not be placed at a disadvantage over those that do. If the learning objective of the assignment includes digital literacy skills related to GenAI, then consider the above option (Sample Conversation), as it will simulate the type of interaction that a student using the tool would experience.

To Do

Whichever approach you choose, you must devise an alternative option for students who opt to not use GenAI in your assignment. Consider how they may do so and add these instructions to the student assessment instructions.

13. Assessment of Learners’ Work

Students need guidance on what’s expected of them as they complete their assignment. This is particularly true of alternative assessments where several criteria could be used to determine the quality of a submission.

What are the markers of a quality submission in your class? How much are each of the criteria you plan to use to assess the work worth? For example, do you value professional writing (writing free of typos and grammatical mistakes) above the depth of the research conducted, or is a professional product a “nice to have” that pushes a good project into the “great” category?

Students need transparency about how their work will be assessed. This not only reduces anxiety but also helps them learn, by focusing their attention on what a good deliverable looks like.

Rubrics

There are many ways to convey this to learners. One way is to create a rubric that outlines the criteria by which their work will be assessed, how much each criterion contributes to the quality of the deliverable (how many points each is worth), and what the performance looks like for different levels of proficiency. The following webpage from the University of Waterloo’s Centre for Teaching Excellence provides an overview of the why, what, and how of rubrics.

Consider developing a rubric to communicate your expectations to learners. The rubric can be used for summative assessment, meaning that you will use it to assign a grade to each learner for their submission. You can use the same rubric in formative assessment. For example, in a scaffolded project, students may first submit their project to peers, who use the rubric to provide feedback about the strengths and weaknesses of the student’s work. The student then uses the feedback to improve the work before their final submission. Note that when peers use the rubric to evaluate their colleague’s work, they gain a better understanding of what constitutes good quality work (by examining an example, which they will contrast to their own work) In other words, this formative assessment activity helps both the evaluator and the evaluee learn.

A GenAI chatbots can help you create the first draft of a rubric for your assessment, but you will likely need to customize it for your course.

VALUE Rubrics

The American Association of Colleges and Universities (AAC&U) developed a set of 16 rubrics to assess skills commonly taught in an undergraduate education – skills like Critical Thinking, Ethical Reasoning, Quantitative Literacy, Teamwork, Oral Communication, Problem Solving, and Written Communication. These rubrics are called the VALUE rubrics (VALUE is an acronym for Valid Assessment of Learning in Undergraduate Education). They are called “valid” because much research has gone into their development, to ensure that they evaluate the skill that they say they evaluate. These rubrics are open educational resources, available free of charge to any educator. Consider adopting a VALUE rubric, creating your own by combining elements from different VALUE rubrics, or adapting a VALUE rubric.

Student Co-Created Rubrics

Consider another way to use a rubric to communicate what excellent work looks like to your learners. Give learners several examples of final products. These examples should vary in their quality. Some should be excellent, some less so. You can collect excellent examples from past student submissions (ask them for permission to show their work to future classes). You can ask a GenAI tool to create “less good” examples.

When you assign the project, give students 3-5 examples of final products and ask them to compare them. Then, lead a class discussion where students identify elements of excellent submissions. Together, co-create a rubric. Doing it in this way – with student input – will ensure that they understand what each criterion is and the type of excellent work to aim for.

Consider this: As an educator, you have examined dozens of student products, so you are able to tell apart good from not-so-good work. Students don’t have the benefit of this experience. This activity will expose them to a variety of examples and help them differentiate what makes good from great work.

To Do

Develop your evaluation plan for the assessment. Some of the things you may want to consider as you do so include:

1. Review the learning objectives of your course and for this assignment. These should be top of mind as you develop your evaluation plan (your assessment criterion should align with the learning objectives).
2. If the assessment is scaffolded (if students will be submitting work in stages) consider whether each (or some) of the intermediate products will be assessed for grades. For example, in an essay, would the submission of an outline merit any grade? How about a first draft? If you plan to assign a meta-cognitive component to the project (e.g., a reflection), will it be graded?
3. Articulate the assessment criterion that you will use to assess student submissions. What are the elements of an excellent submission, one that shows that students have mastered the learning objective of the assignment?
4. What is the relative weight of each of these criteria?
5. How might you describe excellent performance for each criterion? Do you have examples that you could share with students?
6. If you decide to use a rubric to communicate your expectations, create it. Also decide when and how it will be used in the assignment (e.g., will there be a peer review formative activity, as well as a final summative evaluation using this rubric?)
7. If you decide that you will not use a rubric, how will you communicate your expectations to learners? You could, for example, provide several examples of past student works with their associated final grades, so that students can gain a sense of what constitutes excellent, good, and developing submissions. Even if you do not use a rubric, you may want to note how you plan to ascribe a grade to each submission, for example, a plan to deduct 2 points for each spelling mistake.

14. Instructions for Learners

In the previous chapters, you have planned each of the elements of your alternative assessment that invites learners to use GenAI. Now is the time to put together everything that you have planned into clear and complete instructions for your students.

As a reminder, consider including language (and/or resources) to address each of the following elements of your assessment:

• Learning objective of the assessment;
• The task or deliverable to be completed;
• The process to be used in completing this assessment, including:
• The use of GenAI, along with instructions for its use;
• An alternative option that does not require the use of GenAI and instructions/resources for doing so;
• Parameters for completing the assignment. For example, whether teamwork is encouraged or forbidden; which resources may or may not be consulted; requirements for citation of work used (especially for GenAI resources), etc;
• (optional) A description of the scaffolded elements / Stages of submission of the assessment;
• (optional) A description of the meta-cognitive (reflective) component;
• The assessment plan, including clear and transparent criteria by which the work will be assessed;
• (optional) Examples of past student work, along with indication of the grade obtained so that students can gauge how the work aligns with your expectations;
• Logistics of submission: Format, where/how to submit, due date, etc.

15. Monitor the Assessment

This brings us to the final step in the development of an assessment: deployment and monitoring. In other words, you will try out, and then evaluate the assessment.

You may want to decide, in advance, how you will assess the performance of this assessment and how you will collect data to determine how to improve it. Some of the ways in which you could collect data on its performance include:

• What appear to be common points of confusion for students about the assignment? Consider modifying the assignment instructions to address this.
• What are aspects of the assessment that students tend to do well and aspects where many struggle? Consider whether this is an important aspect of the learning for this assignment (i.e., whether it aligns with the learning objectives). If it does, you may bolster opportunities for learning this objective. If the aspect of the assignment that is challenging is not important for your course objectives, consider remove it.
• Collect student perspectives on the assessment – for example by administering a short anonymous survey. How are students responding to the assignment? What do they like and what critiques do they have?
• As you grade assignments, consider asking students who submitted excellent work whether they would give you permission to use their work as examples in future courses. You can add these samples in the instructions/ resources for the assignment.

Based on the data you collect and the reflections you make, iterate, and improve the assignment. It will only get better.

To Do

Congratulations – You are done with your assessment design!

The final step is to share what you developed with your peers, review what others have done, celebrate, and share ideas with one another.

Note: Images created using Bing Image Creator (September 2023)