rockidscience.com Instructional Design Basics

31May/10Off

Assessing Your Assessment—Part II

Before getting into Instructional Design, I started off as a high school social studies teacher.  While prepping for that field, a main area of study for me was history and in one of my classes a professor once asked me, “why is the study of history important.”

Up until that point I had never really considered the why—for me, it was just something we had to do.  So in answer, I gave the usual, “you should study history so that you don’t repeat the problems encountered in the past.”  I thought that sounded pretty good, after all, there is a famous quote similar to what I said so it must hold some relevance.

However, my professor had a very different reason:

“The study of history is about cause and effect—this event lead to this result, which then lead to this...  So in essence, the purpose behind studying history is that it improves your critical thinking skills and is why history is taught in our schools.”

Hmmmm, now that actually sounds like a real reason for the study, unfortunately, it doesn’t match up with how history is taught. For instance, think back to your old high school history class—what kind of tests did you take; were the questions something like:

  • When did Columbus discover the new world?
  • Who conquered the Aztec empire?

I bet that’s pretty close to the truth—most of the tests you took probably centered on dates, names, and places.  And for the most part, these tests assessed your ability to recall and memorize facts and figures.  What they didn’t do was assess your critical thinking skills.

Now as may be common with my posts you may be asking yourself what this has to do with you. Well it’s simple, this is a common problem associated with test construction, this being, creating test questions that are not focused on the right thing.  Often the tests we create are focused on recall and memorizing facts and figures rather then the real purpose behind our training.

Knowledge Types
When constructing a test a way to avoid this problem is to step back a moment and reexamine your objectives, specifically, you should check to see how your objectives match up to your content knowledge types.

By using knowledge types you can break down any content into the following categories:

  • Fact
  • Concept
  • Rule/Principle
  • Procedure
  • Interpersonal
  • Attitude

These categories are important because they provide you with information on how each objective should be covered and assessed in your course.

In this light your test question needs becomes a little clearer, for instance:

  • If you have an objective that is a procedure, your assessment for that objective should be focused around testing the procedure. “Demonstrate how to …”
  • If you have an objective that is a rule/principle, you should assess that rule/principle,  “Calculate the ….”
  • If you have an objective that is a concept, you should assess that concept, “Distinguish between …”

Levels of Learning
The verbs you use in your objectives will start to highlight what is factual information or recall level and those that are at a deeper level of learning—the application level.  Verbs such as define, identify, list,… are aimed at recalling factual information and as such, like the history tests we took, probably don’t fully address your training needs.

Now it’s hard to talk about verbs for objectives and levels of learning without someone bringing up Bloom.  Bloom and his colleagues did some great work identifying levels of learning and establishing a hierarchy of these levels.  And if you want to use their work to help you develop your objectives and assessment needs you will probably be fine.

I will caution you though to think about your true needs while consulting Bloom’s work.  For instance, synthesis and evaluation are at the top of Bloom’s hierarchy—it is doubtful though that you will need to assess at that level with all of your content.  The important thing to remember here is your goal for the training; is it to change a specific behavior or to create a subject matter expert? Answering this should help you determine what level of learning your need to focus on.

In later posts I will expand on knowledge types and examples of questions to use for each type. For now just make sure you focus your test questions around the real purpose of your training.  Is it recall, is it application, is it to change behavior, or is it to create a subject matter expert?

2Feb/10Off

Assessing your assessment—Part 1

Assessment is a critical part of your training intervention—you must be able to determine whether your users have mastered the materials and are ready to transfer these skills to their jobs. That being said, assessments are often one of the least developed parts in a course.

Why is this?

I’m not sure, but it may have to do with fact that we all have a lot of experience taking tests—for many of us, too much experience!

So we have a lot of experience and this experience has provided us some knowledge on the basics behind test construction.  It’s looks pretty straightforward, how hard can it be?

Well before we start creating questions, let’s go back to our test taking experience and remember our bad experiences. We have all taken tests that were full of give-me questions, covered unrelated content, were difficult to understand and or had any number of common test construction problems.

To help you remember some of these experiences take the following tests

Test Construction Problems 1

Test Construction Problems 2

Test Construction Problems 3

Now that you have had the chance to re-experience these common problems try to remember them before creating your next test.

And keep an eye out for part two and three of this topic, which will provide additional information on creating better tests.