Final Exam: Part II
/The Procedure
Over the course of 12 hours, we performed for 36 people in groups of three. Each group would see four different effects performed by four different performers in an intimate, close up setting.
The effects were:
1. A color-changing deck routine.
2. A coin trick (three silver coins change to three copper coins)
3. A Rubiks cube trick (an instantaneous solve sort of thing)
4. A mentalism trick.
With the first three effects, the objects that had just been magically altered (deck, coins, cube) could all be examined. (With the mentalism trick, there wasn't really anything to be examined. We included it because we thought we might need something that was constant among the groups when analyzing the data, but that didn't turn out to be the case. For the purposes of this write-up, you can ignore the mentalism trick.)
[Note: The point of this exercise was to look at the importance of examinability in regards to the object of the effect. The thing that changes state in some manner. Not the tangential items used in an effect. I say this to preemptively avoid the emails that are like, "Oh, so you're saying I need to have the sharpie examined during the ambitious card?" No, dummy, I'm not.]
Each trick was performed by the same person each time. The only difference in performance from group to group is that half the time the object was handed out for examination and half the time it wasn't. At no point did we actually say, "Please, examine this deck of cards," (for example). Instead it would just be a casual action at the end of the effect to hand out the object(s) to one or more members of the audience.
Some of the groups got to handle each object after each effect, some of the groups didn't get to see any of the objects, and with some groups they examined some and not others.
The Rating Scale
This is, unexpectedly, one of the harder parts of this type of testing. Often, if you ask people to rate things on a scale of 1-10, almost everything will be a 7, 8, or 9. I'm not sure why this is. It may come from school where if you got a 60% you did terrible. We've found we sort of have to train people out of that mindset so that there's a large enough scale to compare things on. If you don't, everything gets compressed. Ask them to rate a 1 in 3 multiple out trick and they give it a 7; ask them to rate vanishing a lear jet in front of them, and they give it a 9. It works for them because they think 7=ok, but not great and 9=outstanding. But we need a more finely tuned scale.
So when we asked people to rate a trick based on "how amazing or impossible it seems" we emphasized that they should make use of the full scale, with a 50 being pretty decent. The spiel we gave went something like this, "Since you've never judged how 'amazing/impossible' something is before, we want to give you some guidelines. If the trick doesn't fool you, then it gets a zero. If a trick fools you, but you think you have a good idea how it might be done, then it would probably fall somewhere in the first third of the scale. If it it completely fools you and you have no idea whatsoever how it was accomplished, then you would likely rank it near the top of the scale. If it's somewhere in-between—maybe you have some vague ideas, but you were still pretty fooled—then it would probably be in the middle for you." This may sound a little too leading to you, but the alternative is to give them no instruction, which we've found doesn't work well.
In all they were asked to rate each effect on three criteria on a scale of 1 to 100.
1. How amazing or impossible the trick was.
2. How likable the performer was.
3. Their overall enjoyment of the performance.
The only thing we were concerning ourselves with was the first question. The other two were just to flesh things out so undo focus wasn't put on the initial question.
The Findings
Here's something I thought we might find going into this testing. I thought we might find that handing out something for examination would just increase how "impossible" an effect seemed by a small amount. My logic was this: either way they know it's a trick, so it's not like being able to examine something is going to all of a sudden make them it's some kind of miracle.
Well... it might not turn your effect into a miracle, but based on the results we got, I can't imagine there's much you can do that will increase the strength of an effect more than giving out the magically altered item to be freely handled and examined by the spectator. The numbers were not close.
Average Score for how Impossible/Amazing the Trick Seemed
Coin Trick - Unexamined - 58
Coin Trick - Examined - 77
Increase of 32%
Color Changing Deck - Unexamined - 56
Color Changing Deck - Examined - 85
Increase of 52%
Rubiks Cube Trick - Unexamined - 39
Rubiks Cube Trick - Examined - 82
Increase of 110%
Over the course of the day, 57 tricks were performed and "unexamined." Of those 57, only 8 of them scored above a 70. Above 70 is what we described as the "I don't even have a vague idea how this could be done," range. When the objects could be examined, they all averaged above that threshold.
Why did the Rubik's trick jump up so much? I'm not really sure because we didn't do a real breakdown with the people in regards to why they rated the things the way they did. My theory though is that when it comes to a coin, the audience understand that it might be suspect, but they probably don't have any good ideas how. But with a Rubik's cube that goes from mixed up to solved in an instant, there are potentially many possibilities. Perhaps there are flaps on the side, or something that slides, or maybe all the colors are projected from some internal computer (who knows?). Whatever their thought process was, the low score for the unexamined performances suggests the people watching had come up with some explanation that they thought was reasonable. Those who got to examine the cube couldn't fall back on that explanation.
I think it's like that Simon Aronson quote: "There’s a world of difference between a person’s not knowing how something is done versus his knowing it can’t be done.” I don't think they watched the cube trick and thought "I know how it's done," but I think they had ideas how it might be done. But if you give them the cube at the end, it hits home that such a thing can't be done with a normal Rubik's cube (or deck, or coin, or whatever).
The Unexpected Result
While I had assumed being able to take a look at the object of a magic effect would make the trick more powerful, I was a little surprised by the magnitude of the difference. But another surprise came when my friend looked at the scores given for "overall enjoyment." When comparing the examined tricks to the non-examined tricks, he found that examination increased the overall enjoyment score by almost 25% on average. And that's pretty significant because they didn't get the speech about using the full 1-100 scale for enjoyment, so most of the "enjoyment" scores were already pretty decent to begin with. So not only does examination make a trick more powerful and deceptive, it makes it more enjoyable.
Is it more enjoyable because it seems more impossible? Is it more enjoyable because the magic is closer and literally more tangible? I'm not sure. And I'm not sure it matters.
Final Thoughts
The next time someone suggest examinability doesn't add to the impact of an effect, ask them if they would do Anniversary Waltz if they couldn't give the card out at the end. Of course not. It would surprise no one to find that Anniversary Waltz is twice as strong a trick because you hand out the card. Honestly, it's probably 10 times stronger.
Is there any trick where a magically altered item can be handed out that you would choose not to? Probably not, certainly not in a casual close-up performance, because you realize the impact it has. But then we delude ourselves into thinking, "Well, when I change this credit card into a dollar bill, I can just put it in my pocket and that's good enough. No one will want to see if if I use my trusty AUDIENCE MANAGEMENT."
Now, part of the reason Anniversary Waltz is so much stronger when it's examined is that it has such an "obvious" alternate solution i.e., the two cards are stuck together with something. But as technology advances these "obvious" solutions are just going to proliferate. So combatting them is going to require more effort than it did 150 years ago when you just had to rule out magnets, mirrors, and leeches (or whatever else they used to use).
As time passes, more and more people will see a piece of visual close-up and think, "I didn't know how that was done, but I didn't get to look at it, so it probably just used some technology I don't understand." And that will be explanation enough for a lot of people.
As we continue down that path, the only way to get people to feel they've truly seen something extraordinary is to leave them, at the end, holding something ordinary.