Iteration Testing
/A year ago today I found myself wondering what I was in the unique position to do as the person behind this site. I had always considered the focus-group testing I’ve helped conduct to be some of the most useful content on this site. While other people had tested magical concepts before, and even more have done so since, they usually do so in the manner of a “scientific” or “psychological” test. While interesting, I feel this framing doesn’t get the best results. It’s how we first started testing originally as well. And what we realized is that people think there is a “right” answer to a scientific or psychological test. This made them less likely to give their honest assessment of things out of fear that it wasn’t the “correct” one.
When we switched to a focus-group testing style and began soliciting people’s opinions they opened up 500% with us. There was no longer any concern with being right, or doing things “properly.”
We also learned that people were naturally inclined to be nice to a performer. People who would tell us they were very fooled and had no idea how a trick was done, would then give us a detailed and often accurate guess of the method when we offered them $5 to do so. We’ve since learned ways to prompt people for their full feedback without the need to offer them an additional incentive. But there was definitely a learning curve to pull this information out of people.
I enjoy the testing and think it’s beneficial, and there is more testing results coming out in the next book. However it has the downside of needing multiple people to organize it and pull it off. And most of the people I used to do the testing with aren’t in NYC full time anymore, including myself. So it’s something we have to plan long in advance. Plus it’s very expensive. The year before the pandemic we spent $14,000 just on paying focus-group participants alone. And when the pandemic came it was essentially impossible to do large-scale testing.
So last year I was sitting around and thinking of what else, besides the focus-group testing, I was in a unique position to do. And then it hit me:
Iterating
In other words, testing different versions of something over and over..
Most amateurs might be lucky to perform once or twice a week. And they’re not necessarily looking to “test” new ideas. They, understandably, want to make the most of their limited performance opportunities.
On the other hand, professionals may be willing to hone an effect until it’s good, but they’re unlikely to fuck around with it too much after that point. What’s the point? When you have something that works, there’s little incentive to take steps back and try different paths that may not work, in hopes of stumbling over something potentially stronger. (And, honestly, most “professionals” working restaurants or bar mitzvahs aren’t really innovating with magic. They’re doing other people’s tricks with other people’s patter and telling other people’s jokes. There are, of course, incredible exceptions to this rule. But if you open your yellow pages to “magician.” 95+% of the names you see there aren’t doing anything interesting. (You still have the yellow pages?))
As the world’s only professional amateur magician, I have the time, the funding, and the inclination to test ideas out over and over. Performing multiple times a day (on average) in casual settings, I have the opportunity to give a shot to ideas that are unlikely to work. And I’m perfectly happy to abandon ideas that are already working just for the sake of testing new variables.
I call this “iteration testing.” Taking a technique or a concept and trying out as many possible variations of the technique as possible.
On November 28th of last year I started my first series of Iteration Testing devoted to peek wallets. 500 days in a row where I would test minor variations on how to peek, when to peek, what to peek, what wallet to use, what type of information to ask for, etc. etc.
I’m also getting feedback in multiple ways. For some I’m just collecting it in real time as the trick progresses. For some I’m talking with the person immediately afterwards and questioning them about their experience. And for others I’m contacting them a week later to get their thoughts.
I am now 358 days in (I took a week off when I had Covid). I will be doing a full write-up on my takeaways from this testing in the next book (given that the supporters are the ones affording me the time do this). But tomorrow I’ll post about what I’ve learned so far about the biggest concern people have with peek wallets (how to handle the seemingly illogical nature of asking for someone to write down a word and putting it back in your wallet). It builds off something I wrote in a post a few years ago. But now I have some concrete data to back-up some of what I thought at the time (and refute some of it as well).
I now have three different Iteration Tests running. The other two are likely to be shorter. About 200 days each I think. More details on those to come. If you think of a subject that could be interesting to look at through the lens of Iteration Testing, let me know.
