Closer look: Coin Soccer

This post explores a project by the well known Mixed Methods Research pioneer, Dr John W. Creswell. This video is really interesting as it serves as an example of the end to end of a research plan. The process is deconstructed below to observe the component parts to more deeply understand this particular research method. Further suggestions are also made for additional ways to add value to this research project design. 

The Study

In the video linked above, Creswell discusses a mixed methods research design with a goal to “develop a survey that can be given to hundreds of people”. We do not learn of the actual research question throughout the video, but this is an exploratory sequential design mixed methods research process. “The exploratory sequential mixed methods design is characterized by an initial qualitative phase of data collection and analysis, followed by a phase of quantitative data collection and analysis, with a final phase of integration or linking of data from the two separate strands of data.” (Berman 2017, p1)

This study begins with participants playing a game called ‘Coin Soccer’ on an iPad. 

Steps in the process

1. Qualitative first part of the study, where participants record their responses/experiences
2. Qualitative second step where the participants are brought together for a focus group
3. Analysis of responses 
4. In a break in the focus group participants are then given a Quantitative survey from the analysis of the first Qualitative responses
5. Qualitative focus group continues
6. Analysis
7. Outcome – Develops a final Quantitative survey

The outcome of this study and journal article are not provided, however, based on the video above and other texts, we can expand on elements explored in the process to more fully understand an example of what might make up a mixed methods research process.

1. Qualitative first part of the study, where participants record their responses/experiences

Before this step is considered, this study should also contain a formal analysis of the game, a method discussed by Petri Lankoski and Staffan Bjork, in ‘Game Research Methods’. This method is used frequently in visual fields and in this context would be used to describe the “systemic features of the game such as game elements, rules and goals” (Lankoski & Bjork, 2015 p24). We do have a basic understanding that the goal of the game is to flick the soccer ball through the goal posts, but we don’t understand any of the illegal moves referred to later on in the video. The visual we are shown of the game ‘Coin Soccer’ demonstrates an underdeveloped interface and user experience design. This would impact the usability and engagement of participants to the game. It also appears very basic for the participants and it would be interesting to understand if this was a deliberate choice, why this game was chosen or if it was perhaps developed specifically for the experiment. Other important issues would include the demographics of the participants, do they have experience playing games? Or do they self identify as ‘gamers’? Or are they novices at playing games of any sort?

Playing this game could be seen as a chore, however, writing in the journal afterward may well have increased the level of involvement and created an artificial level of engagement in the game. The impact the researcher and this task will have on the process is unacknowledged in the brief video as is the epistemology, however, the language he uses throughout the video is “cooperative, interactive, and humanistic”, qualities Lather describes as consistent with an interpretivist approach. (Lather, P. 2008 p38).  Interpretivist would be suitable for the qualitative part of this mixed methods research. In this particular and brief discussion of the study we are at a disadvantage to know further details, and it is at times unclear the motivation and reasoning for different steps. The question of where the researcher positions himself is highly relevant here, is the goal simply improving communication feedback loops for the developers of this one particular game? Or is it about improving the process of user research as a whole for use in other contexts?

Creswell discusses in his book that a best practice mixed method question should include the “qualitative question, the quantitative question or hypothesis, and a mixed methods question” (Creswell, 2018, p234). For this Coin Soccer study, the intention stated in the video “to develop a survey that can be given to hundreds of people” doesn’t meet this requirement. We need to understand how “This mixed methods question represents what the researcher needs to know about the integration or combination of the quantitative and qualitative data.” (Creswell, 2018, p234)

The full and proper question would likely include something like ‘Does using a mixed-methods approach impact the development of a quantitative survey questionnaire in understanding the user experience of games’. It would be great to have some more insight here into the tool researchers are trying to develop as well.

Many questions remain about other methods that could be used included in a study like this. Quantitative data such as heatmaps that will see high touch areas of the iPad interface, or how frequently participants are using the app, user statistics such as time of day they log in, duration and frequency that subjects play for would all be interesting to investigate with reference to the qualitative data from the journal and the focus group. These types of stats are available from all iPad apps and could have contributed a valuable step particularly at the beginning. Questions it might answer are ‘on days where subjects played for a shorter time do they report higher experiences of dissatisfaction’. Other statistics such as how often they won, and how often the computer won would have also been interesting to reflect on in terms of usage and engagement. It could provide an answer to ‘How long does a player need to spend playing this game to see an improvement in winning the game’? And qualitative questions like ‘Are they playing to win or for enjoyment?’

2. Qualitative second step where participants are brought together for a focus group

The next step is a focus group led and facilitated by Creswell, who if using an interpretivist or critical theory paradigm would lead and tailor questions. Here, Creswell says in the video he learns more about things that didn’t come up in the journal such as how participants were feeling, and how often they played. He would likely ask open ended questions in the first half of the focus group to highlight anything he may use moving forward to the next phase.

While this qualitative focus group is going on, Creswell will also collect quantitative data.

3. Analysis of responses 

The journals from the 8 participants in this survey were all digitally recorded in word documents, and then sent to Creswell at the end of the first phase of Qualitative research. These appeared to be quite basic comments such as being frustrated that they could not move the coin as they had wanted to in the game. The data from the journals is broken down into themes, to help identify commonalities and patterns in what users have reported. 

Collection of qualitative and quantitative data simultaneously is described as “Convergent mixed methods” (P63, Cresswell, 2017) where the researcher will integrate the interpretation of the two types of data. The next phases of design will explore areas that this analysis has identified that may need further explanation or study.

The video describes this step as being taken after part of the qualitative phase of the focus group takes place as the narrator tells us that Creswell ‘quickly’ puts together a quantitative survey. However, undoubtedly the review and analysis of the journals should have taken time and would have benefitted, in practice, from being completed well before the next qualitative step. Creswell also mentions that he would use software usually for analysis in the qualitative step.

4. In a break in the focus group participants are then given a quantitative survey from the analysis of the first qualitative responses

While there is not much specific information about the outcomes of this survey, it asks questions about broad topics such as ‘The name of the game needs to match it’s objectives’, and ‘I like learning how to score points’. This survey seems like it could have been better prepared with questions grouped into areas and then refined to get greater insights. The wording of the questions seems clumsy and not refined. At this point this would be good to reflect again on the purpose of the study and the research questions and evaluate if all the questions have been answered through the survey and the focus group, and add any questions still needing an answer to the last part of the focus group.

5. Qualitative focus group continues

Creswell then continues the focus group and is able to deepen his understanding of their responses and get quick feedback about the survey that he has just used. At this point Creswell gains some great insights into the need to diversify the survey in terms of demographics, he also discovers that people aren’t playing the game everyday. User stats could help here to verify and validate the qualitative usage data. 

6. Analysis

While the video above doesn’t cover the analysis in detail, it is easy to see from this research design and practice example that the qualitative data could be validated by the quantitative data and vice versa. This would mean that the initial research question would be considered with the results, patterns would emerge and conclusions would be able to be drawn. As stated before with convergent mixed methods study design, researchers are able to review the progress throughout the study and identify if the next stage of the research needs to be adjusted in order to respond to the purpose of the research. Creswell does this with his quantitative survey that is followed by the second half of a qualitative focus group.

This example of investigating user feedback is an important example for all digital or interactive products, and demonstrates very similar processes currently used in digital and interactive media. Working as a designer it is crucial to understand the way your audience interprets your design, or uses the interactive product. Digital designers need to review user experience data, including details about how someone moves through a website, who accesses the site or interactive, what location they are they from, and what is the reach (ie, how many people saw the digital product). This type of data is used regularly to be able to communicate concrete outcomes to clients and to make improvements to their work in an agile project management methodology. This is a process used in digital media of continuous improvement via multiple iterations.

7. Outcome – Develops a final Quantitative survey

This is the final stage of the research study, and combines all the earlier analysis and data that has been collected. There is no clear information about what the final quantitative survey Creswell develops looks like and what areas were most beneficial, ie: was the qualitative or quantitative most instructive for this project? This study example has perhaps a larger amount of qualitative data used throughout it, but it would have been really great to include more quantitative data and statistics as mentioned.

Conclusion

In the case of ‘Coin Soccer’, the relationship of the qualitative and quantitative research methods Creswell demonstrates in practice could assist in the validation of the research as a whole. The research tools are designed on an ongoing basis (could perhaps even be called agile) to validate the earlier research. Although this example does not go into detail about the outcome of the study, the steps he took to get there, would quite clearly provide good information for how to improve the game and the user experience. His use of open ended questions would also identify some previously unconsidered ideas about individual participants experiences. It appears it may have also shed light on some universal reasons why people play games and could help to inform the researchers of aspects of participants experiences that they had not considered, or had not been previously considered before in research. The qualitative aspects inform the quantitative which then informs the qualitative which then informs the outcome of the study in the form of a quantitative survey that may be used more widely. It could be described as a loop of the subject – object relationship.

This process gave the researchers a more full picture of the experience of playing the game and included both a macro and a micro view.

The study design for this game example is sound for identifying complex ways that digital interfaces and games operate and interact with each user.