When Results from Survey Instruments do not Make Sense: Reflections on Using Mixed Methods in Remote Communities with Indigenous, Marginalized Populations

Michael G Young*

School of Peace and Conflict Management, Royal Roads University Victoria, British Columbia, Canada

*Corresponding Author:
Young MG
Associate Professor, School of Peace and Conflict Management, Royal Roads University Victoria, British Columbia, Canada
Tel: 250-391-2600
Fax: 250-391-2619
E-mail: [email protected]

Received date: July 7, 2018; Accepted date: July 30, 2018; Published date: August 3, 2018

Citation: Young MG (2018) When Results from Survey Instruments do not Make Sense: Reflections on Using Mixed Methods in Remote Communities with Indigenous, Marginalized Populations. J Addict Behav Ther Vol 2 No 2:10.

Copyright: © 2018 Young MG. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Visit for more related articles at Journal of Addictive Behaviors and Therapy


Between 2010 and 2016 I led two separate but related research projects that examined the contributing factors surrounding the rise of homelessness and comorbid disorders in the town of Inuvik, Northwest Territories in the Western Canadian Arctic. Gaps in services and the potential contribution of outlying communities as a source of homeless persons with comorbid disorders to Inuvik were also examined. We utilized the concept of Hard-to-House to capture the persons experiencing difficulties with housing and comorbid disorders, which include substance abuse and mental health challenges [1]. Several publications on the results of the projects were produced which highlight the overall objectives of the research, methods used, results and recommendations [1-6]. This paper explores a lesser articulated problem related to the results of the survey instruments used in both research projects, which included the QLHHI (Quality of Life for Hard to House Individuals), the RCQ (Readiness to Change Questionnaire), and the SF-12 Health Survey. Essentially, the instruments did not accurately capture the information for which they were designed.

A brief summary of the projects, methods, instruments used, outcomes and assessment of the results are presented below. Included in the Appendix are selected outputs for both projects which illustrate the problems associated with the implementation of the QLHHI, SF-12 and RCQ. As will be apparent, mixed methods were used in both projects, and the benefits of this research design will be incorporated into the discussion. However, the questions guiding the research, while important for the projects overall, are not the focus of this paper and will not be included here.

Project I: 2011-2012

Using a mixed method design, this project examined the breadth and scope of homelessness and comorbid disorders, and identified key issues related to service provision to affected populations in Inuvik [5,6]. Based on its previous application in Canadian contexts, the QLHHI [7] was used to measure the functional health and well-being of 17 Indigenous and non- Indigenous ethnicity men and women. The QLHHI contains three domains developed using multiple discrepancy theory relating to housing, health and social support. Yet, in all three domains the QLHHI revealed little in terms of significant differences between participants and what would be expected from none- HtH populations with comorbid disorders [3,5]. Selected T-Test results from the data analysis can be found in the Appendix.

However, the qualitative results gleaned from interviews and focus groups with HtH persons, service providers and other stakeholders in the community were instructive as to the problems experienced by participants, their life situations, and the gaps in services available to them. The data identified a lack of affordable, available and appropriate housing, restrictive rental polices and lack of adequate and culturally appropriate services as key issues related to the experiences of HtH persons (Table 1). Whether or not the result was directly linked to the research, shortly after completion of the research project, stakeholders in the community worked to open an Emergency Warming Centre (EWC) for HtH persons. The EWC operated over the winters of 2013-14 and 2014-15 in the local Anglican Church. The shelter was open during the evenings. HtH users of the centre were provided meals and a place to sleep, even if they were intoxicated or under the influence of illicit substances, providing they were not violent or overly disruptive. Sleeping arrangements were not ideal with men and women sharing an open space on the floor, but the centre helped keep HtH persons safe [3,8].

Variable pair Sample size Mean SD
Housing 1 15 4.0667 1.90738
Housing 10 15 5.6667 1.1127
Degrees of freedom 14  -  -
One-tailed distribution  -  -  -
p-level 0.001  -  -
Health 1 15 4.94118 4.6838
Health 10 15 5.11765 3.48528
Degrees of freedom 14  -  -
One-tailed distribution  -  -  -
p-level 0.346    
Social support 1 15 5.17647 3.5294
Social support 10 15 5.6667 3.98529
Degrees of freedom 14  -  -
One-tailed distribution  -  -  -
p-level 0.451  -  -

Table 1: T-Test output of selected QLHHI variable pairs*.

Project II: 2013-2015

Conducted in 2014-15, the second research project examined the effectiveness of the EWC at reducing morbidity and mortality rates among HtH persons in Inuvik. As with the first project, mixed methods informed the research design which examined the health and social functioning of participants. A pre and post design was employed to examine health and social functioning at the beginning and end of winter using the SF-12 short form health survey [9] and the Readiness to Change Questionnaire (RCQ) [10]. A total of nine HtH Indigenous and non-Indigenous men and women participated in the both pre and post-test as showed in Table 2a and 2b.

Variable pairs Sample size Time 1 Time 2
1 9 2.5556 2.4444
2 9 2.3333 2.2222
3 9 2.1111 2.0000
4 9 0.7778 0.6667
5 9 0.6667 0.4444
6 9 0.7778 0.8889
7 9 0.6250 0.6250
8 9 3.5556 3.5556
9 9 3.0000 2.7778
10 9 4.4444 3.7778
11 9 4.2222 3.6667
12 9 3.8889 4.0000

Table 2a: Pre and post-test output from SF-12*.

RCQ domain**
Time 1 Time 2 Number of participants
Pre-contemplative/contemplative Action 1
Pre-contemplative/action Action 2
Action Action 1
Contemplative Action 2
Action Pre-contemplative 1
*Cases with missing data excluded.
**See Rollnick, et al. for a description of RCQ domains [10].

Table 2b: Pre and post-test output from RCQ*.

At the same times that the surveys were administered, researchers conducted face-to-face interviews with participants that explored health and well-being, lifestyle, social supports and the effect of the EWC on participants’ lives. As well, interviews with service providers in the EWC were conducted to capture their views on HtH persons using the centre and to explore the functioning of the EWC [3].

In the main, results from the SF-12 noted in the Appendix indicated a negative trend in 10 out of the 12 items with participants’ levels of social functioning declining over the study period. However, none of the T-test results of variable pairs were significant. As well, the results for the RCQ in the Appendix indicate that participants had changed their intentions and were anticipating reduction in alcohol consumption. Yet, in a few cases participants indicated no change or an increase in substance use over the study period.

Reflections on the Research

Notwithstanding the apparent, albeit partial contribution of the RCQ results, the instruments used to determine health and social functioning of participants in both projects did not perform adequately. Indeed it could be argued that the QLHHI and SF-12 did not contribute to knowledge regarding the lives of HtH persons in Inuvik. Fortunately, the addition of face-to-face interviews and focus group sessions with participants, community service workers and other stakeholders in the community provided a significant amount of data on gaps in services and the lived experiences of HtH persons. From this data a picture emerges that can explain why the quantitative surveys yielded such poor results.

Although Hubley et al. note that the QLHHI has been used in research on rural populations it does not appear to have been used with marginalized and/or Indigenous populations [7]. In one respect, the QLHHI assumes that participants are aware of their disadvantaged situation and can use that situation as a point of reference when completing the survey. Despite the dire situations faced by participants, during interviews and focus groups, most HtH persons did not identify themselves a permanently disadvantaged or in need of extensive services, although the need for services was noted by many. This was true in cases where the need for medical attention, housing, counseling and other social services was quite obvious [5].

The results from the SF-12, and to a lesser degree the RCQ, appear to suffer from the same contextual issues faced by the QLHHI. However, in addition to the observation that participants did not view themselves as that different to other people in Inuvik, they were faced by a prospect that would impact their living conditions the following winter. As the second season of operation of the EWC came to an end, the Anglican Church announced that it would not continue to operate the EWC [12]. Although a small sample completed both pre and post-tests in the second project, all mentioned concern about where they would be living the following winter [2,3]. Undoubtedly, the anxiety related to facing another winter without a safe place to stay in a harsh northern environment coloured participants’ perceptions of their health, well-being and capacity to function. No wonder then that their measures of health, social functioning and substance use did not fit expected outcomes to any degree even if participants did not view their personal status as disadvantaged or that they were in need of assistance.


Given the survey outcomes of the two research projects it is important that research involving Indigenous and marginalized populations in rural contexts consider extraneous variables that are not captured in survey design and application. Invariably, small sample sizes can influence research results, and both project 1 and 2 had limited final sample sizes of 14 and 9 respectively. Yet, the mixed methods design used in both projects clearly demonstrate the contribution of participants’ personal experiences. The qualitative results underscore the difficulties experienced by HtH persons regarding the availability of appropriate and adequate services, housing and addictions. Research informing the development of policies and practices for HtH persons must accurately reflect the contextual and cultural environments in which they live. Without further development, survey instruments are a poor choice for research involving HtH persons in remote northern communities, and qualitative methods remain the best option if research is to effect developments in policy and practice for HtH persons with concurrent disorders.


Select your language of interest to view the total content in your interested language

Viewing options

Post your comment

Share This Article

Flyer image

Post your comment

captcha   Reload  Can't read the image? click here to refresh