Week 7 Exploring qualitative analysis

This week we have been carrying out a qualitative analysis of a subset of the data resulting from the School Leavers Study, 1978.

The full dataset consists of 141 essays, too many to analyse in a single week, especially on a part-time basis.  I chose to analyse the first twenty boy’s essays and the first 20 written by girls.  I made an assumption that the data is in no particular order and that selecting the first twenty would be no more or less random than any other means of selection.  Forty essays turned out to be far too ambitious as I have documented below.

There are no references at the bottom of this blog post because I’ve deliberately avoided looking at anyone else’s secondary analysis of the data so that I can determine for myself what it might be possible to infer from the data.

The process

I used the online application Dedoose to code the data.  Importing the essays and the table that provides demographic information about the participants was straightforward, as was linking the demographic information*, to the relevant essay (*described as descriptors in the software).

This has been a hugely informative exercise.  As someone who prefers to learn by doing (and by learning from mistakes along the way) simply ‘having a go’ at coding made me realise how powerful it can be when applied appropriately.

The first insight for me was that creating new codes as I progressed through the data resulted in frequent returns to all of the previous data to find out if there was evidence of the code I had just applied.  While this was frustrating at times it was interesting to observe my own thought processes emerging as I wrestled with how I might mine the most useful information from of the data.  For this reason, I would probably work through this iterative process with any future coding exercise where I wasn’t familiar with the data, although giving some thought to codes that could be created apriori would save a lot of time.

The second insight into the coding process came when I realised I had made some of my codes too high level.  For example, I started off coding mentions of having children, but later felt it might be useful to know when participants thought they would start a family and how many children each respondent was predicting they would have (or whether they mentioned children at all).  That depth of empirical data would be useful in any comparative study, both with other groups and between the male and female participants in this study.  This meant that some codes required a degree of mental arithmetic before they could be applied, such as working out what age the participant predicted they would be when they left home or got married.

I found that it’s easy to get caught up in the longer more descriptive essays and I was concerned that they might become more dominant in the output, so I tried to focus on the basic information that is present in both the more mundane essays and the sensational accounts.

It wasn’t until I had figured out that I needed to add a whole range of child codes into the high-level categories that I got a realistic feel for the amount of work required to do justice to the wealth of information in the data.  Forty essays started to feel like I’d taken on far more than I could cope with in the time available and I could see why we’d been instructed to read 6 to 10!

Emergent themes

Most of the essays I coded revealed relatively low-level aspirations, with many participants predicting that they would find it hard to get a job and would experience periods of unemployment and other adversity.  However, in the subset of essays I analysed, there were also references to overcoming adversity through hard work, this was particularly evident amongst the boys.  Most of the boys predicted starting jobs in engineering or low-level clerical roles and most of the girls expected to work in secretarial, clerical or retail.  One of the boys essays was particularly telling; it was evident that he had aspirations of becoming a racing driver, but he expected his dreams to be thwarted by his economic background.  Both the girls and boys essays referred to wives having to give up work to look after children and returning to work as soon as the children were of an age were that was possible.

All but two of the boys referred to having their own accommodation and many of these referred to owning their own house, despite their relatively low-level career aspirations.

Most of the boys also expected to get married (more than once in some cases) and of the essays I coded all of the boys who mentioned marriage also expected to father children.  I started to code how many children participants predicted but I haven’t had time to finish that analysis, however, there appears to be a surprisingly frequent mention of twins amongst the girl’s essays and a few mentions by the boys.  This looks like an example of memory being at odds with the actual data and the subset not reflecting the whole dataset. A quick search for the word ‘twins’ across all of the essays reveals only thirteen mentions in total, clearly I happened upon more than would be expected looking at all of the data.

Where the boys reflected on their lives many saw themselves as having failed and very few seemed to have a positive view (or any view) of what retirement might hold, with one going as far as expecting to end it all when they reach sixty.  In general the girls seemed to be more content with how their lives had panned out, with many taking satisfaction in watching their children grow up and life events such as their own and siblings weddings.

What was noticeable by its absence was mention of possessions.  Other than owning a house, a car, a small garden, or going on foreign holidays, few mentioned any of the luxuries youngsters might expect to have in their lives today.  In the essays I analysed the most ambitious had hopes of owning a small yacht and one expected to be able to rent out a house he had bought as well as buying a flat.

What questions could be answered using this data?

I can see that there would be risks in inferring too much from a secondary analysis of this data.  There’s a lot that we don’t know about the participants and the nature of the exercise means that there are gaps in many of the accounts.  For example, it would be risky to infer that a participant who doesn’t mention marriage or children had no expectations of either.

What the data does seem to be useful for is analysing differences in expectations between the boys and girls and what appears to be important to each group. It would also be possible to draw some conclusions about how boys saw girls lives unfolding and vice versa and as such their attitudes toward the opposite sex.

At the time the data could have been used as evidence to support the need for interventions such as career advice, or the provision of training to enable school leavers to pursue careers in areas other than those that would have been in recession in the late 70s.

If I was conducting a project using this data, what would I want to do next?

There are a few follow-on projects that would be fascinating to pursue.  A longitudinal study of students in the same area would enable researchers to show how attitudes and aspirations have changed over time.

Comparative studies with similar datasets (if they exist) might demonstrate differences and similarities with other areas that had different social and economic conditions.

And it would, of course, be interesting to revisit this particular group of school leavers and compare their actual life-course to their imagined version.

Advertisements

Week 6 – “lying with statistics”

It appears that the graph below, and similar versions of it, are often used as examples of lying with statistics, so, whilst it isn’t an original ‘find’, I’ve found it useful to try to understand why it is problematic.  You’ll find my analysis below.

This version of the chart was published by the Economic Policy institute in the USA and used in a report by Lawrence Mishel titled “The wedges between productivity and median compensation growth”

Growth of real hourly compensation for production/nonsupervisory workers and productivity, 1948–2011

Note: Hourly compensation is of production/nonsupervisory workers in the private sector and productivity is for the total economy.

This is the commentary that relates to the chart, taken from the original article:

“The hourly compensation of a typical worker grew in tandem with productivity from 1948–1973. That can be seen in [the chart above], which presents both the cumulative growth in productivity per hour worked of the total economy (inclusive of the private sector, government, and nonprofit sector) since 1948 and the cumulative growth in inflation-adjusted hourly compensation for private-sector production/nonsupervisory workers (a group comprising over 80 percent of payroll employment). After 1973, productivity grew strongly, especially after 1995, while the typical worker’s compensation was relatively stagnant. This divergence of pay and productivity has meant that many workers were not benefitting from productivity growth—the economy could afford higher pay but it was not providing it.”

There are a number of issues with the way this data has been presented (both in the chart and in the text) in order to give the impression that wages haven’t kept pace with productivity or, to phrase that another way, “we’re all having to work harder but we don’t get paid any more for doing so”:

  1. The two lines on the chart are based on two different populations – pay is based on a specific group of workers, those in “production / non-supervisory” roles, while productivity is based on the economy as a whole.  It would be interesting to see the same graph based on the same population, I suspect the lines on the chart would be less divergent.
  2. We don’t know if the method of calculation of either of the two figures has changed at any point since 1940.  I’m guessing the use of a cumulative figure is designed to factor any such changes out.
  3. Comparing an average (productivity) with a median (hourly compensation) is almost bound to show a disparity.  For an average ‘outliers’ in the population (those with particularly high or low values of the data being measured) will have the effect of skewing the number up or down. So in this case, a sector of workers who had greatly increased (or decreased) their productivity during the time frame in question could skew the productivity figure up or down.  In the USA one thinks of the decline in the automotive industry and the huge rise in tech companies, the latter of which is likely to have dragged the productivity figure up considerably.
  4. The author uses the phrase “the typical worker’s compensation” in their commentary.  While the author might have intended this to mean a typical worker within the selected population, it gives the impression that the productivity/reward gap applies to most of us.

How could these issues be resolved?

The obvious solution would appear to be using the same population for both lines on the graph.

A median figure based on hourly compensation for all sectors of the economy would be impacted by both the large number of workers with low take-home pay at the bottom of the economy and the multi-million dollar earners at the top.  I’ve no idea which way it would move, but at least the graph would be ‘comparing like with like’.

Alternatively, a productivity figure for “production / non-supervisory” roles could have been used, assuming the relevant data was available or could be extrapolated from other data.

Week 5 – Reading research findings

There are the questions we’ve been asked to reflect on for this week’s topic in SOCRMx, you’ll find my responses below:

  1. Why do you think Paddock chose narratives as a way of conveying the main themes in her research?
  2. What is the impact for you of the way the interview talk is presented? What is the point of the researcher noting points of laughter, for example? What about filler sounds like ‘erm’?
  3. How does Paddock go about building a case for the interpretations she is making? How does she compel you, as a reader, to take her findings seriously? Share a specific example of how you think this is done in this article.
  4. Interviewees use many emotive words in the excerpts presented here, but Paddock has focused in on the use of the word ‘disgusting’, and developed this through her analysis. How does this concept help her link the data with her theoretical perspective?
  5. Paddock’s main argument is that food is an expression of social class. Looking just at the interview excerpts presented here, what other ideas or research questions do you think a researcher could explore?

1. Why do you think Paddock chose narratives as a way of conveying the main themes in her research?

Narrative research can be categorised as descriptive or explanatory (Polkinghorne, 1988).   Paddock appears to be using narrative to explain how/why differing views of  ‘alternative’ foods are attibutable to social class.

Churchill and Churchill (1982) (as cited Sandelowskil, 1991) defines narrative in this way:
Generally, narratives are understood as stories that include a temporal ordering
of events and an effort to make something out of those events: to render, or to
signify, the experiences of persons-in-flux in a personally and culturally
coherent, plausible manner. Narration is a threshold activity in that it captures
a narrator’s interpretation of a link among elements of the past, present and
future at a liminal place and fleeting moment in time.

So, rather than a series of isolated ‘facts’ as might be gathered by a survey conducted in isolation of any other methods, narrative accounts can provide the context in which ‘facts’ sit and show how they relate to one another.

Sandelowskil (1991) writes “lives are understood as and shaped by narratives, narrative approaches to inquiry parallel the ways individuals inquire about experience (Cohler, 1982) and, in a sense, naturalise (or remove some of the artifice from) the research process”.  My understanding of this is that narrative tends to be authentic and less open to reinterpretation by the researcher.

2. What is the impact for you of the way the interview talk is presented? What is the point of the researcher noting points of laughter, for example? What about filler sounds like ‘erm’?

The researcher went to some lengths to present the discourse as it was spoken.  Short of inserting audio/video clips of the actual conversation (which would have raised ethical issues in terms of participants remaining anonymous) I’d imagine that making clear such nuances as inflection and emotion in the participant’s delivery and non-verbal cues would not have been an easy task.

To understand the difficulties, one only has to think of the misunderstandings that can occur as a result of a misinterpreted text message or email, and the extensive use that is made of emoticons in attempt to make such subtleties as humor and sarcasm clearer.   Once the interviews had been transcribed subtleties like these could have been lost without the inclusion of annotations to indicate laughter etc. and coding the output could have been compromised

The addition of filler sounds such as ‘erm’ could indicate uncertainty, that the participant needed to think before answering, or, if attached as a prefix or suffix to another word or phrase they could even indicate a different meaning.

4. How does Paddock go about building a case for the interpretations she is making? How does she compel you, as a reader, to take her findings seriously? Share a specific example of how you think this is done in this article.

The first device I noticed is that Paddock includes an explanatory paragraph or two before each section of the transcript so, as readers, we are already primed for how she believes we should interpret the participant’s response.  In this respect, the transcript is positioned as confirmation of the points being made in the explanation.

Paddock then introduces relevant literature in the analysis after a section of the transcript to back up and reinforce her findings. For example, there are four references in the first section of Valerie’s interview transcript alone.

Short subheadings, such as “A ‘Hugely Different Attitude to Life’” also help to build the case by summarising the findings before we get to them.  This is particularly effective where the subheadings lift phrases directly from the interviews.

Over the course of the transcripts, Paddock presents the participants in a way the helps us understand them as characters, so that when we read her conclusions and she refers back to them e.g. “For market customers such as Sophie and Charlie, the market provides a space for sociality” we recognise them and we are, perhaps, more inclined to agree with her summary.

I’m conscious that the above may sound as though I believe these literary devices represent artifice on the part of the researcher.  That is not my intention, I see these as legitimate ways for the researcher to convey the meaning she has constructed from her findings.

4. Interviewees use many emotive words in the excerpts presented here, but Paddock has focused in on the use of the word ‘disgusting’ and developed this through her analysis. How does this concept help her link the data with her theoretical perspective?

Firstly Paddock relates her analysis to that of Lawler, S (2005) “Disgusted subjects: The making of middle-class identities” throughout the paper.  Homing in on this word helps to contrast the views of the two class groups.  For me, it almost set a benchmark level for the comparison, for example, would “What annoys me about you” be directly comparable to “what I find disgusting about you”?

5. Paddock’s main argument is that food is an expression of social class. Looking just at the interview excerpts presented here, what other ideas or research questions do you think a researcher could explore?

It’s difficult for me to separate my own beliefs here but I do think that ‘organic’ food has been hijacked by marketeers in the UK and, for me, it’s associated with high prices and ‘up-market’ shopping. It would be interesting to carry out the same research and compare results to a provincial area of France or Spain, where locally produced produce seems to be a routine purchase for many and where the question of organic versus factory farmed seems to be (in my experience at least) less in evidence.

Reference

Polkinghorne, D. (1988) Narrative Knowing and the Human Sciences, SUNY Press

Sandelowski, M. (1991), Telling Stories: Narrative Approaches in Qualitative Research. The Journal of Nursing Scholarship, 23: 161–166.

 

Focus groups

The second research method I’ve chosen to study is Focus Groups (the first being Surveys).

The questions students reviewing this method have been asked to reflect on in this blog post are:

1. How have focus groups been integrated with other methods in this research?

The paper indicates that the research finding are based on semi-structured interviews and focus groups carried out by students completing Anthropology and Sociology courses, underpinned by observations (ethnography) carried out by the two course leaders.

2. What difficulties did students encounter in the design of focus group questions?

The paper states that “In general, students’ draft lists [of questions] were far too lengthy for the allocated time of 1 hr.” and that “Some students included too many ‘ice-breaker’ questions”.  The course leaders also found that students had “high expectations of how many topics and issues could be adequately covered in a single hour.”

3. What kind of insights did the focus group generate, and how were these different from those derived from other methods used?

This doesn’t appear to be covered in any great depth in the paper, although the following extract does provide a brief comparison of the findings of the two methods “Our students’ focus group research found that there were many reasons why […] engagements [between study abroad and domestic students] did not occur, beyond an individual lack of interest (as might be revealed in an interview exchange on the issue). […]  For instance, […] they also concluded that […] unlike international students, domestic students did not tend to be interested in making new friends in class, as they already had strong friendships with other students.”

4. What are identified as some of the key findings from focus group questions?

Meta-analysis of the student researchers findings by the two course leaders and the improved research capability of students in their second semester revealed the following insights:

  • “Loneliness and confusion regarding cultural and social norms”
  • “Contradictions between the rhetoric and realities of international and study abroad student programs”

  • “A “bubble” effect, whereby international and study abroad students had limited opportunities to engage with their domestic counterparts

5 . What kind of issues did the students face with recruiting focus group participants?

The paper highlights several issues faced by the students when recruiting focus group participants:

  • “The main issue […] faced with recruitment was students’ widespread assumption that it would be easy”
  • “students often knew very few, if any, international or study abroad students, and when this was the case, they found it extremely difficult to recruit effectively [by posting a message on Facebook]

  • “domestic students rarely engage with international or study abroad students, as they generally already have well developed social networks of their own” original paper cites (Bringle & Hatcher, 1999; Soria & Troisi, 2014).

  • “most [students] had to figure out recruitment on their own.” […] “bulk emails announcing a “call for participants.” […] were only partially successful […] and were less effective when it came to recruiting for focus groups

My initial reflections on this exercise:

The business I work for uses focus groups extensively to understand its customer base better and, in particular their thoughts and feelings about our products and services. I’m now aware that the way focus groups are used commercially differs from the way they’re used in educational research.  What I’ve seen of them in the past had made me a little wary of them because of factors such as dominant participants and the consequent pressure on other members of the group.  However, I’m now starting to see that observing shifting perspectives resulting from group discussion can be as beneficial, if not more beneficial, than individual views in gathering insights and that this sort of interaction can move thinking on to reveal new theories.  Indeed, as Barbour, R. (2017) suggests, these “slippery views” are actually the “stuff of focus groups”.

One theme that comes through strongly for me and that builds on my initial thoughts about surveys, is that novice researchers can easily underestimate the complexity and difficulty of a research method that they haven’t employed before.  With regard to focus groups it’s evident that this can lead to:

  • underestimation of the time required to complete the activity from end to end
  • greater difficulty in recruiting participants than the novice researcher might have expected
  • poorly worded questions
  • too much time time spent on ice-breaker type questions versus those exploring the actual topic
  • unrealistic expectations of the number of topics that can be explored in a single sesssion

References:

Barbour, R. 2017,  An introduction to focus groups, SAGE Publications Ltd., London

McKenzie, L & Baldassar, L 2017. ‘Studying internationalization on campus: lessons from an undergraduate qualitative research project’ [online]. SAGE Research Methods Cases.

 

 

 

Activity 2 Design a simple survey

Prior to completing this task students of SOCRx course, who chose surveys as research method, completed an exercise that demonstrated potential difficulties with writing research questions.  I wrote what I thought was a fairly comprehensive analysis of how the set of ‘bad’ questions provided could be improved, however, one of my fellow students was able to add at least a couple more ways each question could be improved.  This made me realise that writing good questions is even more difficult than one might first imagine.  It also highlighted to me the need to take a massive step back from the task of writing questions, once a first draft has been created, to give some really deep thought to what might be ‘bad’ about them and how they might be improved.

The next task for this week is to design a simple survey to collect information from a target group of my choosing about their use of Web 2.0 technologies.

Web 2.0 technologies are those characterised by user created content, such as social networking/media sites, video and photo sharing, blogs and vlogs, ‘collaborative consumption’ sites such as eBay, Wikis and social tagging (also known as ‘folksonomy’).

I’ve chosen clinicians in the company I work for and the clinician blog they’re provided with as a representative Web 2.0 application.  These are my questions:

  1. Have you accessed the clinician blog within the past (tick all that apply)
    [ ] Week?
    [ ] Month?
    [ ] Three months?
    [ ] Six months?
    [ ] Year?
    [ ] I have never accessed the clinician blog
  2. Have you ever left a comment on the clinician blog?
    [ ] Yes
    [ ] No
  3. The clinician blog includes the following different types of content. Please rank them below from 1 – the most beneficial to your clinical knowledge, through to 4, the least beneficial
    [ ] Clinical case studies
    [ ] Notice of events such as CET (continuing education and training)
    [ ] Clinical techniques
    [ ] New product information
  4. Which comments do you read on the clinician blog?
    [ ] all of them
    [ ] those on topics I’m interested in
    [ ] I don’t read the comments
    [ ] Other, please specify___________________________
  5. In general, do comments on the clinician blog improve your knowledge of a topic?
    [ ] Yes
    [ ] Sometimes
    [ ] No
  6. Have you ever learned anything new from the clinician blog?
    [ ] Yes
    [ ] No

 

 

 

 

SOCRMx week one

I’m using this blog as a space for reflective learning in connection with a MOOC that is being run by the University of Edinburgh on the edX platform.

SOCRMx Social Research Methods is an eight-week course, which for me (and a  number of other students) forms part of a twelve-week module that, ultimately, will contribute to an MSc in Digital Education.

As a newcomer to formal research much of my time has been spent trying to get my head around some of the terminology and concepts.   I’ve found the ‘beginner’s guide’ type of materials helpful where they have been available, where they haven’t I’ve resorted to trawling the Internet in search of more digestible content, and that has often helped me make sense of the heavier going topics.  For example, while some of the video explanations provided in the course materials this week were brilliantly articulated (such as Bruce Friezen’s short presentation on Epistemology), others left me more confused than I was before I watched them – such as Kathy Charmaz’ introduction to grounded theory.  Fortunately there are plenty of resources available online, including this Youtube video, that do a pretty good job of getting that particular theory/method across:

These are my responses to the questions we’ve been asked to reflect on at the end of week 1:

What kind of topics are you interested in researching?

I work in the corporate sector, so the need for learning to have an impact or outcome is often driven by commercial considerations.  At the same time, I’m fortunate to work for a purpose-driven organisation, so there are also professional, clinical and ethical reasons for ensuring everyone’s knowledge is current.  My part of the business is just one element of a global company so it would also be possible to study geographical, ethnic or cultural differences.  All of this gives me a very wide canvas for potential research topics and I think my biggest dilemma is going to be narrowing my focus down to a single, primary research question.

I do know that I would like my research to benefit the learners I support and those across the wider business at the very least.  If it were to prove useful in other contexts that would be even better!

What initial research questions might be starting to emerge for you?

I manage a digital academy that I’ve grown from the ground up with a team of software developers.  It’s early days for us yet, but the platform incorporates a number of social tools and I’d like to understand how social learning can evolve in a corporate setting.

On a personal level, I’m partipating in formal learning for the first time in around thirty-five years and I’ve often found myself comparing and contrasting the ‘learner experience’ in a university and a business setting.  There might be the seed of an idea there too.

I’m also interested in the difference in the quality of learning (if quality is the right term to use) when one type of media is used compared to another, such as video versus interactive content.

What are you interested in researching – people, groups, communities, documents, images, organisations?

All I think I’m sure of at the moment is that my research will be about people.  The social learning aspect could well take me in the direction of groups and communities and the multi-national, multi-business company I work for could see me looking at organisations.  If I do look at media it will be in relation to how people learn from it.

Do you have any initial ideas for the kinds of methods that might help you to gather useful knowledge in your area of interest?

Whatever research I undertake will probably require a mixed-methods approach, as it seems likely this might be the only way to validate the findings.

 

What initial questions do you have about those methods? What don’t you understand yet?

Most of the resources we have looked at so far have started to make sense.  As mentioned above I found the grounded theory method a little confusing initially and I’m not 100% there with it yet.  I can see how I would test the coding and categorisation I applied, but to me it seems that issues such as confirmation bias would be particularly problematic.

Do you perceive any potential challenges in your initial ideas: either practical challenges, such as gaining access to the area you want to research, or the time it might take to gather data; or conceptual challenges; such as how the method you are interested in can produce ‘facts’, ‘truths’, or ‘valuable knowledge’ in your chosen area?

As mentioned above, the real challenge for me at the moment is narrowing down the myriad of possibilities and I’m sure I will find the techniques covered in chapter five of Research Methods and Methodologies in Education helpful in doing so.

References:

Arthur, J., Waring, M., Coe, R. and Hedges, L. (eds) (2012). Research Methods and Methodologies in Education London, Sage.