University of Cambridge: Assessments of thinking skills may misrepresent poor, inner-city children in the US

Some of the assessment tools that measure children’s thinking skills in the US may have provided inaccurate information about poor, urban students because they are modelled on wealthier – mostly white – populations.

There is a big question around how we measure executive functions: are we actually using the right tools?
Annie Zonneveld
In a newly-published study of almost 500 children from high-poverty, urban communities in the United States, researchers found that a widely-used assessment, which measures the development of thinking skills called ‘executive functions’, did not fully and accurately evaluate students’ progress. The study links this to probable cultural bias in the assessment design and suggests that this may be replicated in other, similar tools.

Any such design flaw may have influenced a growing body of research which suggests that children from poorer backgrounds tend to start school with less well-developed executive functions. ‘Executive functions’ is a collective term for a set of essential thinking skills needed to carry out everyday tasks, and learning. They include working memory, self-control, the ability to ignore distractions and easily switch between tasks. Children with good executive functions tend to have better test scores, better mental health and greater employment potential.

One common method for measuring the healthy development of these skills involves asking teachers to complete questionnaires about children’s observed behaviours. The results can potentially help pinpoint children – or entire groups – who need extra support. They also provide a rich source of data for research on how executive functions develop.

In the new study, researchers found that one of these teacher rating scales, which has been widely used in the United States, was of limited value when assessing poorer, urban students. Specifically, they found that the executive function screener of a version of the Behaviour Assessment System for Children (BASC), called the BASC-2, “is not a good representation of everyday executive function behaviours by children from schools in high-poverty communities”.

The team, from the University of Cambridge (UK) and Virginia Commonweath University (US) suggest that the likely cause is that both this scale, and others like it, have been developed using an unrepresentative sample of children.

Researchers have previously pointed out that these assessments tend to be modelled on children who are mostly from comfortable socio-economic settings. By mapping their observed behaviours on to executive functions, they may falsely assume that these behaviours are ‘normal’ markers for any child of the same age. In reality, children’s different backgrounds and lived experiences may mean that executive functions express themselves differently across different groups.

Annie Zonneveld, from the Faculty of Education, University of Cambridge and the study’s first author, said: “There is a big question around how we measure executive functions: are we actually using the right tools? If they are based on white, middle-class students, we cannot be sure that they would actually work for the whole population. We may be seeing evidence of that here.”

Michelle Ellefson, Professor of Cognitive Science at the Faculty of Education, said: “Teachers can provide us with really valuable data about children’s executive functions because they can monitor development in ways we could not possibly replicate in a lab, but they need effective measures to do this. This means the assessments must draw on information about children from different backgrounds.”

According to the Children’s Defense Fund, about 14% of children in the United States live in poverty. While nearly 50% of all children are from ethnic minority families, 71% of those in poverty are from these backgrounds. Most psychometric research on executive functions, however, focuses on white, middle-income, or affluent families. It has never been clear how far its findings can be generalised.

The new study examined the executive function components of two versions of the BASC: the BASC-2 and BASC-3. These ask teachers to observe children’s everyday behaviours and rate, on a scale of ‘never’ to ‘always’, how far they agree with statements such as “acts without thinking”, “is easily distracted”, “cannot wait to take turn”, “is a self-starter” and “argues when denied own way”. They then extrapolate information about the children’s executive functions based on the responses.

The researchers analysed two sample groups of children, aged around nine or 10, all from state schools in high-poverty, urban areas in the United States. In total, 472 children took part. The first sample was assessed using the BASC-2; the other using the BASC-3.

Both groups also completed six computer-based tasks which psychologists and neuroscientists use in lab-based tasks to measure specific executive functions. The researchers looked at how far the scores from these computerised tasks – which are accurate but difficult to run with large groups – corresponded to the measures from the teacher-administered surveys.

The findings indicated that while the BASC-2 provides a reasonable overview of students’ general executive functioning, it does not capture accurate details about specific functions like working memory and self-control. The BASC-3 was far more effective, probably because it uses a different and more focused set of questions.

“The BASC-2 has been used extensively in archived datasets and contributes to academic research about how executive functions develop,” Ellefson said. “It is really important to recognise that without modification, it is not an appropriate basis for making judgements about certain groups of children.”

The assessment is just one of many surveys that measure children’s cognitive development in different countries. “It is important that we know how these tools are establishing their baseline understanding of ‘typical’ development,” Zonneveld said. “If they are based on mostly white populations from affluent suburbs, they won’t necessarily be as representative as we might hope.”