Methods for Investigating Digital Personalised Learning in Kenya: Utilising A/B Testing to Evaluate Software Adaptivity Features

Chen Sun, Louis Major, Nariman Moustafa, Rebecca Daltry

This blog focuses on a research study investigating the impact of Digital Personalised Learning (DPL) on pre-primary education in Kenya. The study features utilising A/B testing to examine the pedagogical implications and learning effectiveness of various design features in a classroom-integrated DPL tool.

DPL is attracting increasing interest from researchers and practitioners, given its potential to adapt to individual needs and empower learners to determine their own pace and timing of learning [1, 2]. However, research on DPL to-date has primarily focused on high-income and well-resourced contexts, creating an opportunity to investigate its role in low- and middle-income countries (LMICs) [4]. Growing evidence indicates that DPL [6] could play an important role in improving learning outcomes in LMICs. For instance, DPL might help address challenges such as limited teaching resources, increase learner access to education inside and outside of school, enable remediation based on individual learning levels, and mitigate the negative effects of high teacher-learner ratios. Within the ICT4D community, there has been an expansive conversation on using technology to address development challenges and improve the quality of life in global south economies. Yet, there is scope for further discussion on the development of DPL in these regions.  

Multi-strand Research on Integrating DPL in Kenyan Pre-primary Education

Researchers from the University of Manchester have been working as part of an EdTech Hub research study alongside other Hub colleagues, and in partnership with EIDU and Women Educational Researchers of Kenya, to investigate the development and evaluation of a DPL tool used by early-grade learners. In this context, an ambitious multi-strand research study (2022-25) is rigorously evaluating contextually appropriate pedagogical and software approaches for integrating DPL into schools in Kenya. The principal research question is: How can a classroom-based DPL tool most effectively support early-grade numeracy and literacy outcomes in Kenya? This research focuses on the use of the EIDU DPL tool, deployed on low-cost Android devices.

Research on Adaptivity and Data Feedback

This blog specifically explores one aspect of the larger multi-strand research study introduced above, focusing on adaptivity and data feedback. This particular study strand zeroes in on the question of designing and assessing various personalisation features in the EIDU DPL tool for pre-primary learning. In the study, one or two smartphone devices are distributed to each classroom. The learner-facing interface contains learning units targeting specific literacy and numeracy skills, and the teacher-facing interface (depending on the software version) can include lesson plans aligned with the Kenyan curriculum. The DPL tool also facilitates adaptive assessment and measurement strategies, generating continuous insights into learning. The main adaptivity feature of the DPL tool determines the sequence of learning content for a particular learner. To date, around 250,000 active learners from 4,000 pre-primary schools in Kenya are using EIDU.

DPL Adaptivity Feature Design Through Iterative Rounds of Software Interventions

Here, we reflect on the methods that facilitate this particular strand of research, which is concerned with the question: how can various adaptivity features of the EIDU DPL tool support classroom teaching and learning? One key area of our collaboration is to identify software changes that can be implemented within the DPL tool to test their effects on learning scores and engagement. Three iterative rounds of adaptive feature design have taken place, which have resulted in the development of four software interventions directed at learners, and five that cater to teachers. All nine software features aim at enhancing digital personalisation. 

The learner-facing personalisation features focus on providing adaptive learning paths for individual learners. For example, one intervention tests different personalisation algorithms that are designed to select the most suitable learning activities based on the performance history of learners’ interaction with the DPL tool’s learning content. The designed algorithms select the next learning activity either to maximise scores or learners’ engagement level, as opposed to a learner progressing through a curriculum following a fixed sequence carefully arranged by educational experts. 
The teacher-facing personalisation features explore how to meaningfully present learners’ data to teachers. The intended aim is to empower teachers with a better understanding of learners’ performance, enabling them to make informed pedagogical decisions and intervene when needed. For example, teachers can view a dashboard demonstrating learners’ progress and where learners are categorised in terms of competency levels per curriculum item. This EdTech Hub blog outlines further information on the nine interventions. 

A/B Testing: A Software Design Evaluation Method

Understanding the impact of those personalisation features on learning and teaching poses a challenge. To address this, A/B testing is employed to test the comparative effectiveness of designed software features. A/B testing is a controlled experimental study for evaluating the efficacy of design elements by randomly assigning participants to different software versions [3]. This random assignment intends to minimise bias and ensures comparability between different software versions. Further, A/B testing enables continuous and unobtrusive large-scale experimentation, allowing for the assessment of design changes in technology-enhanced learning environments without interrupting regular teaching activities [5]. The purpose of implementing A/B testing in this study is to identify effective personalisation features that can enhance learning outcomes for pre-primary education in Kenya. 

The personalisation design features are first pilot tested among a small sample of 20 pre-primary schools, where teacher feedback and classroom observations are obtained to further understand the user experience. When no abnormalities are detected during the pilot, the features are then released as an A/B test to EIDU’s mass user base of around 250,000 monthly active learners. The A/B test typically lasts for a school term, as this represents a predetermined period of time to ensure consistent learning experiences throughout the test duration. 

At the time of writing, some of the tests are still running, with these scheduled to conclude by April 2024. Testing is continuously monitored by EIDU, and all data collected via the software is anonymised at source by assigning unique user IDs [3]. Since September 2023, the team have been analysing anonymous data to gauge the impact of those design elements. The findings provide direct input into the design and development of the EIDU tool in the Kenyan context and open up avenues for future research. For example, design features that lead to improved learning outcomes can be implemented as default settings, and new features can be launched as a series of A/B tests to continuously refine and optimise the DPL tool.

A/B Testing: Reflections on Opportunities and Limitations

The use of A/B testing in this research strand opens up new opportunities for understanding the impact of DPL tool design features on pre-primary learning in LMICs. By conducting a series of A/B tests, we are able not only to identify what works best from a pedagogical perspective but also to uncover how personalisation can be effectively integrated into classrooms at scale. Additionally, A/B testing allows for the optimisation of software design through continuous iterations and refinements without disrupting the user experience. This approach offers an innovative methodology in educational research, particularly for large-scale experiments involving a large number of learners. 

Nonetheless, as A/B testing primarily focuses on software-generated data, it is important to acknowledge how some external factors which influence a learner’s performance may be overlooked. Thus, to gain a more comprehensive understanding, it is highly valuable to complement A/B testing with rigorous qualitative data, such as classroom observation. Additionally, there is a separation between researchers and the actual classroom environment. This detachment requires careful ethical consideration of matters related to data collection, usage and storage, in recognition of the distance between researcher and participant. To bridge this gap and ensure the transparency and accountability of the research, the team is committed to sharing a concise summary of findings and outcomes in an accessible format at the study’s end, available to all participants.

Despite these complex considerations, our experience with A/B testing suggests the potential of utilising this method in digital education research. Pioneering an innovative research method requires collaboration between multiple stakeholders, including but not limited to education researchers, data scientists, developers and educators, to think about the different pedagogical, ethical and technological elements involved. Sharing our insights with the ICT4D community is to invite a conversation on how A/B testing and other innovative research methodologies for large-scale software experiments can be adapted and refined for different research contexts.

References

[1] Bernacki, M. L., Greene, M. J., & Lobczowski, N. G. (2021). A systematic review of research on personalized learning: Personalized by whom, to what, how, and for what purpose (s)?. Educational Psychology Review, 33(4), 1675-1715. https://doi.org/10.1007/s10648-021-09615-8

[2] Bhutoria, A. (2022). Personalized education and artificial intelligence in the United States, China, and India: A systematic review using a human-in-the-loop model. Computers and Education: Artificial Intelligence, 3, 100068. https://doi.org/10.1016/j.caeai.2022.100068

[3] Friedberg, A. (2023). Can A/B Testing at Scale Accelerate Learning Outcomes in Low- and Middle-Income Environments?. In: Wang, N., Rebolledo-Mendez, G., Dimitrova, V., Matsuda, N., Santos, O.C. (eds) Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky. AIED 2023. Communications in Computer and Information Science, vol 1831. Springer, Cham. https://doi.org/10.1007/978-3-031-36336-8_119

[4] Major, L., Francis, G. A., & Tsapali, M.. (2021). The effectiveness of technology-supported personalised learning in low- and middle-income countries: A meta-analysis. British Journal of Educational Technology, 52, 1935–1964. https://doi.org/10.1111/bjet.13116

[5] Savi, A. O., Ruijs, N. M., Maris, G. K. J., & van der Maas, H. L. J. (2018). Delaying access to a problem-skipping option increases effortful practice: Application of an A/B test in large-scale online learning. Computers & Education, 119, 84–94. https://doi.org/10.1016/j.compedu.2017.12.008

[6] Van Schoors, R., Elen, J., Raes, A., & Depaepe, F. (2021). An overview of 25 years of research on digital personalised learning in primary and secondary education: A systematic review of conceptual and methodological trends. British Journal of Educational Technology, 52, 1798–1822. https://doi.org/10.1111/bjet.13148

ICTs and Precision Development: Towards Personalised Development

Are ICTs about to deliver a new type of socio-economic development: personalised development?

ICTs can only have a significant development impact if they work at scale; touching the lives of thousands or better still millions of people.  Traditionally, this meant a uniform approach where everyone gets to use the same application in the same way.

Increasingly, though, ICTs have been enabling “precision development”: increasingly-precise in terms of who or what is targeted, what is known about the target, and the specificity of the associated development intervention.  The ultimate end-point would be “personalised development”: interventions customised to each individual.

Elements of digitally-enabled individualisation have already emerged: farmers navigating through web- or IVR-based systems to find the specific information they need; micro-entrepreneurs selecting the m-money savings and loan scheme and level that suited them.  But there is still rigidity and constraints within these systems.

Though we are far from its realisation, the potential for truly personalised development is now emerging.  For example:

• Personalised Learning: “a methodology, according to which teaching and learning are focused on the needs and abilities of individual learners”[1]. ICTs are integral to personalised learning and technology-enabled personalisation has had a demonstrable positive impact on educational performance[2].
• Precision Agriculture: though around as a concept for at least two decades, precision agriculture is only now starting to find implementations – often still at pilot stage – in the global South[3]. Combining data from on-ground sensors and remote sensing, precision agriculture provides targeted guidance in relation to “seeds, fertilizers, water, pesticides, and energy”.  The ultimate intention is that guidance will be customised to the very specific soil, micro-climate, etc. parameters of individual farms; even smallholder farms.
• Personalised Healthcare: diagnosis and treatment may appear personalised but typically involve identifying which illness group a person belongs to, and then prescribing the generic treatment for that group. This is becoming more accurate with improvements in electronic health records that provide a more person-specific history and context[4].  Precision medicine prescribes even more narrowly for the individual; typically based on genetic analysis that requires strong digital capabilities.  Though at early stages, this is already being implemented in developing countries[5].

ICTs are thus leading us on a precision development track that will lead to personalised development.  The promise of this can be seen in the examples above: individualised information on learning level, farm status, or health status that then enables a much more effective development intervention.

It will be interesting to log other examples of “ICT4PD” as they emerge . . .

Follow @CDIManchester

[1] Izmestiev, D. (2012). Personalized Learning: A New ICT-Enabled Education Approach, UNESCO Institute for Information Technologies in Education, Moscow.

[2] Kumar, A., & Mehra, A. (2018). Remedying Education with Personalized Learning: Evidence from a Randomized Field Experiment in India, ResearchGate.

[3] Say, S. M., Keskin, M., Sehri, M., & Sekerli, Y. E. (2018). Adoption of precision agriculture technologies in developed and developing countries. The Online Journal of Science and Technology, 8(1), 7-15.

[4] Haskew, J., Rø, G., Saito, K., Turner, K., Odhiambo, G., Wamae, A., … & Sugishita, T. (2015). Implementation of a cloud-based electronic medical record for maternal and child health in rural Kenya. International Journal of Medical Informatics, 84(5), 349-354.

[5] Mitropoulos, K., Cooper, D. N., Mitropoulou, C., Agathos, S., Reichardt, J. K., Al-Maskari, F., … & Lopez-Correa, C. (2017). Genomic medicine without borders: Which strategies should developing countries employ to invest in precision medicine? Omics: A Journal of Integrative Biology, 21(11), 647-657.

The Quest for the Digitisation of Education in Developing Countries: Are we Forgetting Teachers?

The development of every country partly depends on how strong and reliable its educational system is to produce the best minds to innovate and bring new solutions to that country’s challenges. Often, this task automatically falls on teachers who generally get the blame for all failures of the education system but rarely get the praise for its successes. In this modern the era of technology where students’ ICT literacy has come to the fore because of its potential for economic development, teachers are in the firing line of anyone who thinks the education system is not doing enough to prepare learners for the technology skills they need for surviving in today’s digital world.

With the SDGs showing that education will be the key to many of today’s world ills, it is probably high time we stopped and considered exactly what that education will look like in deprived contexts, should we continue to do business as usual. Yes, we want education for all and all that. We certainly want quality education and of course in the 21st century, quality education must respond to the needs of the next generation. In a world permeated with technology though, quality education is increasingly necessitating the use of technology for teaching and learning enhancement. As much as everyone wants children in developing countries to receive education that will enable them to harness the power of technology as the developing world tries to catch up with the rest, learning with technology remains a dream in many developing countries.

All we need is access to ICTs, right?

In developing countries, access to ICTs in schools is limited and costly in the rare instances where it is provided. The expectation is therefore that as soon as technologies—mainly computers—are available in schools, teachers would unreservedly make great use of them and just like a magic wand, improve students learning and overall ICT literacy, all of which are to contribute to the development of the concerned countries. Such has been the thinking behind many developing countries’ investments in the One Laptop Per Child project and similar projects. This technocentrism that has been decried for a while now [1] is yet to bear fruit despite many developing countries still constantly biting the same bait in hope of … well, a different outcome? After all, the $100 One Laptop per Child is no longer seen as the laptop that will save the world as the New York Times once claimed.

A key flaw of the technocentric view of ICT in education is that when the expected outcomes are not obtained, nothing else could be responsible but the teachers. Given the cost of these technologies to the otherwise deprived developing countries, the thought of teachers not making use of them is often intolerable. Why in the world would they not elect to use such equipment that cost so much to get? Are they not aware that some financial sacrifices were made to bring those devices to the schools? Have they forgotten that the country is betting its development on students’ skills to use those technologies creatively? So, teachers are always seen as potentially problematic in efforts to digitise the education sector. This negative image of teachers has not been helped by claims that teachers are a class of less technologically savvy digital immigrants who can hardly use ICTs to the liking of their supposedly technology-savvy, digital native students [2].

But do we really know the teachers?

If as it is now generally assumed, technology literacy skills of the next generation of learners are the responsibility of teachers [3], then understanding who we are entrusting that task with should be a priority. We are expecting great works from teachers in the building of our digital economies and that should mean we know better who they are and how they become the people we give such big responsibilities. Masterpieces don’t paint themselves, neither are they a product of a brush nonchalantly placed in the clumsy hands of an amateur. So, ICT in education ought not to be summed up with handing computers or laptops to teachers and schools before sitting back and awaiting miracles to happen. They surely won’t. The need for a generation of skilled men and women fully equipped with the ‘21st century skills’ of which ICT skills are central should be considered second to the understanding of the men and women who are going to make that ICT literate generation a reality: the technology-using teachers. There is a need to know what teachers are really willing and capable of doing with ICTs before counting unhatched chickens of economic transformations that we will get from ICTs, especially in the developing world.

Let’s take an example of Rwanda where I am currently doing a study on the development of identities as technology users of pre-service teachers. (You can read more about the study in this blog post). Rwanda is a country that has gained international acclaim for its efforts to digitise itself and its ICT-friendly policies. Without any other substantial resources, ICTs have been put at the centre of its economic ambition, and this privileged position has had them dubbed ‘the heart of the education sector’ [4]. As a result, investments in the acquisition of laptops for schools have been ongoing since 2007. Nevertheless, once in schools, these technologies have not been used as initially hoped. In fact, recently the Ministry of Education found itself left with no choice but to instruct school leaders to ensure their teachers are using the resources available or risk losing their jobs, after it was revealed that many of these devices had remained in their original boxes while others just disappeared. The easy question here is why are the teachers not using these resources in the first place? Why would they wait for their head-teachers to lose their jobs before they start using ICT resources given to their schools, for free? But an even better question would be ‘Who are those teachers, who are failing to use technologies given to them for free?’ How did they come to be who they are? What has made them to be the technophobes they are portrayed to be? These are some of the considerations that are the genesis of my ongoing PhD study.

My research wants to understand who the teachers expected to use the increasing number of technologies in schools are, by approaching the problem from a socio-cultural point of view. Teachers’ ICT training and usage don’t happen in a vacuum. So, it is important to understand how the different contexts they grow up in (socially and professionally) shape who they are in relation to ICTs and therefore influence the likelihood of them using and negotiating the use of any technologies in any way with their students.

In this study I am following pre-service teachers during a year-long internship to understand how ICT policies and training programmes translate into actual technology using teacher identities. This means understanding the key influences that teachers-in-training have and the extent of these influences on the end product that ends up in schools. This understanding can already help predict what route will lead to teachers who are most likely to use the technologies that are available to them in the context.

Given the many factors that come into play regarding ICT use in education, I attempt to follow the training line to understand first, what ICT roles teachers are trained for and expected to play in policies and teacher education programmes respectively. Then I look at the support and the influence that come from those directly in charge of student-teachers development process (teacher educators and school-based mentors). How do they guide them in the use of ICTs? Does their practice encourage or discourage the development of ICT skills for these candidates who want to become teachers in a country that counts on technology to achieve its development goals? How does the existing ICT environment allow the teacher-trainees to cultivate and exercise their agency as professionals trying to achieve learning objectives with(out) the support of ICTs?

Answers to these questions will certainly not solve all the challenges related to ICTs for development especially as seen from an education perspective. However, they will give a picture of who the teachers are and what needs to be done to get them using already available technologies in Rwandan schools and also schools in similar contexts.

References
[1] S. Papert, ‘A Critique of Technocentrism in Thinking About the School of the Future’, in Children in the Information Age, Elsevier BV, 1988, pp. 3–18.
[2] M. Prensky, ‘Digital Natives, Digital Immigrants’, On the Horizon, vol. 9, no. 5. pp. 1–6, 2001.
[3] D. Epstein, E. C. Nisbet, and T. Gillespie, ‘Who’s Responsible for the Digital Divide? Public Perceptions and Policy Implications’, The Information Society, vol. 27, no. 2, pp. 92–104, Feb. 2011.
[4] Ministry of Education, ‘Education Sector Policy’, Kigali, 2003.

Follow @CDIManchester