The current state of knowledge of educational technology research and where from here

Introduction In the last two or three years, there has been an ongoing discussion, both in Norway and internationally, about Open Access, Plan S, and how academia can contribute to let all publicly financed scientific research become accessible for everyone. In this case, the Nordic Journal of Digital Literacy is a good example, since it has been open access since it was established in 2006. This has made NJDL’s research articles accessible free of charge for anyone for 13 years now, and this might have made the journal more visible both nationally and internationally, it might have stimulated the debate around educational technology, and it might have improved the citation rate for authors who have published in NJDL. But even more importantly, it has made it easier for master’s students, doctoral students and researchers to discover a selection of the current state of knowledge within educational technology (because NJDL is a scientific journal with a blind review and referee system), since everyone has had, and has, access to full length articles through NJDL’s open access.


Introduction
In the last two or three years, there has been an ongoing discussion, both in Norway and internationally, about Open Access, Plan S, and how academia can contribute to let all publicly financed scientific research become accessible for everyone. In this case, the Nordic Journal of Digital Literacy is a good example, since it has been open access since it was established in 2006. This has made NJDL's research articles accessible free of charge for anyone for 13 years now, and this might have made the journal more visible both nationally and internationally, it might have stimulated the debate around educational technology, and it might have improved the citation rate for authors who have published in NJDL. But even more importantly, it has made it easier for master's students, doctoral students and researchers to discover a selection of the current state of knowledge within educational technology (because NJDL is a scientific journal with a blind review and referee system), since everyone has had, and has, access to full length articles through NJDL's open access.

The current state of knowledge and different discourses
The current state of knowledge is vital to understand what we know, and what we do not know, within research, and within educational technology. This is continuously debated both among researchers, politicians, practitioners, parents, and others. A critical discussion around the current state of knowledge is important in every subject discipline, and the aforementioned Plan S is a measure to make this more easily accessible for everyone. Research needs to be transparent and easily accessible, to avoid speculations and over-interpretations of research findings outside the research communities.
Since the current state of knowledge is often applied as a backdrop to policy decisions concerning, for example, the practise field, it is important to acknowledge the distinctions between different discourses, where researchers are obligated to follow certain academic requirements in their search for the forefront of knowledge. As part of this, researchers are often aware of what kind of discourse 1 they are participating in when submitting their scientific article to a scientific journal, and what kind of discourse they are entering when they talk about possible practical implications of their research to a newspaper. However, this is not always as easy as it seems, because different discourses are influenced by profession, identity, contexts, role understanding and group cohesion, and therefore it is important to be aware of such phenomenon when examining educational technology in school. For instance, Labaree (2003) experienced some of this attached to teachers entering doctoral education. "He framed the problems in terms of a clash between school and university cultures that occurs when we ask teachers to shift from a normative to an analytic way of thinking, from a personal to an intellectual relationship with educational phenomena, from a particular to a universal perspective, and from an experimental to a theoretical disposition" (Boote & Beile, 2005, p. 5). Below I will use four examples of different discourses that might clarify some of the dilemmas within the educational technology area: 1. The policy level discourse: a systematic review of policy documents of tablet initiatives around the world performed by Tamim et al. (2015a) analysed the policy levels intentions, and what kind of rationale lies behind tablets in schools. In such policy documents, the research revealed that many countries in this review seem to lack sufficient educational frameworks or research-based evidence for their initiatives. This is an important research area, and future research needs to examine more properly if, and eventually how, such initiatives are attached to deeply entrenched structures (Cuban et al. 2001), like the national curriculum, and if education technology is a remedy to fulfil these intentions (or not).
2. The students learning discourse: a meta-analysis and research synthesis by Sung, Chang and Liu (2016) goes beyond the policy level intentions, and investigate the effects of integrating mobile devices with teaching and learning on students' learning performance. They examined 110 experimental and quasi-experimental journal articles, and found an effect size of 0.523 for the educational technology examined, and an effect size of 0.615 (Sung et al. 2016, p. 258) for tablets specifically. In addition, a systematic review and a metaanalysis by Tamim et al. (2015b) also goes beyond the policy level intentions, and investigate the research evidence of tablets used in educational contexts. They examined 27 quantitative studies and 41 qualitative research studies, and found an effect size of 0.23 of comparing tablet use contexts with no-tablet use contexts. They also examined two different uses of tablets by students, and found an effect size of 0.68 that showed a significant favouring of more student-centred pedagogical use of technology (Tamim et al. 2015b). Moreover, a meta-analysis and research synthesis of learning in one-to-one laptop environments by Zheng, Warschauer, Lin & Chang (2016) examined 65 journal articles and 31 doctoral dissertations, and revealed modest, but significantly positive, average effect sizes in English, writing, mathematics, and science.
3. The teacher discourse: the TALIS 2018 study reveals that teachers have different experiences with educational technology, and that they say they need to improve their ICT skills (OECD 2019). The latest PISA study (OECD 2015) on education technology use in school reveals the same tendencies, stating ". . . we have not yet become good enough at the kind of pedagogies that make the most of technology" (OECD 2015, p. 5), and ". . . adding 21st-century technologies to 20th-century teaching practices will just dilute the effectiveness of teaching" (OECD 2015, p. 5).
4. The user opinion discourse: a search on Nettbrett i skolen 2 in the search engine Retriever shows 2,717 hits in Norwegian newspapers from 2010-2019, which indicates there has been a large engagement around tablets in school, and much debate about the pros and cons of such educational technology in school.
Each of these discourses is important in itself, and we need them all. But when they are blended together, it is hard for the public, parents, children, politicians, educational leaders, and so on, to actually understand what the current state of knowledge is, because it depends on what discourse we are talking about. The scientific journals, universities, researchers and research communities around the world have therefore a responsibility to remind us all what the premises of what it is that can be defined as evidence-based research as part of the current state of knowledge, and what is actually attached to other, important discourses in society. So, to illustrate the different discourses, one can use an example: if practitioners sometimes state that there is "a lack of research" within a certain school area, this might be right of course, it can be partly right, but it can also be debatable or used rhetorically to argue for or against interventions in school based on their experiences. We might say that these are understandable statements, since practitioners first and foremost participates in practise discourses based on their own personal narratives, pedagogy, didactics and is seldom researchers per se. They are not expected to be experts on the current state of knowledge (e.g. the latest meta-analysis or systematic review within a certain school area), but rather experts in teaching. Practitioners experiences are very important in itself and it continuously improves the practise field and contributes to school development in general. At the same time, it has an especially focus on the local setting and one of the strengths with research is that it goes beyond the local setting and examines if practise and practise experiences holds beyond the local setting. In this way practise and research are complementary, and each of these domains has its strengths and limitations.
However, when a researcher states that there is "a lack of research" within a certain school-related domain, this statement implies that the actual researcher should have the overview over the current state of knowledge, because this is an important part of the expertise attached to researchers (achieved through their doctoral education, etc.). They participate first and foremost in the researcher discourse where the academic conventions imply many "gate keepers" (reviewers, editors, co-authors, etc.) which secures the quality of the research. E.g. to find this current state of knowledge requires the ability to understand these academic conventions, to carry out "research on research", to find the forefront of knowledge and position the actual researchers' study to the "blind spot" or "gap" within the subject area (or to replicate existing studies). The academic requirements for revealing the forefront of knowledge and knowledge status in a field of study is therefore to carry out a knowledge synthesis that shows what results researchers have previously achieved within the field of study. In the research community, it is assumed that to be able to find the current state of knowledge you need to master this special methodological craft, the literature review craft. The literature review craft requires high academic competence, a lot of hard and systematic work, with retrieving, screening and reviewing often hundreds of scientific articles for each study, and thus requires certain advanced transferable skills for managing such literature reviews.
But this is not enough -such reviews also require high competence within the actual subject area, since they are based on interpretation of the primary studies and previous research. The educational technology area, where there is the very speedy development of 2. See: https://web.retriever-info.com/services/archive/search . Search string: "Nettbrett i skolen". new technology, especially requires awareness and subject expertise within it. For example, it is quite common in meta-analyses and systematic reviews within educational sciences and teacher education today to examine studies in the time span of the last 10-15 years, for example, 2000-2015. This is of course normally fine for the majority of subject disciplines, but within the educational technology discipline, this creates a certain dilemma. This is mainly because the topic is "a moving target", encompassing very speedy education technology development that meta-analyses and systematic reviews are not completely able to capture, since these have a retrospective perspective. This is because these kinds of reviews require already-published empirical studies to build upon, and when we know that the publications process of a single scientific article (primary studies) is a process that takes time, and such reviews take even more time to get published (since they examine the primary studies), one might say (in a rather caricatured way) that the current state of knowledge sometimes is more "yesterday's current state of knowledge" within this area. To solve some of the dilemma, we need to combine systematic reviews and meta-analyses with rapid reviews, scoping reviews, and state-of-the-art-reviews (Grant and Booth 2009;Krumsvik & Røkenes 2019), to keep up with this development.

Different disciplines -different traditions
Another challenge that we need to be aware of when discussing the current state of knowledge is that different subject disciplines have had, and have, different traditions, premises and frame factors for finding the current state of knowledge. This creates different possibilities for carrying out systematic reviews and meta-analyses that have the strongest impact on the current state of knowledge. For example, the premises in educational sciences, teacher education and educational technology research (where the majority of NJDL's authors come from), for carrying out systematic reviews and meta-analyses, are quite different from, for example, the premises in health sciences. The reason for this is that research literature in these educational areas can still be described as diverse, and not always adapted to the conventions usually found for systematic reviews and meta-analyses in other disciplines (especially in other fields of professional education, such as medicine, psychology, dentistry, veterinary medicine, etc.).
The main reasons for this are that the research articles are not always systematically indexed in the large search databases in educational sciences, teacher education and educational technology research, and that the abstracts do not always explicitly mention the research methods used. This makes it sometimes more difficult to find the current state of knowledge by only using search databases, and presents some additional challenges that researchers in educational sciences, in teacher education and in educational technology research have to deal with, compared to researchers within, for example, health education. To use an example -two of the pieces of referred literature (Tamim, et al. 2015, Tamim et al. 2015b) in this editorial paper were not systematically indexed in a way that would appear in, for example, the ERIC database search, and can be described as "grey literature", even though these are quite essential research reports within educational technology and contribute to the current state of knowledge within this area.
This example illustrates the reason that educational sciences, teacher education and educational technology research makes far more use of "grey literature" (e.g. doctoral dissertations, scientific reports, book chapters, etc.) than the health sciences do, and from this we can see that the literature review process sometimes differs from other disciplines, and needs to take this into account. This also means that subject disciplines have different premises and possibilities for carrying out systematic reviews and meta-analyses. Medicine and psychology are disciplines that are registered systematically in indexed databases with long traditions (PubMed and Psychinfo). Such databases are accumulative, making it far easier to identify studies in these areas in a more systematic way than in teacher education, educational research and educational technology research.
How the current state of knowledge is established is also related to the different subject disciplines' research traditions and history within different methodological approaches. Below, Pawson (2006) 3 illustrates the span from RCT design to practitioners' experiences or opinions as a trigger for reminding us that, for example, practitioners' discourse (#9) is important, but something completely different from the research discourses in #1-7.
While, for example, health sciences have been dominated by quantitative research designs (see Pawson's #1, 2, 3 and 4 above), one will often find most of these scientific studies published in scientific journals where replication and aggregation are central. This is different in educational sciences, teacher education and educational technology research, where research in school has often been practice oriented, and starts with dialogue with practitioners in schools (see Pawson's #8 & 9). This can create fruitful action research collaborations, where both researchers and practitioners collaborate in such research partnerships, and is one of the reasons why these areas have been dominated by qualitative research designs (see Pawson's #5 and 6). The implications of this is that one would have to rely largely on a mixture of scientific journal articles and "grey literature" published in other publications channels. Therefore, aggregation and meta-analysis will not always be relevant in the same way (e.g. if no effect studies are carried out in a certain area), and a configurational approach can then be applied (e.g. meta synthesis).
A consequence of this is that other professional education programmes, such as medicine, psychology, veterinary medicine, dentistry and other health education programmes, have much longer histories and traditions for scientific journal publishing, meta-analyses, systematic reviews and intervention studies. Conversely, this is a more recent phenomenon within educational sciences, teacher education and educational technology research both nationally and internationally, but is now gradually changing (as the meta-analysis and systematic reviews referred to in this paper show). And when so much of the methodology literature about different kind of literature reviews that exists are based on the health sciences, there is much here on which educational sciences, teacher education and educational technology research can build on. However, there is also a need to make adjustments that take into account the aforementioned differences and the history of educational sciences, teacher education and educational technology research related to literature review, so that this is more geared to the context of these educational disciplines.
3. Such "lenses" can be clarifying when positioning your own literature review in a research study, but this hierarchy is also debatable, since it also can be used to rank different kind of research designs.

Current state of knowledge and triangulation
So, to be able to find the current state of knowledge and to address these contextual factors in educational sciences, teacher education and educational technology research, it seems like sometimes using only a database search is not sufficient. It seems like it is necessary to conduct systematic literature searches in large databases (ERIC, Web of Science, Scopus, etc.), but combine these with "hand searches" (e.g. examining the reference lists in systematic reviews and large scale studies), "grey literature" (e.g. doctoral dissertations, research reports, book chapters, etc.) and "personal archive" (e.g. national research reports, "classic" books, etc.), thus obtaining a traditional or narrative review. A traditional or narrative review triangulates, and has a slightly different essence than a systematic review, although it is based on the same basic principles. When educational sciences, teacher education and educational technology research still not have the same opportunities for literature review like, for example, the health sciences, this can be a good strategy. And if one in addition applies databases specifically designed for "grey literature", this contributes to quality assurance of the literature review (see: http://dissexpress.umi.com/dxweb/search.html and http:/ /www.worldcat.org/advancedsearch).
With this backdrop, one should keep in mind that Petticrev and Roberts mentions that ". . . many traditional literature reviews are excellent. Moreover, not every review needs to be a 'systematic' review" (Petticrew & Roberts 2006, s. 189). So, when practicing this kind of "triangulation", the researcher will find the relevant literature (based on the research question) in a good way, but it requires a "paper trail", where one stores search strings, searches, choice of databases, and how "hand searches" were justified for the priorities in the "personal archive", so that all is transparent and well-documented. Although criticism may be raised against traditional or narrative reviews, it is also true that ". . . similar biases can apply to systematic reviews, and these have been extensively documented by systematic reviewers and meta-analyst themselves" (Petticrew & Roberts, 2006, p. 159). The criticism is especially attached to publication bias (e.g. submitted articles with no significant findings, or no or negative effect sizes, are more often rejected), and the rationale for searching the grey literature is to minimize publication bias. McAuley and colleagues (2000) examined a random sample of systematic reviews in a range of disease areas. Only 33% of the included systematic reviews contained both grey and published studies (Booth et al. 2016, p. 120). They found that generally, published studies showed larger effects of the intervention when compared with grey literature, thus demonstrating that excluding "grey literature" from systematic reviews can lead to exaggerated estimates of effectiveness (Booth et al. 2016, p. 120).

Summary
Since the start of NJDL in 2006, the majority of research published in NJDL has been practice oriented, which is natural within this educational technology field, with its close connection to educational science and teacher education research. This tendency of practice-oriented research has also had a certain impact on what kind of literature reviews are carried out and what kind of research designs are applied. It is reasonable to say that NJDL so far has been dominated by formative studies, action research, qualitative case studies and ethnographic research. On one hand, this is natural, since the area is a "moving target" with continuously speedy development of new technologies that often call for explorative research designs and case studies. On the other hand, (as with other research disciplines), we also need more RCT studies (and basic research) 4 about educational technology, because such research has a lot to say about how to set up literature reviews. For example, if a subject discipline does not have traditions for carrying out intervention studies, effect studies, RCT studies, and so on, as primary studies, it is hard to carry out meta-analyses, and also difficult to carry our systematic reviews (which are acknowledged as the two of the most cited and important literature reviews within research communities and for informing the policy level). Since educational sciences, teacher education and educational technology research do not have the same long traditions as other disciplines (e.g. medicine and psychology) for such research designs, this seems to be one of the reasons for why there are fewer meta-analyses and systematic reviews within these areas (but this is now gradually changing).
However, if we take a closer look at the time span, there are new tendencies within educational technology that are worth glancing at (and maybe this gives nuance to the picture a little bit). Some will say that within educational technology research, we have more metaanalyses and systematic reviews than it seems like we have in the public debate, and especially given the short time span for some technologies, for example, tablets. Since the first tablet with global impact came in 2010 (Encyclopedia Britannica 5 ), it is of course only within a time span of nine years that it has been possible to carry out both primary research (empirical research) and systematic reviews and meta-analyses on the educational use of tablets. If we compare this time span with PCs, which have existed since the late 1970s, we might say that there are quite different premises and possibilities for the amount of research that has been possible. This can be exemplified with Tamim et al.'s (2011) meta-analysis about PCs and "What Forty Years of Research Says about the Impact of Technology on Learning: A Second-Order Meta-Analysis and Validation Study", which included 1,055 primary studies and 25 meta-analyses, and found an effect size of 0.35. If we see this in relation to the aforementioned shorter time span for the meta-analyses and systematic reviews by Sung et al. 2016 andTamim et al. (2015b), one might say that research on tablets in education it is not as minor as often described in the public debate (since the time span is only 9 years).
The socio-culturist James Wertsch reminds us about ". . . how the introduction of novel cultural tools transforms the action" (Wertsch 1998, p. 42), and we need more research in the future to examine what cultural tools like PC's, tablets, and other mobile devices "give and take" in children's schooling. Wertsch also reminds us about the importance of a competent agent to utilize the cultural tools' affordances: ". . . cultural tools such as poles in pole vaulting and the forms of syntax used in solving multiplication problems are powerless to do anything. They can have their impact only when an agent uses them" (Wertsch 1998, p. 93). This perception of our many cultural tools we use daily in school, in sport, and so on, is in line with Manuel Castells, who says that "[…] Internet use, and educational technology in general, are only as good as the teachers who use it" (Castells, 2001, p. 258). So educational technology seldom leads to educational innovation, better teaching or better learning alone, and in the coming years we need to explore and discover this "moving target" in education with qualitative approaches, and test and confirm this with quantitative approaches that go beyond only our intuition, beliefs, meanings and experiences from our use of technology in our spare time. In this process we need to continuously inform the current state of knowledge with large scale primary studies, systematic reviews, meta-analysis, and so on, within educational technology research based on a high degree of transparency, where Open access will be a natural part of NJDL also in the coming years. We also need more primary studies about digitalization and educational technology use in education, and especi-4. See e.g. Mayer (2014) and Mayer & Alexander (2017). 5. See: https://www.britannica.com/technology/tablet-computer ally also how artificial intelligence/machine learning, Cloud-technology, Blockchaintechnology, Cybersecurity & GDPR (https://gdpr-info.eu/) and Internet of things affects pupils' digital Bildung-journey within and outside school in our digitized society.