Situating Design Science Research and Projects within a Critical Realist’s perspective
DOI:
https://doi.org/10.47672/ijpm.2846Keywords:
M15 - IT Management (Design Science Research, IS Artefact Design, Evaluation of Information systems), C18 - Methodological Issues: General (Mechanisms-Based Explanation, Causal Vs Statistical Inference, Critical Realism)Abstract
Purpose: Design scientists, philosophers of science, and project designers hold mixed views on evaluating participatory design research, projects, processes, and outcomes. We argue that design science research produces varied contributions, including theories, methods, and artefacts. From a critical realist perspective, evaluation emphasizes mechanisms-based explanations. This paper examines the role of mechanisms in design science research and shows how studying them enhances statistical approaches for causal inference. It distinguishes statistical from causal inference; discusses mechanisms and mechanism-based explanations; mechanism quantification; mechanisms in explaining individual actions; and mechanisms in explaining outcomes. Drawing lessons from an empathetic participatory design process, it illustrates how design science studies can be situated within critical realism.
Materials and Methods: This study adopts design science research, combining qualitative and quantitative approaches. It includes an extensive literature review and uses empirical statistical data from mechanisms observed in participatory design processes involving prototyped citizen incident reporting applications. Data collection employed qualitative repertory grids with potential citizen users; constructs were then incorporated into a quantitative structured questionnaire for empathetic prototype testing with designers. Prototype evaluation follows the Realist Evaluation model of Context-Mechanism-Outcome Configuration (CMOc), which explains how and why interventions work by analyzing Contexts (conditions), Mechanisms (activated forces), and Outcomes (intended or unintended results).
Findings: Inferentially, co-designed artifact features show a causal-mechanism relationship to perception (sig.=0.397) and projection (sig.=0.222), but not to understanding (sig.=0.177). Significant causal interrelationships exist among situation awareness constructs. There are also significant relationships between perception and action (sig.=0.382), comprehension and action (sig.=0.312), and projection and action (sig.=0.450). Relationships involving situation awareness constructs, PAL, and e-government artifact adoption were not tested.
Implications to Theory, Practice and Policy: Informed by Personal Constructs Theory (PCT) and Situational Awareness, which view individuals as scientists forming bipolar mental frameworks to interpret and predict the world. The paper advances debates on rigor and relevance in design science research, arguing that a critical realist lens balances rigor (via statistical inference) and relevance (via problem-solving prototypes). Researchers may treat context variables as explicit or implicit ceteris paribus conditions and explore additional causal relationships based on design and evaluation contexts.
Downloads
References
Archer, M., Bhaskar, R., Collier, A., Lawson, T. and Norrie, A., 2013. Critical realism: Essential readings. Routledge.
Astbury, B. and Leeuw, F.L., 2010. Unpacking black boxes: mechanisms and theory building in evaluation. American journal of evaluation, 31(3), pp.363-381.
Baert, P., 1996. Realist philosophy of the social sciences and economics: a critique. Cambridge Journal of Economics, 20(5), pp.513-522.
Baskerville, R., 2008. What design science is not, European Journal of Information Systems, 17:5, 441-443, DOI: 10.1057/ejis.2008.45
Bedny, G. and Meister, D., 1999. Theory of activity and situation awareness. International Journal of cognitive ergonomics, 3(1), pp.63-72.
Bhaskar, R. (1978). On the possibility of social scientific knowledge and the limits of naturalism. Journal for the Theory of social Behaviour.
Bhaskar, R. and Danermark, B., 2006. Metatheory, interdisciplinarity and disability research: a critical realist perspective. Scandinavian Journal of Disability Research, 8(4), pp.278-297.
Bhaskar, R., 1975. Feyerabend and bachelard: two philosophies of science. New Left Review, 94(3), pp.31-55.
Bhaskar, R., 2014. The possibility of naturalism: A philosophical critique of the contemporary human sciences. Routledge.
Biesbroek, G.R., Termeer, C.J., Klostermann, J.E. and Kabat, P., 2014. Rethinking barriers to adaptation: Mechanism-based explanation of impasses in the governance of an innovative adaptation measure. Global Environmental Change, 26, pp.108-118.
Bisman, J., 2010. Postpositivism and accounting research: A (personal) primer on critical realism. Australasian Accounting, Business and Finance Journal, 4(4), pp.3-25.
Bloch, P.H. 1981. An exploration into the scaling of consumers' involvement with a product class. Advances in Consumer Research Volume 8: 61-65.
Bovill, C. & Bulley, C.J. 2011. A model of active student participation in curriculum design: exploring desirability and possibility. In C. Rust (Ed.), Improving student learning (pp. 176-188). Oxford: The Oxford Center for Staff and Learning Development.
Bovill, C., Bulley, C.J. & Morss, K. 2011. Engaging and empowering first-year students through curriculum design: perspectives from the literature. Teaching in Higher Education, 16(2): 197-209.
Burton-Jones, A. and Straub, D.W., 2004. Minimizing method variance in measures of system usage. Proceedings of the Southern Association for Information Systems (SAIS), pp.336-342.
Carlsson, S.A., 2006. Design science research in Information Systems: a critical realist perspective. ACIS 2006 Proceedings, p.40.
Carolan, M.S., 2005. Society, biology, and ecology: Bringing nature back into sociology’s disciplinary narrative through critical realism. Organization & Environment, 18(4), pp.393-421.
Cicmil, S., Williams, T., Thomas, J., & Hodgson, D. (2006).
Rethinking Project Management: Researching the actuality of projects.
International Journal of Project Management, 24(8), 675–686.
Davis, F.D., 1985. A technology acceptance model for empirically testing new end-user information systems: Theory and results (Doctoral dissertation, Massachusetts Institute of Technology).
De Wit K, Heerwegh D, Verhoeven JC. Can openness to ICT and scientific research predict the ICT skills and ICT use of bachelor's students? Computers & Education. 2014 Sep 1;78:397-413.
Downward, P. and Mearman, A., 2002. Critical realism and econometrics: constructive dialogue with Post Keynesian economics. Metroeconomica, 53(4), pp.391-415.
Downward, P. and Mearman, A., 2005. Critical realism and econometrics: interaction between philosophy and post Keynesian practice. In Applied Economics and the Critical Realist Critique (pp. 123-140). Routledge.
Downward, P., 2016. 10 Regression analysis: a review. Handbook of research methods and applications in heterodox economics, p.210.
Downward, P., Finch, J.H. and Ramsay, J., 2002. Critical realism, empirical methods and inference: a critical discussion. Cambridge journal of economics, 26(4), pp.481-500.
Druin, A. & Fast, C. 2002. The child as learner, critic, inventor, and technology design partner: An analysis of three years of Swedish student journals. International Journal of Technology and Design Education, 12(3): 189-213.
Druin, A. 2010. Children as codesigners of new technologies: Valuing the imagination to transform what is possible. New Directions for Youth Development, 2010 (128): 35-43.
Ellis, R.D. and Kurniawan, S.H., 2000. Increasing the usability of online information for older users: A case study in participatory design. International Journal of Human-Computer Interaction, 12(2), pp.263-276.
Elsenbroich, C., 2012. Explanation in agent-based modelling: Functions, causality or mechanisms?. JASSS, 15(3).
Endsley, M.R., 1995. Measurement of situation awareness in dynamic systems. Human factors, 37(1), pp.65-84.
Endsley, M.R., 2015. Situation awareness: operationally necessary and scientifically grounded. Cognition, Technology & Work, 17(2), pp.163-167.
Finch, J.H. and McMaster, R., 2002. On categorical variables and non‐parametric statistical inference in the pursuit of causal explanations. Cambridge Journal of Economics, 26(6), pp.753-772.
Fishbein, M. and Ajzen, I., 1975. Intention and Behavior: An introduction to theory and research.
Fleetwood, S., 2002. Boylan and O'Gorman's causal holism: a critical realist evaluation. Cambridge Journal of Economics, 26(1), pp.27-45.
Fox, S. and Do, T., 2013. Getting real about Big Data: applying critical realism to analyse Big Data hype. International Journal of Managing Projects in Business, 6(4), pp.739-760.
Gregor, S. and Hevner, A.R., 2013. Positioning and presenting design science research for maximum impact. MIS quarterly, pp.337-355.
Hawley, M. 2007. The Repertory Grid: Eliciting user experience comparisons in the customer’s voice, Accessed at www.uxmatters.com on June, 2017.
Hedström, P. and Ylikoski, P., 2010. Causal mechanisms in the social sciences. Annual review of sociology, 36, pp.49-67.
Hedström, P., 2008. Studying mechanisms to strengthen causal inferences in quantitative research. The Oxford handbook of political methodology, pp.319-335.
Hensen, J.L. and Lamberts, R. eds., 2012. Building performance simulation for design and operation. Routledge.
Hevner, A. and Chatterjee, S., 2010. Design science research in information systems. In Design research in information systems (pp. 9-22). Springer, Boston, MA.
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design science in information systems research. MIS quarterly, 75-105.
Holland, P.W., 1986. Statistics and causal inference. Journal of the American statistical Association, 81(396), pp.945-960.
Iannacci, F. and Hatzaras, K.S., 2012. Unpacking ostensive and performative aspects of organisational routines in the context of monitoring systems: A critical realist approach. Information and Organization, 22(1), pp.1-22.
Imbens, G.W. and Rubin, D.B., 2015. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press.
Ives, B., Olson, M.H. & Baroudi, J.J. 1983. The measurement of user information satisfaction. Communications of the ACM, 26(10): 785-793
Kardong-Edgren, S., Adamson, K.A. and Fitzgerald, C., 2010. A review of currently published evaluation instruments for human patient simulation. Clinical Simulation in Nursing, 6(1), pp.e25-e35.
Kouprie, M. & Visser, F.S. 2009. A framework for empathy in design: stepping into and out of the user's life. Journal of Engineering Design, 20(5): 437-448.
Kyakulumbye, S. (2020). A framework for developing citizen-centric e-government applications in developing countries: The design-reality gap in Uganda.
Kyakulumbye, S., Pather, S., & Jantjies, M. (2019). Towards design of citizen centric e-government projects in developing country context: the design-reality gap in Uganda.
Lawani, A. (2021). Critical realism: what you should know and how to apply it. Qualitative research journal, 21(3), 320-333.
Lindsay, S., Brittain, K., Jackson, D., Ladha, C., Ladha, K. & Olivier, P. 2012, May. Empathy, participatory design and people with dementia. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (521-530). ACM.
McEvoy, P. and Richards, D., 2006. A critical realist rationale for using a combination of quantitative and qualitative methods. Journal of research in nursing, 11(1), pp.66-78.
Meyer, S.B. & Lunnay, B. 2013. The application of abductive and retroductive inference for the design and analysis of theory-driven sociological research. Sociological Research Online, 18(1): 1-11.
Miller, K.D. and Tsang, E.W., 2011. Testing management theories: critical realist philosophy and research methods. Strategic Management Journal, 32(2), pp.139-158.
Mingers, J. 2006. A critique of statistical modelling in management science from a critical realist perspective: its role within multimethodology. Journal of the Operational Research Society, 57(2): 202-219.
Mingers, J., 2004. Real-izing information systems: critical realism as an underpinning philosophy for information systems. Information and organization, 14(2), pp.87-103.
Moore, G.C. & Benbasat, I. 1991. Development of an instrument to measure the perceptions of adopting an information technology innovation. Information Systems Research, 2(3): 192-222.
Olsson, E. and Jansson, A., 2005. Participatory design with train drivers—a process analysis. Interacting with Computers, 17(2), pp.147-166.
Pather, S. and Uys, C.S., 2008. Using scale reduction techniques for improved quality of survey information. SA Journal of Information Management, 10(3).
Patrick, J., & Morgan, P. L. (2010). Approaches to understanding, analysing and developing situation awareness. Theoretical Issues in Ergonomics Science, 11(1-2), 41-57.
Pawson, R. and Tilley, N., 1997. An introduction to scientific realist evaluation. Evaluation for the 21st century: A handbook, pp.405-418.
Peffers, K., Rothenberger, M., Tuunanen, T., & Vaezi, R. (2012, May). Design science research evaluation. In International conference on design science research in information systems (pp. 398-410). Berlin, Heidelberg: Springer Berlin Heidelberg.
Peirce, C.S., 1957. The logic of abduction. Peirce's Essays in the Philosophy of Science ed. by Vincent Thomas, 235-55. New York.
Pries-Heje, J., Baskerville, R. and Venable, J.R., 2008, June. Strategies for Design Science Research Evaluation. In ECIS (pp. 255-266).
Roberts, J. M. (2014). Critical realism, dialectics, and qualitative research methods. Journal for the theory of social behaviour, 44(1), 1-23.
Rogers, P.J. and Weiss, C.H., 2007. Theory‐based evaluation: Reflections ten years on: Theory‐based evaluation: Past, present, and future. New directions for evaluation, 2007(114), pp.63-81.
Ron, A., 2002. Regression analysis and the philosophy of social science: A critical realist view. Journal of Critical Realism, 1(1), pp.119-142.
Rosli, D., 2015. Cognitive awareness prototype development on user interface design. TOJET: The Turkish Online Journal of Educational Technology, 4(2), pp.32-40.
Sanders, E.B.N., 2003. From user-centered to participatory design approaches. In Design and the Social Sciences (18-25). CRC Press.
Sayer, A., 1992. Method in social science: A realist approach. Psychology Press.
Seo, S.D., 2016. A study on interaction-driven comparison between analog and digital gaming control interface on smartphone.
Smith, M.L., 2006. Overcoming theory-practice inconsistencies: Critical realism and information systems research. Information and organization, 16(3), pp.191-211.
Soros, G., 2013. Fallibility, reflexivity, and the human uncertainty principle. Journal of Economic Methodology, 20(4), pp.309-329.
Srinivasan, A. 1985. Alternative measures of system effectiveness: associations and implications. MIS Quarterly, 243-253.
Streiner, D.L. 2003. Starting at the beginning: an introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1): 99-103.
Thomas, T., Singh, L. and Gaffar, K., 2013. The utility of the UTAUT model in explaining mobile learning adoption in higher education in Guyana. International Journal of Education and Development using ICT, 9(3).
Todd, P. & Benbasat, I. 1992. The use of information in decision-making: an experimental investigation of the impact of computer-based decision aids. Mis Quarterly, 373-393.
Tsang, E.W. and Kwan, K.M., 1999. Replication and theory development in organizational science: A critical realist perspective. Academy of Management review, 24(4), pp.759-780.
Tsang, E.W., 2014. Case studies and generalization in information systems research: A critical realist perspective. The Journal of Strategic Information Systems, 23(2), pp.174-186.
Tuunanen, T., Winter, R., & Brocke, J. V. (2024). Dealing with complexity in design science research: A methodology using design echelons. MIS quarterly, 48(2), 427-458.
Venkatesh, V., Morris, M.G., Davis, G.B. and Davis, F.D., 2003. User acceptance of information technology: Toward a unified view. MIS quarterly, pp.425-478.
Venkatesh, V., Thong, J.Y., Chan, F.K., Hu, P.J.H. & Brown, S.A. 2011. Extending the two‐stage information systems continuance model: Incorporating UTAUT predictors and the role of context. Information Systems Journal, 21(6): 527-555.
Wright, M.C., Taekman, J.M. and Endsley, M.R., 2004. Objective measures of situation awareness in a simulated medical environment. BMJ Quality & Safety, 13(suppl 1), pp.i65-i71.
Wynn Jr, D. and Williams, C.K., 2012. Principles for conducting critical realist case study research in information systems. MIS quarterly, pp.787-810.
Zachariadis, M., Scott, S. and Barrett, M., 2013. Methodological implications of critical realism for mixed-methods research. MIS quarterly, pp.855-879.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Stephen Kyakulumbye, Anny Katabaazi Bwengye, Shaun Pather
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution (CC-BY) 4.0 License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
