Lessons learned on development and application of agent-based models of complex dynamical systems

The field of agent-based modelling (ABM) has gained a significant following in recent years, and it is often marketed as an excellent introduction to modelling for the novice modeller or non-programmer. The typical objective of developing an agent-based model is to either increase our mechanistic understanding of a real-world system, or to predict how the dynamics of the real-world system are likely to be affected by changes to internal or external factors. Although there are some excellent ABMs that have been used in a predictive capacity across a number of domains, we believe that the promotion of ABM as an ‘accessible to all’ approach, could potentially lead to models being published that are flawed and therefore generate inaccurate predictions of real-world systems. The purpose of this article is to use our experiences in modelling complex dynamical systems, to reinforce the view that agent-based models can be useful for answering questions of the real-world domain through predictive modelling, but also to emphasise that all modellers, expert and novice alike, must make a concerted effort to adopt robust methods and techniques for constructing, validating and analysing their models, if the result is to be meaningful and grounded in the system of interest. © 2017 Elsevier B.V. All rights reserved.


Introduction
Through the ever-continuing advancements in hardware and software technologies, the scale and sophistication of modelling tools continues to increase. In particular, the focus on developing agent-based modelling (ABM) tools over the last two decades, has resulted in several standardized libraries emerging (see [1,2] for discussion). These have allowed researchers with subject matter expertise (but not necessarily computer science expertise or progamming experience) to quickly develop models of their real-world systems of interest, and to perform cause-and-effect experiments through computer simulations. The underlying conceptual logic of the ABM approach is relatively simple for a newcomer to grasp, because real-world entities, such as a person or a biological cell, can have a straightforward mapping to a computational analogue known as an agent.
ABMs can provide a powerful means by which to formulate a model that replicates the known characteristics and behaviour(s) of a real-world system, or to explore and experiment on the dynamics and underlying mechanistic behaviour(s) of the system, where individual agents are autonomous and respond to the simulated environment [3] . The behaviour of agents is governed by pre-defined agent rules, and even the simplest agent rules can give rise to complex population level phe-nomena. It is the study of the resultant population behaviours with respect to the heterogeneous and autonomous individuals that is one of the key advantages of agent-based modelling [4] , and has led to its rapid rise in adoption over the past decade [5][6][7] . For instance, the agent-based approach has been used in a diverse range of disciplines including the modelling of: diseases in biology [8] ; financial markets [9] and the British banking sector [10] in economics; the movement of households and emergent segregation of communities within a reimplementation of Shelling's Bounded Neighbourhood Model [11] and violent crime [12] in social science; the effects of communication technologies on virtual project team performance [13] in management science; and seasonal impacts in agriculture [14] . We believe that this has been driven primarily by the ability of agent-based modelling and simulation to investigate questions regarding the causal relationships and mechanistic underpinnings to system dynamics, which traditional modelling techniques, such as differential equation-based, system dynamics (system-based models), and discrete event simulation (process-based models) cannot address [3,16] . This is supported by Epstein [4] who states that "agent-based models provide computational demonstrations that a given microspecification is in fact sufficient to generate a macrostructure of interest." In addition, the popularity of an agent-based approach has further risen through the increase in the number of software development frameworks for agent-based modelling and simulation [17] .
Computational models can provide an interpretation (be that mechanistic or dynamic) of the underlying real-world system data upon which the model is constructed [18][19][20] . As per all types of models, agent-based models are simplifications of the real-world systems upon which they are based. In the course of computational model design and development, decisions are made regarding the interpretation of real-world system data and how this should be translated into a form that can be used within a computational model. It is usually intractable (from both a computational and domain knowledge perspective) to represent every aspect of the real-world system, for all but the most simplistic of case studies [21] . In addition, an inherent, but often forgotten aspect of simulation studies is that the definitive mechanisms and values of various spatiotemporal interactions within complex dynamical systems are not currently known. This necessitates the incorporation of assumptions into the model, which adds a layer of abstraction between the computational model and the real-world system under study. It is therefore essential that the effects of these decisions on the simulation results are fully understood, as any assumptions made during the design and development of the computational model, could have critical impact on the simulation response [22,23] . However, as discussed by Alden et al. [24] , it is rare that a published model is accompanied by an in-depth description and justification of the abstraction level taken during model design, and assumptions made during model development.
Developing, running and analysing ABMs can be a challenge because even simple ABMs, such as the predator-prey model [25] , can generate complex emergent behaviours. We believe that the majority of novice modellers (including domain experts who have begun to use ABMs within their research) are not always aware of the varying aspects of uncertainty that are inherent to simulation results, especially those of stochastic dynamical systems, or indeed the underlying ABM development frameworks. This occurs in the most part, from an absence of training in software development during the years of formal education that researchers in diverse fields such as biology, economics, and the social sciences, undergo at university. This is emphasised by Wooldridge [26] who states that the development of agent-based models is a process of experimentation, but that the experimental process tends to encourage developers to forget that they are actually developing software, and thus the core activities of software engineering (e.g. requirements analysis, specification, design, verification, and testing) are often forgotten. In fact, it has previously been argued that the ABM approach has been deemed inadequate for scientific use due to the perceived lack of engineering rigour [27] . This is in part, due to the weak validation and verification that has been performed on a number of the published models over the past decade, which makes the results and predictions from these models hard to trust [28] . We believe that with increasing experience of developing ABMs of complex dynamical systems, these software engineering principles can be acquired and strengthened over time however.
As highlighted above, many ABM environments allow a user to quickly instantiate models and produce results, however, we believe that this introduces the risk of inexperienced modellers incorrectly interpreting their simulation results just as easily as they construct their models, which supports Bonabeau [5] who argues that "although ABM is technically simple, it is also conceptually deep." (see [29] for a primer on modelling and simulation-based systems engineering). This introduces risks into projects that aim to use ABMs in a predictive capacity regarding behaviours and dynamics of a real-world system. We believe that these risks fall into three main categories around: lack of detailed design and specifications; analysis of simulation results; along with calibration and uncertainty analysis. These are expanded below: No Discernible Specifications: Due to space limitations of journal articles, which do not always allow online supporting information, published models are not always supported by a functional specification or formalised conceptual model of the real-world system under study. As such the reader is sometimes left feeling uncertain whether the ABM has represented the real-world system appropriately. Without a functional specification (conceptual model), it is unclear how validation can be performed, as the specification provides context for judgements on the acceptability of interpolation or extrapolation of simulation results with respect to the real-world system of interest [30] . Similarly, we have also found that technical specifications, which link the conceptual model to the computational model are also sometimes omitted. As there are always constraints (and indeed different abstraction levels taken) regarding how a modeller chooses to implement specific realworld mechanisms, the functional and technical specifications are pivotal in setting the context in which the simulation data should be interpreted. After all, different abstractions and implementations, may produce different results.
Analytics Methods: A large number of ABM development and simulation frameworks do not provide any corresponding analytics tools, and thus require the research team to develop their own. Conversely, some of the more accessible simulation frameworks for non-Computer Scientists provide a number of methods for live analytics within the tools, whereas in reality,