{"id":508,"date":"2017-05-22T07:41:41","date_gmt":"2017-05-22T07:41:41","guid":{"rendered":"http:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/?p=508"},"modified":"2017-05-22T07:41:41","modified_gmt":"2017-05-22T07:41:41","slug":"energy-systems-modelling-models-and-the-real-world","status":"publish","type":"post","link":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/2017\/05\/22\/energy-systems-modelling-models-and-the-real-world\/","title":{"rendered":"Energy systems modelling: models and the real world"},"content":{"rendered":"

By\u00a0Dr. Chris Dent<\/a>, School of Mathematics,\u00a0University of Edinburgh<\/p>\n

Introduction by Dr Mark Winskel, University of Edinburgh<\/em><\/p>\n

Systems modelling has played an important role in the development of the Scottish Government\u2019s draft Climate Change Plan and Energy Strategy, and is seen by the policymakers in Scotland as key enabler of its vision for policy integration. The kind of modelling adopted in Scotland has been used by many national governments, advisory bodies and international agencies in formulating response to energy policy challenges, especially climate change. Despite their widespread use, however, there are some basic analytic and pragmatic concerns, as Dr Chris Dent explores.<\/em><\/p>\n

\n

This article will discuss key methodological issues in energy systems modelling, with a little help from some well-known experts…<\/p>\n

George Box: \u201cAll models are wrong, but some are useful\u201d<\/strong><\/p>\n

This is probably the most famous quotation about statistics (which might demonstrate that there is no such thing as a famous quotation about statistics). Box meant by this simply that that no model represents the world perfectly, or provides outputs which are the whole truth about the world \u2013 one needs therefore to identify carefully the purposes for which a given model is useful and those for which it is not.<\/p>\n

My overall thesis is thus that it is necessary when carrying out an applied modelling study to provide a clear statement about what the model is claimed to tell us regarding the real world, along with a logical argument to support that statement.<\/p>\n

Ben Hobbs[1]<\/a>: \u201cSimple models for insight, complex models for quantification\u201d<\/strong><\/p>\n

This statement, from a prominent researcher on the boundary of energy systems and operational research, summarises well a key issue in quantitative modelling. When it is necessary to take a decision involving a complex real world system, in order for a modelling study to have credibility for decision support, the model must sometimes be correspondingly complex \u2013 for instance detail of power generation operation or investment decisions depends on detail of the network as well as on the generation options alone. However, it can then be hard to understand how inputs and modelling assumptions feed through into model outputs.<\/p>\n

On the other hand, while it is clearly easier to understand what features of data and assumptions are driving the results of simpler models, one must ensure that insights gained about the model are relevant to the real world rather than only being relevant to the model. This brings to mind the quote attributed (possibly incorrectly[2]<\/a>) to Einstein that \u201cEverything should be made as simple as possible, but not simpler\u201d.<\/p>\n

It remains the case, however, that for any model (simple or complex) a key part of making it useful (in Box\u2019s term) is an honest assessment of its capabilities.<\/p>\n

Niels Bohr: \u201cPrediction is difficult, especially about the future\u201d<\/strong><\/p>\n

This, to me, leads to several key issues.<\/p>\n

    \n
  1. The nature of applied modelling. I would argue that any applied modelling study must be aiming to predict some feature of the real world \u2013 otherwise how can the study be called applied, or be of any use in supporting decision making? The word \u2018predict\u2019 is used here in a very general sense \u2013 for instance it might be about understanding at a qualitative level what the key issues are in determining a given real world outcome, or about reconstructing historic events at times or places where observations are not available.<\/li>\n
  2. It is also important to recognise the nature of scientific prediction. It is not, in general, sufficient to make point predictions without quantification of uncertainty. There is a very great difference between a point prediction (i.e. the most likely value) of a future observation being 10, and being confident that it will be between 9 and 11 \u2013 and on the other hand making a point prediction of 10 but judging that the outcome could be anywhere between 1 and 100. Certainly any decisions based on that central prediction could be very different, depending on which of these quantifications of uncertainty is made.<\/li>\n
  3. Issues in quantifying and managing uncertainty are very different on operational and planning timescales. For instance, when operating an engineering system one accrues relevant data for model calibration, or constructing statistics of forecast error, relatively rapidly. On longer planning timescales (where one might be looking years or decades ahead) many more uncertainties enter the picture including imperfect knowledge of system background and technology development, and one accrues relevant historic data much more slowly \u2013 indeed for many purposes one might not have directly relevant historic data at all, and then a substantial degree of expert judgment is inevitably required to determine model inputs and quantify uncertainty in real-world predictions.<\/li>\n
  4. As a consequence of the uncertain relationship between model outputs and the world, conventional ideas of single optimal solutions do not apply when thinking about the real world on planning timescales \u2013 the equivalent is being confident that a policy or decision will perform well in the real world, along with a systematic argument to support this.<\/li>\n<\/ol>\n

    Darth Vader: \u201cDon\u2019t be too proud of this technological terror you\u2019ve constructed\u201d<\/strong><\/p>\n

    This is a good reminder that as modellers we should take a realistic and honest view of the strengths and weaknesses of our work, and certainly not take an unrealistic salesperson\u2019s perspective. In the original Star Wars film, Tarkin and Motti did not recognise the weaknesses of their Death Star, and the consequences for them were not good.<\/p>\n

    Specific issues to guard against in presenting one\u2019s own work include justifying an approach as the best available or an industry standard (this does not necessarily mean that it is fit for the purpose at hand), and choosing performance metrics which unduly favour the chosen approach over others.<\/p>\n

    Juvenal: \u201cQuis custodiet ipsos custodies\u201d (\u201cWho will watch the watchmen?\u201d)<\/strong><\/p>\n

    The Roman poet Juvenal probably was not thinking of peer review processes when writing this line in his Satires \u2013 but nevertheless this quotation is very apt. The reliability of any peer review process is strongly dependent on how well the reviewers are selected and managed, and on the way in which the authors of a document act upon their comments. In addition, even the most appropriate reviewers will occasionally miss something or otherwise make a mistake.<\/p>\n

    There are specific causes for concern in circumstances where the authors of a document or study (or their organisation) select the reviewers themselves (and sometimes pay them) \u2013 great integrity is then required on the part of all concerned, including willingness on the part of the organisation commissioning the review to regard a correctly negative review as being to their benefit. Another circumstance in which peer review can become unreliable is where a community is quite closed, and does not seek relevant expertise from other communities which have relevant knowledge.<\/p>\n

    As an individual looking to understand others\u2019 modelling studies, the safe course of action is to take one\u2019s own judgment, rather than trusting entirely the authors or reviewers \u2013 of course some authors or forms of publication will be more worthy of trust than others. One must also understand that over time knowledge develops, so what was deemed best practice some years ago might not remain thus.<\/p>\n

    In order for people to take their own judgment, a sufficient level of detail must be supplied to them by a study\u2019s authors. This can be a serious issue in public policy debates, where sometimes quite limited detail of study methodology is placed in the public domain \u2013 it is then very hard for the modelling study to play a proper role in debate, as no one external to the organisation designing the study can critique it, and use of its conclusions by anyone external relies on a very high degree of trust.<\/p>\n

    Brian Clough[3]<\/a>: \u201cfalse confidence\u201d and \u201ctrue confidence\u201d<\/strong><\/p>\n

    This great football manager talked about taking away his players\u2019 false confidence and giving them true confidence based on a proper understanding of their abilities. Unfortunately, many studies at some point implicitly rely on an unsupported assertion that their results can say something meaningful about the world, in which case confidence in their use may be false. True confidence can be gained by having logical arguments about what the study can say regarding the real world, as discussed above.<\/p>\n

    Conclusion<\/strong><\/p>\n

    This article advocates an approach to energy systems modelling which is quite different from much current practice. The key, as noted above, is that where models are used in decision support it is necessary to have logical arguments regarding what the modelling says about the real world.<\/p>\n

    A key part of this is deciding an appropriate level of detail in a given modelling study. Additional detail is typically added with an aim of learning more about the world, and there may well be a minimal level of detail below which the model cannot say anything about the real system. On the other hand, as the complexity of a model increases, it can become harder to assess what it is saying about the world \u2013 for instance quantifying uncertainty in predictions typically requires multiple model runs, with greater complexity implying a need for more runs but also making each run take longer.<\/p>\n

    This necessary comprehensive analysis of model-real world relationship requires significant additional resource, and may also require quite different expertise from that needed to build a system model. In addition, it is necessary to confront explicitly the subjective judgments involved in determining model structure and model inputs. All this should be thought about right from the start of a modelling study, with the system modelling and uncertainty quantification being designed together, in order to maximise learning about the real world. Thus this analysis is not easy, but carrying it out will mean decisions can be based on the state of knowledge about the world, rather than about the model \u2013 and thus that better decisions based on modelling evidence are made.<\/p>\n

    \n

    The author thanks colleagues (H. Du, B. Hobbs, A. Wilson and M. Winskel) who have commented on drafts of this article, and acknowledges discussions with many further colleagues which have helped shape the thinking which it contains. The views expressed are the author\u2019s own, and should not be assumed to represent those of any other person or organisation (including the Galactic Empire).<\/p>\n

    [1]<\/a> This is not a direct quote, but rather a combination of ideas expressed by its author.<\/p>\n

    [2]<\/a> See http:\/\/quoteinvestigator.com\/2011\/05\/13\/einstein-simple\/<\/a> for a quite scarily exhaustive investigation of this point.<\/p>\n

    [3]<\/a> Similarly to Ben Hobbs, this is not a direct quotation.<\/p>\n

     <\/p>\n","protected":false},"excerpt":{"rendered":"

    By\u00a0Dr. Chris Dent, School of Mathematics,\u00a0University of Edinburgh Introduction by Dr Mark Winskel, University of Edinburgh Systems modelling has played an important role in the development of the Scottish Government\u2019s draft Climate Change Plan and Energy Strategy, and is seen … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":114,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[211],"tags":[],"acf":[],"_links":{"self":[{"href":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/wp-json\/wp\/v2\/posts\/508"}],"collection":[{"href":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/wp-json\/wp\/v2\/users\/114"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/wp-json\/wp\/v2\/comments?post=508"}],"version-history":[{"count":1,"href":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/wp-json\/wp\/v2\/posts\/508\/revisions"}],"predecessor-version":[{"id":509,"href":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/wp-json\/wp\/v2\/posts\/508\/revisions\/509"}],"wp:attachment":[{"href":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/wp-json\/wp\/v2\/media?parent=508"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/wp-json\/wp\/v2\/categories?post=508"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.sps.ed.ac.uk\/global-environment-society-academy\/wp-json\/wp\/v2\/tags?post=508"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}