diff --git a/.quarto/cites/index.json b/.quarto/cites/index.json index 8cc144d..6eab2de 100644 --- a/.quarto/cites/index.json +++ b/.quarto/cites/index.json @@ -1 +1,2 @@ -{"intro.qmd":[],"accessibility_statement.qmd":[],"delivery_and_communication.qmd":[],"summary.qmd":[],"engagement_and_scoping.qmd":[],"additional_resources.qmd":[],"quality_assurance_culture.qmd":[],"definitions_and_key_concepts.qmd":["glover2014","glover2014"],"proportionality.qmd":[],"index.qmd":[],"design.qmd":[],"references.qmd":["knuth84"],"forward.qmd":[],"analytical_lifecycle.qmd":[],"analysis.qmd":[]} +{"index.qmd":[],"analysis.qmd":[],"engagement_and_scoping.qmd":[],"quality_assurance_culture.qmd":[],"delivery_and_communication.qmd":[],"analytical_lifecycle.qmd":[],"proportionality.qmd":[],"accessibility_statement.qmd":[],"forward.qmd":[],"summary.qmd":[],"additional_resources.qmd":[],"references.qmd":["knuth84"],"design.qmd":[],"definitions_and_key_concepts.qmd":["glover2014","glover2014"],"intro.qmd":[]} + diff --git a/.quarto/idx/definitions_and_key_concepts.qmd.json b/.quarto/idx/definitions_and_key_concepts.qmd.json index d8e76de..dc2510d 100644 --- a/.quarto/idx/definitions_and_key_concepts.qmd.json +++ b/.quarto/idx/definitions_and_key_concepts.qmd.json @@ -1 +1,2 @@ -{"title":"Definitions and key concepts","markdown":{"headingText":"Definitions and key concepts","containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\n\nThis chapter sets out definitions and key concepts that are used throughout the rest of the book. \n\n## Analysis {.unnumbered}\n\nAnalysis is the collection, manipulation and interpretation of information and data for use in decision making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process. \n\nExamples of types of analysis that are frequently encountered in government are: \n\n* actuarial \n* data science \n* economic \n* financial \n* geographical \n* operational research \n* scientific, technical and engineering research \n* statistical \n* social research \n\n## Assurance {.unnumbered}\n\nAnalytical assurance is the process and set of practices to ensure that the analysis is fit for purpose. \n\n## Assurance activities {.unnumbered}\n\nAssurance activities are anything that is undertaken to validate and verify analysis. \n\nFor example: \n\n* analyst testing \n* peer review \n* analytical audits \n\n## Business critical analysis {.unnumbered}\n\nBusiness critical analysis is analysis which plays such a role in decision making that it influences significant financial and funding decisions, is necessary to the achievement of a Departmental business plan, or where an error could have a significant reputational, economic or legal impact for the public sector. \n\nThe first edition of the AQuA book described business critical models. This has been generalised to business critical analysis, as it is possible for analysis to be business critical without including a model. Some departments may continue to use the term business critical models (BCM). \n\n## Change control {.unnumbered}\n\nChange control is the set of processes followed when changes are made to a piece of analysis. For example, authorising and accepting changes, version numbering, documentation, assurance of changes. \n\n## Documentation {.unnumbered}\n\n### Specification documentation {.unnumbered}\n\nThis documentation captures the initial engagement with the commissioner and identifies the question, the context, and any boundaries of the analysis. It provides a definition of the scope and a mechanism for agreeing project constraints such as deadlines, available resources, and capturing what level of assurance is required by the commissioner. \n\n### Design documentation {.unnumbered}\n\nThis document outlines the design of the analysis, including conceptual models to illustrate the analytical problem. It forms an important tool for ensuring that the Analytical Assurer has the confidence that the Analyst can deliver quality analysis. \n\n### Assumptions log {.unnumbered}\n\nA register of assumptions, whether provided by the Commissioner or derived by the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should describe each assumption, quantify its impact and reliability and set out when it was made, why it was made, who made it and who signed it off.\n\n### Decisions log {.unnumbered}\n\nA register of decisions, whether provided by the Commissioner or derived by the analysis. Decisions logs should describe each decision and set out when it was made, why it was made, who made it and who signed it off.\n\n### Data log {.unnumbered}\n\nA register of data provided by the Commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder. \n\n### Quality assurance plan {.unnumbered}\n\nA detailed plan of what verification and validation activities should be produced identifying the key risk areas within the analysis. The plan should be thought about throughout the analytical lifecycle, from initial scoping, designing the analysis, to delivering the analysis. The areas identified in the plan can also form the basis of a log for those analysts conducting the verification and validation checks. Any additional verification and validation checks that have been performed should be recorded on the quality assurance plan. \n\n### User / technical documentation {.unnumbered}\n\nAll analysis shall have user-documentation, even if the user is only the analyst leading the analysis. This is to ensure that they have captured sufficient material to assist them if the analysis is revisited in due course. For analysis that is likely to be revisited or updated in the future, documentation should be provided to assist a future analyst and should be more comprehensive. This documentation should include a summary of the analysis including the context to the question being asked, what analytical methods were considered, what analysis was planned and why, what challenges were encountered and how they were overcome and what verification and validation steps were performed. In addition, guidance on what should be considered if the analysis is to be revisited or updated is beneficial. \n\n### Assurance statement {.unnumbered}\n\nA brief description of the analytical assurance that have been performed to assure the analysis. The statement should refer to known limitations and conditions associated with the analysis.\n\n::: {.callout-tip}\n# Example of publishing quality assurance tools\nThe Department for Energy Security and Net Zero and Department for Business and Trade have published a range of quality assurance tools and guidance to help people with Quality Assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the two departments to ensureanalysis meets the standards set out in the AQuA book and provide assurance to users of the analysis that proportionate quality assurance has been completed. \n\n:::\n\n## Materiality {.unnumbered}\n\n[Materiality](https://en.wikipedia.org/wiki/Materiality_(auditing)) is a concept or convention in auditing and accounting relating to the importance of a feature. Information is said to be material if omitting it or misstating it could influence decisions that users make. Materiality is \"an entity-specific aspect of relevance, based on size, magnitude or both\".\n\n## Multi-use models {.unnumbered}\n\nSome models, often complex and large, are used by more than one user or group of users for related but differing purposes, these are known as **multi-use models**. \n\nOften, a Steering Group is created to oversee the analysis. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model, and contain senior, ideally empowered, representatives of each major user area. \n\n## Principles of analytical quality assurance {.unnumbered}\n\nNo single piece of guidance provides a definitive assessment of whether a piece of analysis is of sufficient quality for an intended purpose. However, the following principles support commissioning and production of fit-for-purpose analysis: \n\n**Proportionate:** Quality assurance effort should be appropriate to the risk associated with the intended use of the analysis and the complexity of the analytical approach. These risks include financial, legal, operational and reputational impacts. More details can be found in chapter [3] \n\n**Assurance throughout development:** Quality assurance should be considered throughout the life cycle of the analysis and not just at the end. Effective communication is crucial when understanding the problem, designing the analytical approach, conducting the analysis and relaying the outputs. More details on the analysis life cycle can be seen in chapter [5]. \n\n**Verification and validation:** Analytical quality assurance is more than checking that the analysis is error-free and satisfies its specification (verification). It should also include checks that the analysis is appropriate, i.e. fit for the purpose for which it is being used (validation). Validation and verification are covered in more depth in chapters [5-9]. \n\n**Accept that uncertainty is inherent** in the inputs and outputs of any piece of analysis. Chapter [8] covers assurance of the analytical phase of the project, including the treatment of uncertainty . Further support can be found in the Uncertainty Toolkit for Analysts in Government (analystsuncertaintytoolkit.github.io) \n\n**Analysis with RIGOUR:** One acronym some users find helpful to consider when completing analysis is RIGOUR. This is described in the box below. \n\n::: {.callout-tip collapse=\"true\"}\n### RIGOUR\nThroughout all the stages of an analytical project, the analyst should ask questions of their own analysis. The helpful mnemonic \"RIGOUR\" may assist:\n\n* **R**epeatable\n* **I**ndependent\n* **G**rounded in reality\n* **O**bjective\n* **U**ncertainty-managed\n* **R**obust\n\n**Repeatable:** For an analytical process to be considered valid we might reasonably expect that the analysis produces the same outputs for the same inputs and constraints. Different analysts might approach the analytical problem in different ways, while methods might include randomised processes. In such cases, exact matches are not guaranteed or expected. Taking this into account, repeatability means that if an approach is repeated the results should be as expected. \n\n**Independent:** Analysis should be free of prejudice or bias. Care should be taken to balance views appropriately across all stakeholders and experts. \n\n**Grounded in reality:** Quality analysis takes the Commissioner and Analyst on a journey as views and perceptions are challenged and connections are made between the analysis and its real consequences. Connecting with reality like this guards against failing to properly grasp the context of the problem that is being analysed. \n\n**Objective:** Effective engagement and suitable challenge reduce the risk of bias and enables the Commissioner and the Analyst to be clear about the interpretation of results. \n\n**Uncertainty-managed:** Uncertainty is identified, managed and communicated throughout the analytical process. \n\n**Robust:** Analytical results are error free in the context of residual uncertainty and accepted limitations that make sure the analysis is used appropriately. \n\n:::\n\n## Quality analysis {.unnumbered}\n\nQuality analysis is analysis which is fit for the purpose(s) it was commissioned to meet. It should be accurate, have undergone appropriate assurance, be evidenced, proportionate to its impact, adequately communicated, documented and accepted by its commissioners. \n\n## Reproducible analytical pipelines {.unnumbered}\n\n[Reproducible Analytical Pipelines (RAPs)](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/) are automated analytical processes. They incorporate elements of software engineering best practice to ensure that the pipelines are reproducible, auditable, efficient, and high quality.\n\n## Roles and responsibilities {.unnumbered}\n\nThe AQuA book defines the following roles:\n\n* **Commissioner**\n* **Analyst** \n* **Assurer** \n* **Approver** \n\nSee [Roles and Responsibilities](analytical_lifecycle.qmd/#roles_and_responsibilities) for details.\n\n\n## Uncertainty {.unnumbered}\n\nThe outcome of a decision is never known perfectly in advance. For each option within analysis, a range of real outcomes is possible: the outcome is uncertain. \n\n::: {.callout-note}\n### Defining uncertainty\n\n[Wikipedia defines uncertainty](https://en.wikipedia.org/wiki/Uncertainty) as referring to [epistemic](https://en.wikipedia.org/wiki/Epistemology) situations involving imperfect or unknown information. It applies to predictions of future events, to physical measurements that are already made, or to the unknown. \n\n:::\n\nThere are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns, and unknown unknowns. The type of uncertainty will impact the analytical approach and assurance activities required. \n\nThe [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) is a tool produced by a cross government group to help assessing and communicating uncertainty.\n\n## Validation {.unnumbered}\n\nLiterally meaning to make valid, through the agreement of those judged competent to take such views. The central question that validation raises is the extent to which the right work is being engaged in, given the purpose and constraints placed upon that work. The key output from the validation process is a judgment, based on evidence, concerning the extent to which the work is \"fit for purpose\". See @glover2014 for more information.\n\n\n## Verification {.unnumbered}\n\nThe extent to which the work that has been agreed to is being done in the 'right' or 'accepted' way, given the 'art of the possible'. The key output from the verification process is a judgment, based on evidence, concerning the extent to which the agreed work has been conducted appropriately. See @glover2014 for more information.\n\n## Version control {.unnumbered}\n\nIt is important to ensure that changes that have been made to analysis can be easily seen and quality assured by the analytical assurer, and the latest version of the analysis is being used. Tools and templates can be used to support with evidencing updates and the checks completed throughout a project providing a log of changes that have occurred, why, when, and by whom. \n","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"definitions_and_key_concepts.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.554","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file +{"title":"Definitions and key concepts","markdown":{"headingText":"Definitions and key concepts","containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\n\nThis chapter sets out definitions and key concepts that are used throughout the rest of the book. \n\n## Analysis {.unnumbered}\n\nAnalysis is the collection, manipulation and interpretation of information and data for use in decision making. Analysis can vary widely between situations and many different types of analysis may be used to form the evidence base that supports the decision-making process. \n\nExamples of types of analysis that are frequently encountered in government are: \n\n* actuarial \n* data science \n* economic \n* financial \n* geographical \n* operational research \n* scientific, technical and engineering research \n* statistical \n* social research \n\n## Assurance {.unnumbered}\n\nAnalytical assurance is the process and set of practices to ensure that the analysis is fit for purpose. \n\n## Assurance activities {.unnumbered}\n\nAssurance activities are anything that is undertaken to validate and verify analysis. \n\nFor example: \n\n* analyst testing \n* peer review \n* analytical audits \n\n## Artificial Intelligence {.unnumbered}\n\nArtificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing, and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making, and language understanding. Artificial Intelligence models are a subset of [Black Box Models](#black_box_models)\n\n\n## Black box models {.unnumbered}\n\nBlack Box models internal workings are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. [Artificial Intelligence](#Artificial Intelligence) models (including [Machine Learning](#Machine Learning)) are the most common type of black box models used today. Other forms of black box models may arise in future. \n\n\n## Business critical analysis {.unnumbered}\n\nBusiness critical analysis is analysis which plays such a role in decision making that it influences significant financial and funding decisions, is necessary to the achievement of a Departmental business plan, or where an error could have a significant reputational, economic or legal impact for the public sector. \n\nThe first edition of the AQuA book described business critical models. This has been generalised to business critical analysis, as it is possible for analysis to be business critical without including a model. Some departments may continue to use the term business critical models (BCM). \n\n## Change control {.unnumbered}\n\nChange control is the set of processes followed when changes are made to a piece of analysis. For example, authorising and accepting changes, version numbering, documentation, assurance of changes. \n\n## Documentation {.unnumbered}\n\n### Specification documentation {.unnumbered}\n\nThis documentation captures the initial engagement with the commissioner and identifies the question, the context, and any boundaries of the analysis. It provides a definition of the scope and a mechanism for agreeing project constraints such as deadlines, available resources, and capturing what level of assurance is required by the commissioner. \n\n### Design documentation {.unnumbered}\n\nThis document outlines the design of the analysis, including conceptual models to illustrate the analytical problem. It forms an important tool for ensuring that the Analytical Assurer has the confidence that the Analyst can deliver quality analysis. \n\n### Assumptions log {.unnumbered}\n\nA register of assumptions, whether provided by the Commissioner or derived by the analysis, that have been risk assessed and signed off by an appropriate governance group or stakeholder. Assumption logs should describe each assumption, quantify its impact and reliability and set out when it was made, why it was made, who made it and who signed it off.\n\n### Decisions log {.unnumbered}\n\nA register of decisions, whether provided by the Commissioner or derived by the analysis. Decisions logs should describe each decision and set out when it was made, why it was made, who made it and who signed it off.\n\n### Data log {.unnumbered}\n\nA register of data provided by the Commissioner or derived by the analysis that has been risked assessed and signed-off by an appropriate governance group or stakeholder. \n\n### Quality assurance plan {.unnumbered}\n\nA detailed plan of what verification and validation activities should be produced identifying the key risk areas within the analysis. The plan should be thought about throughout the analytical lifecycle, from initial scoping, designing the analysis, to delivering the analysis. The areas identified in the plan can also form the basis of a log for those analysts conducting the verification and validation checks. Any additional verification and validation checks that have been performed should be recorded on the quality assurance plan. \n\n### User / technical documentation {.unnumbered}\n\nAll analysis shall have user-documentation, even if the user is only the analyst leading the analysis. This is to ensure that they have captured sufficient material to assist them if the analysis is revisited in due course. For analysis that is likely to be revisited or updated in the future, documentation should be provided to assist a future analyst and should be more comprehensive. This documentation should include a summary of the analysis including the context to the question being asked, what analytical methods were considered, what analysis was planned and why, what challenges were encountered and how they were overcome and what verification and validation steps were performed. In addition, guidance on what should be considered if the analysis is to be revisited or updated is beneficial. \n\n### Assurance statement {.unnumbered}\n\nA brief description of the analytical assurance that have been performed to assure the analysis. The statement should refer to known limitations and conditions associated with the analysis.\n\n::: {.callout-tip}\n# Example of publishing quality assurance tools\nThe Department for Energy Security and Net Zero and Department for Business and Trade have published a range of quality assurance tools and guidance to help people with Quality Assurance of analytical models. [Modelling Quality Assurance tools and guidance](https://www.gov.uk/government/publications/energy-security-and-net-zero-modelling-quality-assurance-qa-tools-and-guidance) are used across the two departments to ensureanalysis meets the standards set out in the AQuA book and provide assurance to users of the analysis that proportionate quality assurance has been completed. \n\n:::\n\n## Machine Learning {.unnumbered}\n\nMachine Learning (ML) is a class of [Artificial Intelligence](#Artificial Intelligence). ML models aim to learn and improve from experience without being explicitly programmed. They use algorithms that analyze data, identify patterns, and make\ndecisions with minimal human intervention. ML models are part of the class of models known as [Black Box models](#black_box_models).\n\n## Materiality {.unnumbered}\n\n[Materiality](https://en.wikipedia.org/wiki/Materiality_(auditing)) is a concept or convention in auditing and accounting relating to the importance of a feature. Information is said to be material if omitting it or misstating it could influence decisions that users make. Materiality is \"an entity-specific aspect of relevance, based on size, magnitude or both\".\n\n## Multi-use models {.unnumbered}\n\nSome models, often complex and large, are used by more than one user or group of users for related but differing purposes, these are known as **multi-use models**. \n\nOften, a Steering Group is created to oversee the analysis. This Steering Group would be chaired by the senior officer in charge of the area that maintains the model, and contain senior, ideally empowered, representatives of each major user area. \n\n## Principles of analytical quality assurance {.unnumbered}\n\nNo single piece of guidance provides a definitive assessment of whether a piece of analysis is of sufficient quality for an intended purpose. However, the following principles support commissioning and production of fit-for-purpose analysis: \n\n**Proportionate:** Quality assurance effort should be appropriate to the risk associated with the intended use of the analysis and the complexity of the analytical approach. These risks include financial, legal, operational and reputational impacts. More details can be found in chapter [3] \n\n**Assurance throughout development:** Quality assurance should be considered throughout the life cycle of the analysis and not just at the end. Effective communication is crucial when understanding the problem, designing the analytical approach, conducting the analysis and relaying the outputs. More details on the analysis life cycle can be seen in chapter [5]. \n\n**Verification and validation:** Analytical quality assurance is more than checking that the analysis is error-free and satisfies its specification (verification). It should also include checks that the analysis is appropriate, i.e. fit for the purpose for which it is being used (validation). Validation and verification are covered in more depth in chapters [5-9]. \n\n**Accept that uncertainty is inherent** in the inputs and outputs of any piece of analysis. Chapter [8] covers assurance of the analytical phase of the project, including the treatment of uncertainty . Further support can be found in the Uncertainty Toolkit for Analysts in Government (analystsuncertaintytoolkit.github.io) \n\n**Analysis with RIGOUR:** One acronym some users find helpful to consider when completing analysis is RIGOUR. This is described in the box below. \n\n::: {.callout-tip collapse=\"true\"}\n### RIGOUR\nThroughout all the stages of an analytical project, the analyst should ask questions of their own analysis. The helpful mnemonic \"RIGOUR\" may assist:\n\n* **R**epeatable\n* **I**ndependent\n* **G**rounded in reality\n* **O**bjective\n* **U**ncertainty-managed\n* **R**obust\n\n**Repeatable:** For an analytical process to be considered valid we might reasonably expect that the analysis produces the same outputs for the same inputs and constraints. Different analysts might approach the analytical problem in different ways, while methods might include randomised processes. In such cases, exact matches are not guaranteed or expected. Taking this into account, repeatability means that if an approach is repeated the results should be as expected. \n\n**Independent:** Analysis should be free of prejudice or bias. Care should be taken to balance views appropriately across all stakeholders and experts. \n\n**Grounded in reality:** Quality analysis takes the Commissioner and Analyst on a journey as views and perceptions are challenged and connections are made between the analysis and its real consequences. Connecting with reality like this guards against failing to properly grasp the context of the problem that is being analysed. \n\n**Objective:** Effective engagement and suitable challenge reduce the risk of bias and enables the Commissioner and the Analyst to be clear about the interpretation of results. \n\n**Uncertainty-managed:** Uncertainty is identified, managed and communicated throughout the analytical process. \n\n**Robust:** Analytical results are error free in the context of residual uncertainty and accepted limitations that make sure the analysis is used appropriately. \n\n:::\n\n## Quality analysis {.unnumbered}\n\nQuality analysis is analysis which is fit for the purpose(s) it was commissioned to meet. It should be accurate, have undergone appropriate assurance, be evidenced, proportionate to its impact, adequately communicated, documented and accepted by its commissioners. \n\n## Reproducible analytical pipelines {.unnumbered}\n\n[Reproducible Analytical Pipelines (RAPs)](https://analysisfunction.civilservice.gov.uk/support/reproducible-analytical-pipelines/) are automated analytical processes. They incorporate elements of software engineering best practice to ensure that the pipelines are reproducible, auditable, efficient, and high quality.\n\n## Roles and responsibilities {.unnumbered}\n\nOrganisations may have their own titles for the main functional roles involved in analysis that are set out here. \n\nEach role may be fulfilled by a team or committee of people. However, a single individual will have overall accountability (such as the chair of a committee) for each role. \n\nThe roles of Analyst and Assurer shall be distinct from each other. The analyst should carry out their own assurance but responsibility for formal assurance to the commissioner lies with the assurer. \nIn some instances, particularly for quick and / or simple analysis, an individual may deliver more than one of the roles apart from the Assurer and Analytical roles which shall be seperate from one another in all cases \n\nThe AQuA book defines the roles as: \n\n* **Commissioner** (may be known as customer) \n + Requests the analysis and sets out their requirements \n + Agrees what the analyst is going to do will satisfy the need \n + Accepts the analysis and assurance as fit for purpose \n\n* **Analyst** \n + Designs the approach, including the assurance, to meet the commissioner’s requirements \n + Agrees the approach with the commissioner \n + Carries out the analysis \n + Carries out their own assurance \n + Acts on findings from the assurer \n + Can be a group of analysts, in which case the lead analyst is responsible \n\n* **Assurer** (may be known as Analytical Assurer, Assuring Analyst, or Model Senior Responsible Owner (\"SRO\")) \n + Reviews the assurance completed by the analyst \n + Carries out any further validation and verification they may see as appropriate \n + Reports errors and areas for improvement to the analyst \n + Re-reviews as required \n + Confirms the work has been appropriately scoped, executed, validated and verified and documented to the approver \n + Can be a group of assurers. In which case the leader of the group is responsible. They must be independent from the analysts. \n\n* **Approver** (may be known as Senior Analyst or Senior Responsible Officer (“SRO”))\n + Scrutinises the work of the analyst and assurer \n + Confirms (if necessary) to the analyst, assurer and commissioner that the work has been appropriately assured \n\n## Uncertainty {.unnumbered}\n\nThe outcome of a decision is never known perfectly in advance. For each option within analysis, a range of real outcomes is possible: the outcome is uncertain. \n\n::: {.callout-note}\n### Defining uncertainty\n\n[Wikipedia defines uncertainty](https://en.wikipedia.org/wiki/Uncertainty) as referring to [epistemic](https://en.wikipedia.org/wiki/Epistemology) situations involving imperfect or unknown information. It applies to predictions of future events, to physical measurements that are already made, or to the unknown. \n\n:::\n\nThere are different types of uncertainty. A common classification divides uncertainty into known knowns, known unknowns, and unknown unknowns. The type of uncertainty will impact the analytical approach and assurance activities required. \n\nThe [Uncertainty Toolkit for Analysts in Government](https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html) is a tool produced by a cross government group to help assessing and communicating uncertainty.\n\n## Validation {.unnumbered}\n\nLiterally meaning to make valid, through the agreement of those judged competent to take such views. The central question that validation raises is the extent to which the right work is being engaged in, given the purpose and constraints placed upon that work. The key output from the validation process is a judgment, based on evidence, concerning the extent to which the work is \"fit for purpose\". See @glover2014 for more information.\n\n\n## Verification {.unnumbered}\n\nThe extent to which the work that has been agreed to is being done in the 'right' or 'accepted' way, given the 'art of the possible'. The key output from the verification process is a judgment, based on evidence, concerning the extent to which the agreed work has been conducted appropriately. See @glover2014 for more information.\n\n## Version control {.unnumbered}\n\nIt is important to ensure that changes that have been made to analysis can be easily seen and quality assured by the analytical assurer, and the latest version of the analysis is being used. Tools and templates can be used to support with evidencing updates and the checks completed throughout a project providing a log of changes that have occurred, why, when, and by whom. \n\n","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"definitions_and_key_concepts.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.553","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} + diff --git a/.quarto/idx/index.qmd.json b/.quarto/idx/index.qmd.json index c9bcfa0..bd870a8 100644 --- a/.quarto/idx/index.qmd.json +++ b/.quarto/idx/index.qmd.json @@ -1 +1 @@ -{"title":"Preface","markdown":{"headingText":"Preface","headingAttr":{"id":"","classes":["unnumbered"],"keyvalue":[]},"containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\n\nPlaceholder text","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"index.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.554","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file +{"title":"Preface","markdown":{"headingText":"Preface","headingAttr":{"id":"","classes":["unnumbered"],"keyvalue":[]},"containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\n\nPlaceholder text","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"index.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.553","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file diff --git a/.quarto/idx/intro.qmd.json b/.quarto/idx/intro.qmd.json index 9a5805c..8e11702 100644 --- a/.quarto/idx/intro.qmd.json +++ b/.quarto/idx/intro.qmd.json @@ -1 +1 @@ -{"title":"Introduction","markdown":{"headingText":"Introduction","containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\nThe last version of the **Analytical Quality Assurance (AQuA) Handbook** was published in 2015, following Sir Nicholas Macpherson's [Review of quality assurance of government models](https://www.gov.uk/government/publications/review-of-quality-assurance-of-government-models). Since then, assurance has become part of the fabric of good practice for developing evidence to support policy development, delivery and operational excellence. \n\nThe AQuA Book has made a significant contribution to the cultural change in assurance practices in government. It is about the process for assuring analytical evidence in all forms. It sets out the core framework for assuring all forms of analytical evidence. \n\nThe world of analysis has developed since we published the first edition of the AQuA Book. \n\nIncreasingly in our data driven world, insights from analysis underpin almost all policies and support operational excellence. This underlines the continuing importance of assuring our evidence. In parallel our working practices have developed. The dominant analytical tools when we wrote the last edition were spreadsheets and proprietary software. Since then we have broadened the range of methods to include open-source software, machine learning and artificial intelligence. \n\nUsers of the AQuA book have pointed out some things we did not cover in the first edition and areas where guidance was unclear or insufficient. In this edition we have added guidance on: \n\n* multi-use models - large models used for many purposes with many stakeholders; \n* assuring \"black box\"[^1] analysis, including artificial intelligence; \n* development, maintenance and continuous review; \n* working with third parties such as contractors and academic groups and, \n* publishing models. \n\nWe provide improved guidance on what [a proportionate approach to assurance means](https://best-practice-and-impact.github.io/aqua_book_revision/proportionality.html) and have made the whole guide applicable to all types of analysis.\n\nThe AQuA Book describes what you need to do but not how to do it, although it does contain many worked examples. Large organisations will have their own processes and practices covering “the how”. \n\nFor those of you who do not work in places with bespoke guidance you will find a collection of helpful resources in chapter 10.\n\nThe AQuA Book is a key supporting guide for the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). It is referred to by the [Green Book](https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-government/the-green-book-2020), the [Magenta Book](https://www.gov.uk/government/publications/the-magenta-book) and the [Finance Function](https://www.gov.uk/government/publications/government-finance-standards-page) Standards. \n\n\n## Who is the AQuA Book for?\n\nIn this edition we have tried to make our guidance relevant to anyone who commissions, uses, undertakes or assures analysis. It is about the whole process of producing analysis that is fit for purpose and not just about the checks after the analysis has been completed. \n\nWe would like to see producers and users of analysis from all backgrounds using this book, especially those producing analysis, evidence and research to support decision making in government. Our intended audience includes: \n\n* Users of analysis – helping you to get the most out of your commission; \n* Members of the government analytical professions. This might include:\n + operational researchers, statisticians and economists;\n + geographers;\n + finance professionals making spending forecasts;\n + actuaries;\n + social researchers carrying out qualitative research;\n + data scientists developing advanced analytics;\n + and anyone else carrying out analysis. \n* Senior leaders with an interest in analytical assurance. \n\n## Why should I pay attention to this guidance? \n\nHere are a few reasons why.\n\n* **Your analytical insights will be used for major decisions and operations.** You need to do your best to get them right, thus minimising the risk of being complicit in causing operational, business or reputational damage; \n* **Trust is hard to obtain but easy to lose.** A simple error that could have been prevented by assurance could lead to your and your team’s work being doubted; \n* **Prevention is better than cure.** Analysis is more likely to be right first time when you consider quality from the start. Having appropriate quality assurance in place helps to manage mistakes, handle changes to requirements and ensure appropriate re-use; \n* **Delivering quality analysis provides the confidence that is needed for transparency and public openness;** \n* **Assurance is required for audit purposes**[^2]; and \n* **Professional pride in your work.** \n\n## How to use this book\n\nThe first four chapters of this book cover definitions and overarching themes, whilst the second half of the book goes into more detail on the assurance activities needed at each stage of the assurance life cycle.\n\nThis guidance uses the following terms to indicate whether recommendations are mandatory or advisory. \n\nThe terms are: \n\n* **‘shall’** denotes a requirement, a mandatory element, which applies in all circumstances, at all times \n* **‘should’** denotes a recommendation, an advisory element, to be met on a ‘comply or explain’ basis \n* **‘may’** denotes approval \n* **‘might’** denotes a possibility \n* **‘can’** denotes both capability and possibility \n* **is/are** is used for a description \n\nThese are the same terms as those in the [UK Government functional standards](https://www.gov.uk/government/collections/functional-standards). \n\n# Acknowledgements {.unnumbered}\n\nThe AQuA Book is the work of many authors from across the Government Analysis Function. The original version of the book was compiled by the Quality Assurance Working Group set up after Sir Nicholas Macpherson's review of modelling in government. This revised edition of the book was produced by a task and finish group drawn from across the Analysis Function community. We would like to thank everybody who has given of their time and expertise to produce the revised edition.\n\n\n[^1]: [Black box](https://en.wikipedia.org/wiki/Black_box): system which can be viewed in terms of its inputs and outputs (or transfer characteristics), without any knowledge of its internal workings.\n[^2]: [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money ), Annex 4.2 Use of models ","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"intro.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.554","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file +{"title":"Introduction","markdown":{"headingText":"Introduction","containsRefs":false,"markdown":"::: {.callout-important}\nThis version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs. \n\nThe draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised. \n:::\n\nThe last version of the **Analytical Quality Assurance (AQuA) Handbook** was published in 2015, following Sir Nicholas Macpherson's [Review of quality assurance of government models](https://www.gov.uk/government/publications/review-of-quality-assurance-of-government-models). Since then, assurance has become part of the fabric of good practice for developing evidence to support policy development, delivery and operational excellence. \n\nThe AQuA Book has made a significant contribution to the cultural change in assurance practices in government. It is about the process for assuring analytical evidence in all forms. It sets out the core framework for assuring all forms of analytical evidence. \n\nThe world of analysis has developed since we published the first edition of the AQuA Book. \n\nIncreasingly in our data driven world, insights from analysis underpin almost all policies and support operational excellence. This underlines the continuing importance of assuring our evidence. In parallel our working practices have developed. The dominant analytical tools when we wrote the last edition were spreadsheets and proprietary software. Since then we have broadened the range of methods to include open-source software, machine learning and artificial intelligence. \n\nUsers of the AQuA book have pointed out some things we did not cover in the first edition and areas where guidance was unclear or insufficient. In this edition we have added guidance on: \n\n* multi-use models - large models used for many purposes with many stakeholders; \n* assuring \"black box\"[^1] analysis, including artificial intelligence; \n* development, maintenance and continuous review; \n* working with third parties such as contractors and academic groups and, \n* publishing models. \n\nWe provide improved guidance on what [a proportionate approach to assurance means](https://best-practice-and-impact.github.io/aqua_book_revision/proportionality.html) and have made the whole guide applicable to all types of analysis.\n\nThe AQuA Book describes what you need to do but not how to do it, although it does contain many worked examples. Large organisations will have their own processes and practices covering “the how”. \n\nFor those of you who do not work in places with bespoke guidance you will find a collection of helpful resources in chapter 10.\n\nThe AQuA Book is a key supporting guide for the [Analysis Function Standard](https://www.gov.uk/government/publications/government-analysis-functional-standard--2). It is referred to by the [Green Book](https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-government/the-green-book-2020), the [Magenta Book](https://www.gov.uk/government/publications/the-magenta-book) and the [Finance Function](https://www.gov.uk/government/publications/government-finance-standards-page) Standards. \n\n\n## Who is the AQuA Book for?\n\nIn this edition we have tried to make our guidance relevant to anyone who commissions, uses, undertakes or assures analysis. It is about the whole process of producing analysis that is fit for purpose and not just about the checks after the analysis has been completed. \n\nWe would like to see producers and users of analysis from all backgrounds using this book, especially those producing analysis, evidence and research to support decision making in government. Our intended audience includes: \n\n* Users of analysis – helping you to get the most out of your commission; \n* Members of the government analytical professions. This might include:\n + operational researchers, statisticians and economists;\n + geographers;\n + finance professionals making spending forecasts;\n + actuaries;\n + social researchers carrying out qualitative research;\n + data scientists developing advanced analytics;\n + and anyone else carrying out analysis. \n* Senior leaders with an interest in analytical assurance. \n\n## Why should I pay attention to this guidance? \n\nHere are a few reasons why.\n\n* **Your analytical insights will be used for major decisions and operations.** You need to do your best to get them right, thus minimising the risk of being complicit in causing operational, business or reputational damage; \n* **Trust is hard to obtain but easy to lose.** A simple error that could have been prevented by assurance could lead to your and your team’s work being doubted; \n* **Prevention is better than cure.** Analysis is more likely to be right first time when you consider quality from the start. Having appropriate quality assurance in place helps to manage mistakes, handle changes to requirements and ensure appropriate re-use; \n* **Delivering quality analysis provides the confidence that is needed for transparency and public openness;** \n* **Assurance is required for audit purposes**[^2]; and \n* **Professional pride in your work.** \n\n## How to use this book\n\nThe first four chapters of this book cover definitions and overarching themes, whilst the second half of the book goes into more detail on the assurance activities needed at each stage of the assurance life cycle.\n\nThis guidance uses the following terms to indicate whether recommendations are mandatory or advisory. \n\nThe terms are: \n\n* **‘shall’** denotes a requirement, a mandatory element, which applies in all circumstances, at all times \n* **‘should’** denotes a recommendation, an advisory element, to be met on a ‘comply or explain’ basis \n* **‘may’** denotes approval \n* **‘might’** denotes a possibility \n* **‘can’** denotes both capability and possibility \n* **is/are** is used for a description \n\nThese are the same terms as those in the [UK Government functional standards](https://www.gov.uk/government/collections/functional-standards). \n\n# Acknowledgements {.unnumbered}\n\nThe AQuA Book is the work of many authors from across the Government Analysis Function. The original version of the book was compiled by the Quality Assurance Working Group set up after Sir Nicholas Macpherson's review of modelling in government. This revised edition of the book was produced by a task and finish group drawn from across the Analysis Function community. We would like to thank everybody who has given of their time and expertise to produce the revised edition.\n\n\n[^1]: [Black box](https://en.wikipedia.org/wiki/Black_box): system which can be viewed in terms of its inputs and outputs (or transfer characteristics), without any knowledge of its internal workings.\n[^2]: [Managing Public Money](https://www.gov.uk/government/publications/managing-public-money ), Annex 4.2 Use of models ","srcMarkdownNoYaml":""},"formats":{"html":{"identifier":{"display-name":"HTML","target-format":"html","base-format":"html"},"execute":{"fig-width":7,"fig-height":5,"fig-format":"retina","fig-dpi":96,"df-print":"default","error":false,"eval":true,"cache":null,"freeze":false,"echo":true,"output":true,"warning":true,"include":true,"keep-md":false,"keep-ipynb":false,"ipynb":null,"enabled":null,"daemon":null,"daemon-restart":false,"debug":false,"ipynb-filters":[],"ipynb-shell-interactivity":null,"plotly-connected":true,"engine":"markdown"},"render":{"keep-tex":false,"keep-typ":false,"keep-source":false,"keep-hidden":false,"prefer-html":false,"output-divs":true,"output-ext":"html","fig-align":"default","fig-pos":null,"fig-env":null,"code-fold":"none","code-overflow":"scroll","code-link":false,"code-line-numbers":false,"code-tools":false,"tbl-colwidths":"auto","merge-includes":true,"inline-includes":false,"preserve-yaml":false,"latex-auto-mk":true,"latex-auto-install":true,"latex-clean":true,"latex-min-runs":1,"latex-max-runs":10,"latex-makeindex":"makeindex","latex-makeindex-opts":[],"latex-tlmgr-opts":[],"latex-input-paths":[],"latex-output-dir":null,"link-external-icon":false,"link-external-newwindow":false,"self-contained-math":false,"format-resources":[],"notebook-links":true},"pandoc":{"standalone":true,"wrap":"none","default-image-extension":"png","to":"html","output-file":"intro.html"},"language":{"toc-title-document":"Table of contents","toc-title-website":"On this page","related-formats-title":"Other Formats","related-notebooks-title":"Notebooks","source-notebooks-prefix":"Source","other-links-title":"Other Links","code-links-title":"Code Links","launch-dev-container-title":"Launch Dev Container","launch-binder-title":"Launch Binder","article-notebook-label":"Article Notebook","notebook-preview-download":"Download Notebook","notebook-preview-download-src":"Download Source","notebook-preview-back":"Back to Article","manuscript-meca-bundle":"MECA Bundle","section-title-abstract":"Abstract","section-title-appendices":"Appendices","section-title-footnotes":"Footnotes","section-title-references":"References","section-title-reuse":"Reuse","section-title-copyright":"Copyright","section-title-citation":"Citation","appendix-attribution-cite-as":"For attribution, please cite this work as:","appendix-attribution-bibtex":"BibTeX citation:","title-block-author-single":"Author","title-block-author-plural":"Authors","title-block-affiliation-single":"Affiliation","title-block-affiliation-plural":"Affiliations","title-block-published":"Published","title-block-modified":"Modified","title-block-keywords":"Keywords","callout-tip-title":"Tip","callout-note-title":"Note","callout-warning-title":"Warning","callout-important-title":"Important","callout-caution-title":"Caution","code-summary":"Code","code-tools-menu-caption":"Code","code-tools-show-all-code":"Show All Code","code-tools-hide-all-code":"Hide All Code","code-tools-view-source":"View Source","code-tools-source-code":"Source Code","tools-share":"Share","tools-download":"Download","code-line":"Line","code-lines":"Lines","copy-button-tooltip":"Copy to Clipboard","copy-button-tooltip-success":"Copied!","repo-action-links-edit":"Edit this page","repo-action-links-source":"View source","repo-action-links-issue":"Report an issue","back-to-top":"Back to top","search-no-results-text":"No results","search-matching-documents-text":"matching documents","search-copy-link-title":"Copy link to search","search-hide-matches-text":"Hide additional matches","search-more-match-text":"more match in this document","search-more-matches-text":"more matches in this document","search-clear-button-title":"Clear","search-text-placeholder":"","search-detached-cancel-button-title":"Cancel","search-submit-button-title":"Submit","search-label":"Search","toggle-section":"Toggle section","toggle-sidebar":"Toggle sidebar navigation","toggle-dark-mode":"Toggle dark mode","toggle-reader-mode":"Toggle reader mode","toggle-navigation":"Toggle navigation","crossref-fig-title":"Figure","crossref-tbl-title":"Table","crossref-lst-title":"Listing","crossref-thm-title":"Theorem","crossref-lem-title":"Lemma","crossref-cor-title":"Corollary","crossref-prp-title":"Proposition","crossref-cnj-title":"Conjecture","crossref-def-title":"Definition","crossref-exm-title":"Example","crossref-exr-title":"Exercise","crossref-ch-prefix":"Chapter","crossref-apx-prefix":"Appendix","crossref-sec-prefix":"Section","crossref-eq-prefix":"Equation","crossref-lof-title":"List of Figures","crossref-lot-title":"List of Tables","crossref-lol-title":"List of Listings","environment-proof-title":"Proof","environment-remark-title":"Remark","environment-solution-title":"Solution","listing-page-order-by":"Order By","listing-page-order-by-default":"Default","listing-page-order-by-date-asc":"Oldest","listing-page-order-by-date-desc":"Newest","listing-page-order-by-number-desc":"High to Low","listing-page-order-by-number-asc":"Low to High","listing-page-field-date":"Date","listing-page-field-title":"Title","listing-page-field-description":"Description","listing-page-field-author":"Author","listing-page-field-filename":"File Name","listing-page-field-filemodified":"Modified","listing-page-field-subtitle":"Subtitle","listing-page-field-readingtime":"Reading Time","listing-page-field-wordcount":"Word Count","listing-page-field-categories":"Categories","listing-page-minutes-compact":"{0} min","listing-page-category-all":"All","listing-page-no-matches":"No matching items","listing-page-words":"{0} words"},"metadata":{"lang":"en","fig-responsive":true,"quarto-version":"1.4.553","bibliography":["references.bib"],"theme":"flatly"},"extensions":{"book":{"multiFile":true}}}},"projectFormats":["html"]} \ No newline at end of file diff --git a/.quarto/xref/a0b88893 b/.quarto/xref/a0b88893 index 33c3d75..ce74976 100644 --- a/.quarto/xref/a0b88893 +++ b/.quarto/xref/a0b88893 @@ -1 +1 @@ -{"headings":["analysis","assurance","assurance-activities","business-critical-analysis","change-control","documentation","specification-documentation","design-documentation","assumptions-log","decisions-log","data-log","quality-assurance-plan","user-technical-documentation","assurance-statement","materiality","multi-use-models","principles-of-analytical-quality-assurance","quality-analysis","reproducible-analytical-pipelines","roles-and-responsibilities","uncertainty","validation","verification","version-control"],"options":{"chapters":true},"entries":[]} \ No newline at end of file +{"options":{"chapters":true},"entries":[],"headings":["analysis","assurance","assurance-activities","artificial-intelligence","black-box-models","business-critical-analysis","change-control","documentation","specification-documentation","design-documentation","assumptions-log","decisions-log","data-log","quality-assurance-plan","user-technical-documentation","assurance-statement","machine-learning","materiality","multi-use-models","principles-of-analytical-quality-assurance","quality-analysis","reproducible-analytical-pipelines","roles-and-responsibilities","uncertainty","validation-.unnumbered","verification","version-control"]} diff --git a/definitions_and_key_concepts.qmd b/definitions_and_key_concepts.qmd index 51ce23f..7907d34 100644 --- a/definitions_and_key_concepts.qmd +++ b/definitions_and_key_concepts.qmd @@ -38,6 +38,16 @@ For example: * peer review * analytical audits +## Artificial Intelligence {.unnumbered} + +Artificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing, and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making, and language understanding. Artificial Intelligence models are a subset of [black box models](#black_box_models) + + +## Black box models {.unnumbered} + +Black box models internal workings are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. [Artificial Intelligence](#Artificial Intelligence) models (including [Machine Learning](#Machine Learning)) are the most common type of black box models used today. Other forms of black box models may arise in future. + + ## Business critical analysis {.unnumbered} Business critical analysis is analysis which plays such a role in decision making that it influences significant financial and funding decisions, is necessary to the achievement of a Departmental business plan, or where an error could have a significant reputational, economic or legal impact for the public sector. @@ -88,6 +98,11 @@ The Department for Energy Security and Net Zero and Department for Business and ::: +## Machine Learning {.unnumbered} + +Machine Learning (ML) is a class of [Artificial Intelligence](#Artificial Intelligence). ML models aim to learn and improve from experience without being explicitly programmed. They use algorithms that analyze data, identify patterns, and make +decisions with minimal human intervention. ML models are part of the class of models known as [Black Box models](#black_box_models). + ## Materiality {.unnumbered} [Materiality](https://en.wikipedia.org/wiki/Materiality_(auditing)) is a concept or convention in auditing and accounting relating to the importance of a feature. Information is said to be material if omitting it or misstating it could influence decisions that users make. Materiality is "an entity-specific aspect of relevance, based on size, magnitude or both". @@ -184,3 +199,4 @@ The extent to which the work that has been agreed to is being done in the 'right ## Version control {.unnumbered} It is important to ensure that changes that have been made to analysis can be easily seen and quality assured by the analytical assurer, and the latest version of the analysis is being used. Tools and templates can be used to support with evidencing updates and the checks completed throughout a project providing a log of changes that have occurred, why, when, and by whom. + diff --git a/docs/definitions_and_key_concepts.html b/docs/definitions_and_key_concepts.html index 0dd9ac9..be48c08 100644 --- a/docs/definitions_and_key_concepts.html +++ b/docs/definitions_and_key_concepts.html @@ -2,7 +2,7 @@ - + @@ -223,6 +223,8 @@

Table of contents

  • Analysis
  • Assurance
  • Assurance activities
  • +
  • Artificial Intelligence
  • +
  • Black box models
  • Business critical analysis
  • Change control
  • Documentation @@ -236,6 +238,7 @@

    Table of contents

  • User / technical documentation
  • Assurance statement
  • +
  • Machine Learning
  • Materiality
  • Multi-use models
  • Principles of analytical quality assurance
  • @@ -322,6 +325,14 @@

    Assurance
  • analytical audits
  • +
    +

    Artificial Intelligence

    +

    Artificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing, and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making, and language understanding. Artificial Intelligence models are a subset of Black Box Models

    +
    +
    +

    Black box models

    +

    Black Box models internal workings are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. Artificial Intelligence models (including Machine Learning) are the most common type of black box models used today. Other forms of black box models may arise in future.

    +

    Business critical analysis

    Business critical analysis is analysis which plays such a role in decision making that it influences significant financial and funding decisions, is necessary to the achievement of a Departmental business plan, or where an error could have a significant reputational, economic or legal impact for the public sector.

    @@ -379,6 +390,10 @@

    Assurance s

    +
    +

    Machine Learning

    +

    Machine Learning (ML) is a class of Artificial Intelligence. ML models aim to learn and improve from experience without being explicitly programmed. They use algorithms that analyze data, identify patterns, and make decisions with minimal human intervention. ML models are part of the class of models known as Black Box models.

    +

    Materiality

    Materiality is a concept or convention in auditing and accounting relating to the importance of a feature. Information is said to be material if omitting it or misstating it could influence decisions that users make. Materiality is “an entity-specific aspect of relevance, based on size, magnitude or both”.

    diff --git a/docs/search.json b/docs/search.json index 6c32516..fec4899 100644 --- a/docs/search.json +++ b/docs/search.json @@ -678,5 +678,35 @@ "crumbs": [ "References" ] + }, + { + "objectID": "definitions_and_key_concepts.html#artificial-intelligence", + "href": "definitions_and_key_concepts.html#artificial-intelligence", + "title": "2  Definitions and key concepts", + "section": "Artificial Intelligence", + "text": "Artificial Intelligence\nArtificial intelligence (AI) attempts to simulate human intelligence using techniques and methods such as machine learning, natural language processing, and robotics. AI aims to perform tasks that typically require human intelligence, such as problem-solving, decision-making, and language understanding. Artificial Intelligence models are a subset of Black Box Models", + "crumbs": [ + "2  Definitions and key concepts" + ] + }, + { + "objectID": "definitions_and_key_concepts.html#machine-learning", + "href": "definitions_and_key_concepts.html#machine-learning", + "title": "2  Definitions and key concepts", + "section": "Machine Learning", + "text": "Machine Learning\nMachine Learning (ML) is a class of Artificial Intelligence. ML models aim to learn and improve from experience without being explicitly programmed. They use algorithms that analyze data, identify patterns, and make decisions with minimal human intervention. ML models are part of the class of models known as Black Box models.", + "crumbs": [ + "2  Definitions and key concepts" + ] + }, + { + "objectID": "definitions_and_key_concepts.html#black-box-models", + "href": "definitions_and_key_concepts.html#black-box-models", + "title": "2  Definitions and key concepts", + "section": "Black box models", + "text": "Black box models\nBlack Box models internal workings are not visible or easily understood. These models take input and produce output without providing clarity about the process used to arrive at the output. Artificial Intelligence models (including Machine Learning) are the most common type of black box models used today. Other forms of black box models may arise in future.", + "crumbs": [ + "2  Definitions and key concepts" + ] } ] \ No newline at end of file