Secure Enterprise AI Systems: An Architecture-First Security Strategy for Enterprise Leaders

What if the issue with AI isn’t that it’s out to take our jobs but instead distort our reality? Despite the merits of AI, some organisations are reluctant to implement AI as the risk of data loss outweighs the potential benefits. What if the real risk of AI is due to improper design, implementation and maintenance? Many organisations implement new systems with pace in a “set it and forget it” manner, AI systems being no different, security as an afterthought taken seriously post-breach. Without architecting clear trust boundaries and controls, AI becomes a significant threat vector for stealing sensitive data, personally identifiable information (PII) and intellectual property (IP), in addition to manipulating the knowledge base of AI systems. Before deploying such an influential tool, it is in public interest to architect a robust solution for architecting clear trust boundaries and data controls to mitigate risks related to data loss and uphold trust within Enterprise AI. This article will discuss Enterprise AI systems, outlining the risks, how to maintain AI systems deployed within enterprises and why we must design and fortify architecture ahead of going live. 

Definitions & Concepts

• Large Language Model (LLM) - a machine learning model that can comprehend and generate human language, trained on large datasets (CloudFlare). Functions that LLMs can undertake are varied, including generating text, summarising long documents and generating code to name a few.

• AI system - a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments (EU Artificial Intelligence Act).

• Retrieval Augmented Generation (RAG) - the process of optimising the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response (AWS)

• Fine Tuning - the further training of a pre-trained LLM on a task-specific dataset, a transfer learning process (Google).

• Prompt Engineering - the practice of designing inputs for AI tools that will produce optimal outputs (McKinsey).

• Architecture-First Approach - before starting implementation (coding) of the software architectural design must be completed. The software development team must have defined a solution that meets all the technical and operational requirements (Kibru et al, 2020).

• A trust boundary is a logical construct used to demarcate areas of the system with different levels of trust such as: Public Internet vs. private network, Service APIs, Separate operating system processes, Virtual machines, User vs. kernel memory space, Containers (Microsoft).

How Enterprise AI Systems Work

AI remains a growing technology trend with individuals and organisations integrating AI systems into their day to day life. In 2024, Deloitte reported that 36% of the UK population had used Generative AI (around 18 million people between 16-75), additionally, the use of AI at work increased to around 14% of population. The adoption of AI individually and by corporations makes trust in AI a matter of public safety. 

At a high level, AI systems use a large body of data and models to generate decisions, output text and make recommendations, amongst many other functions. Data quality determines the success of the AI model outputting reliable, trustworthy information, models must be monitored and updated to ensure accuracy over time (IBM). If the data within a model is flawed, it is likely to produce biased responses, outdated information or AI hallucinations, all to the detriment of AI users. For example, self representing litigants that rely on AI for court cases leading to inaccurate citations and time delays in cases (Thomson Reuters). It is important that AI models data sources are constantly refined and updated to ensure quality responses. 

The methods for refining AI models act as a control for ensuring that the models maintain relevance, so outputs can be relied upon for decision-making. There are three methods used for refining AI models, each with varying levels of complexity, costs and time investment, shown in the diagrams below. 

Image Source: Daily Dose of DS - Avi Chawla

Common AI Failure Points 

AI systems present many severe risks that I will divide into two categories: decision integrity risk and enterprise compromise risk. Decision Integrity Risk compromise happens when decisions are made based on inaccurate, flawed or unethical data (IBAC). This presents a risk to the public because the spread of misinformation can cause panic, poor decision making or even radicalisation as we saw during the covid 19 pandemic. Regenerative AI is typically more convincing than the run of the mill misinformation campaign as it is possible to create convincing videos and recreation of voices. Alternatively, Enterprise Compromise Risk is about the probability of attackers penetrating corporate networks using a myriad of attack pathways. AI systems, by design, are a very attractive target as they hold huge volumes of data which could include personally identifiable information (PII) and intellectual property.

There is dual potential gain for potential attackers, manipulating data and stealing data, AI models are reliant on data to function. NIST outlines the need to maintain balance in the risk-reward scale in the NIST AI Risk Management Framework, through the use of AI systems we need to ensure that trustworthiness is upheld. To support this, the NIST AI RMF Playbook provides support for benchmarking organisations AI security posture management and developing a roadmap for improving maturity.  

Why Enterprise AI Security Must Start with Architecture 

The risk potential for AI systems is high, so necessitates careful architectural design choices to ensure that decision integrity and enterprise compromise risks are mitigated. Embedding security from the design phases enables the mapping of trust boundaries, authentication checkpoints and identification of crown jewels i.e. data (OWASP). This process of threat modelling in turns allows for intentional and granular design of controls for mitigating risks and gauging the impact if risks crystallise. Why is this important? I cannot count the number of times I have encountered attempts to retrospectively fit security controls, at best, it's expensive and difficult and at worst, a breach has already occurred. The best time to address security is always pre-production!

What Successful AI Adoption Requires

AI is here to stay, in February 2026 OpenAI reported a 12.5% increase in users since October making their total weekly users 900 million. To prepare your workforce for AI, consider the following:

Invest in AI skills 

The UK government have a  identified AI as a key skills gap and have invested in programmes  with a view to generate £400bn for the UK by 2030, offering support to UK businesses for addressing this skills gap. Organisations should be including AI skills development as part of professional development as the jobs market may be unable to meet the demands of businesses. Investing in the skills and development of the existing workforce can have a two-fold benefit, firstly, the potential to improve staff retention as such investment can increase job satisfaction and loyalty. 

Alternatively, growing the AI skills of your workforce can improve the success of change management initiatives related to AI, instead of hiring externally and potentially experiencing resistance to change or external consultants that have less in depth knowledge of your organisation. Fostering the AI capability within the workforce will also have a boost to productivity, as the workforce learns how best to extract value from AI systems. 

Adapt workflows to include AI

The successful implementation of AI hinges on complete integration within existing workflows, rather than standalone implementations. For example, implementing an AI chatbot for IT ticketing that is not integrated within the existing ticketing system, as follows: