While artificial intelligence (AI) companies keep making impassioned pitches about the benefits of AI agents who can automate certain workflows in an enterprise environment, the Singapore government has released a new framework designed to safeguard against risks posed by the deployment of agentic AI, and to ensure responsible development of the technology. The Singapore’s Infocomm Media Development Authority (IMDA) released its Model AI Governance Framework for Agentic AI, with specific focus on the risks of this technology, and its impact on humans in the workplace.
The new framework, released this month, builds on Singapore’s 2019 Model AI Governance Framework, which at the time had focused on AI principles such as transparency, fairness, and human-centricity. There has been conversation about the risks of AI, particularly agentic AI and the march towards artificial general intelligence (AGI), with Anthropic CEO Dario Amodei’s latest essay published this week, also noting the potential of widespread societal and economic changes.
Singapore is one of the countries using AI in government services, having successfully deployed virtual assistants for its public sector, though not the only one. The Government’s ‘Ask Jamie’ virtual assistant has already responded to more than 15 million queries since 2014, across 80 government websites. Official data suggests it has been successful in resolving half of the queries, which would have otherwise been redirected to a human at a call centre.
Parminder Singh, who is the Co-Founder and Chief AI Whisperer at advisory firm ClayboxAI noted in a post on X that “this is a step ahead of the AI Governance Frameworks released by some other governments. It’s a practical blueprint for designing and deploying Agentic AI.”
The UK has relied on tech company Salesforce to deploy an autonomous AI agent called Bobbi, that is helping the Thames Valley, Hampshire & Isle of Wight, and Humberside police forces to deliver around-the-clock answers to frequently asked questions and assisting citizens in logging cases themselves. They say that in the first week of deployment in December, Bobbi resolved 82% of queries without a human officer stepping in.
The City of Barcelona too has adopted a centralised platform, also built by Salesforce, that gives civil servants a complete view of their interactions with citizens, and allows them to track solution delivery and share information across departments.
“The shift to AI agents could also offer a path to greater transparency, moving administrations from ‘black box’ governance to ‘glass box’ governance. But this transparency relies on a critical principle: the human must remain the architect,” notes Kendall Collins, President and Chief Executive Officer, Government Cloud, at Salesforce.
Human oversight is mandatory
Singapore agentic AI framework notes that the start point of risks with agentic AI, is the large language models (LLMs) that they are based on. The threat is dual-pronged with inherent shortcomings such as hallucination, bias, data leakage or adversarial prompt injections, as well as software vulnerabilities such as infection of a malicious code into the agent framework.
Singapore’s framework makes it clear that human oversight does remain a crucial, mandatory element. “While agents may act autonomously, human responsibility continues to apply. Once the’ green light’ is given to deploy agentic AI, an organisation should take immediate steps to make humans meaningfully accountable. This includes clearly defining responsibility across multiple actors within and outside the organisation involved in the agent lifecycle; and taking measures to ensure that human-in-the-loop remains effective over time notwithstanding automation bias,” the framework notes.
It is also the responsibility of an organisation deploying AI agents, to have technical processes and controls in place for the entirety of the AI deployment, including testing for baseline safety, guardrails as well as continuous monitoring.
This framework also urges organisations to weigh suitable use cases for agent deployment with due consideration to agent-specific factors such as impact of the risk, as well as deliberate design choices to bound the risks upfront by applying limits on agent’s access to tools and permissions.
Research firm McKinsey & Co., in a research late last year, estimated that agentic AI systems, which are projected to help unlock $2.6 trillion to $4.4 trillion annually in value across more than 60 Gen AI use cases, including customer service, software development, supply chain optimisation, and compliance.
Evaluating AI agents
AI company Anthropic, in its evaluation methodology released earlier this month, points to three types of graders that define an agent. The first, code-based tests on parameters including binary tests and transcript analysis, while the model-based evaluation tests based on natural language assertions, reference based evaluation and multi-judge consensus. The human grader parameter includes crowdsourced judgment and spot-check sampling.
It is easy to confuse AI agents as a singular form of artificial intelligence in a workplace, but this is a multi-faceted technology.
For instance, there are coding agents that can write or test code. Conversational agents, which can be useful in domains such as support and coaching, have an ability to take actions as part of an ongoing conversation. Research agents can gather, synthesise, and analyse information, with then intent to generate an answer or a detailed report. There are also Computer use agents, which can interact with software through the same interface as humans, that is screenshots, mouse clicks, keyboard input, and scrolling.
“AI agent evaluation is still a nascent, fast-evolving field. As agents take on longer tasks, collaborate in multi-agent systems, and handle increasingly subjective work, we will need to adapt our techniques,” notes Anthropic in its guidance, titled ‘Demystifying evals for AI agents’.
“Not only do AI agents provide new external entry points for would-be attackers, but because they are able to make decisions without human oversight, they also introduce novel internal risks,” the McKinsey & Co. research warns.
“Threat modelling also makes risk assessment more rigorous by systematically identifying specific ways in which an attacker may take to compromise the system. Common security threats to agentic systems include memory poisoning, tool misuse, and privilege compromise,” the new framework warns.
A way to work around the threat perception is setting up an agentic AI system that has checkpoints or steps that require human approval. Some instances of sensitive workflows include when an AI agent is framing guidance in healthcare or for a legal query.
Limiting an agent’s powers
For organisations deploying agentic AI within their workflows, Singapore’s guidelines make it imperative for them to ensure that the agent’s powers are limited in terms of balancing potential risks. It is mandatory to define policies that allow agents access to the minimum required tools and data required to do its tasks. For instance, a coding assistant does not necessarily require access to web search, if it has been given access to the latest software documentation. There is also a requirement for protocols that limit an agent’s freedom at various steps.
Organisations and businesses deploying AI agents will be required to design mechanisms and procedures to take agents offline and limit their potential scope of impact when they malfunction.
AI agents would need identification as well. “Identity management and access control is one of the key means in which organisations enable traceability and accountability today for humans. As agents become more autonomous, identity management has to be extended to agents as well to track individual agent behaviour and establish who holds accountability for each agent,” notes the Singapore framework.