Enterprise AI Assistants: Combatting Fragmentation

I regularly speak to clients and prospects about the use of Enterprise AI Assistants to act as a unified interface to their institutional knowledge. The AI use cases that underpin this AI Knowledge Management System range from question-answering to data analysis. It is interesting because traditional tools do not provide that level of scope. For example, you cannot ask your analytics dashboard whether anyone knows what to do when production goes down. If it were possible to connect all your disparate data sources to an intelligent AI assistant, you could fulfil all of these use cases and much more.

This article is a summary of my thoughts after a conversation I had with a client. I will expand upon this idea by considering what happens when trying to implement a system in practice. This brings me to the insight that in the future, we need a pattern that looks very much like an assistant of assistants.

The Challenge: Enterprise Data Fragmentation and AI Integration

Modern businesses are built of a wide variety of data sources. These include emails, documents, internal databases, chats, logs, and much, much more. The challenge is that these data sources are often siloed. Companies like Notion, Microsoft, and Google are trying to solve this problem by providing a single AI Knowledge Management System that has access to all of their data sources.

In isolation, these assistants work well for a limited set of use cases that are related to each vendor’s capabilities. For example, Notion AI is excellent at answering questions about Notion documents. However, it does not integrate as effectively with Notion databases (such as Kanban boards), nor does it currently support external data sources like emails. So what can you do?

One answer is that you build a variety of data pipelines to extract data from their source and push them into some kind of unified location (e.g., a data lake). Then you can build a single assistant that has access to all of that data.

I will expand on why this is hard in a moment, but first, let us consider what the user experience might look like.

Real-Time Access

The problem with this wide scope is that there are probably a subset of use cases that need near-real-time access to data. For example, you might want to know if a customer has complained about a product in an email. Or you might want to know if a product has been mentioned in a chat.

Most traditional export APIs and Extract, Transform, Load (ETL) tools assume you are moving data batches. To satisfy the real-time access requirement, you need to search the data and find the most relevant information in real-time or, at the very least, stream updates (change data capture) to the processing pipeline to get near-real-time results. Again, this might not be adequate depending on your use case.

Unified Global Assistant

In an ideal world, you want one unified user experience for all of your assistant needs, comprising a UI for human users and an API for machine users. I am certain there will be a need for domain-specific assistants as well, like a coding assistant that lives in your IDE, or a finance assistant that lives in your accounting software. But I think most would like a single global assistant for most of the institutional knowledge use cases. The domain-specific assistants probably access a subset of the global assistant’s data sources, so there is an opportunity there for reuse.

Identity and Access Control

The global assistant needs to be able to access all of your data sources. But it also needs to be able to access private data sources that are only appropriate for you or a subset of your users. You could separate the assistants by role or group. But in general, you probably want to solve the data access control problem at the data source level, which the assistant is aware of.

Performance and Scalability

Assuming you have data ingested or APIs exposed as tools, you can build a single assistant that has access to all of your data sources. But language models only have a limited amount of capability. They cannot do everything, and they cannot do it all at once. For example, in tool or retrieval-based LLM applications, the language model makes the decision on what tool to use and when. You could search everything, but that is likely to be slow, and it might make it hard for the language model to ascertain which data is most relevant. For example, if you ask a question like “what is the status of the project?” you might get a response from Jira, from Google Drive, from Slack, and from emails.

But which is the most relevant? Retrieval-based relevancy is usually optimised on a per-data source basis. They are not designed to compare the outputs of APIs with Word documents. And therefore, performance in some use cases will be poor.

You have to consider the size of your engineering team compared to those that are providing these large-scale LLM systems. I would suggest that commercial vendors are probably better placed to optimise the use of their LLMs to work with their data sources. And we want to take advantage of that.

Missing Proprietary Features

Let us take it as a given that data-centric vendors provide assistants with a variety of features. It is still likely that you will need something more.

For example, I was just chatting about the need to audit the use and performance of assistants. Without visibility, you cannot improve. Some of the more advanced vendors provide a simple thumbs up/down rating interface, but that information is passed on to the vendor, not you! In addition, the feedback is probably not granular enough to be useful to help alleviate your complicated use case landscape.

You need to know how it is being used. You need to know what it is being used for. You need to know why the assistant is good or bad. Only then can you surface issues and prioritise improvements.

This is just one example of a missing feature. Depending on your use case, you will probably need a variety of other features.

The Key Challenges of AI Assistants

In summary, the key challenges of AI assistants are:

• The assistant must be connected to all of your data sources (public and private)
• The user must only be able to access the data sources they are authorised to access
• The assistant must be able to access data in near-real-time
• You need features that do not exist in data-vendor assistants

All of these challenges are solvable, but they are not easy, and they do not come out of the box.

Custom AI Assistant Development

One of the oldest engineering tricks of the trade is to carve a large complicated problem into a set of smaller, easier to solve problems. This is most commonly achieved in software engineering by breaking problems apart at the seams that exist between domains (i.e. Conway’s Law). In the case of AI assistants, the domains are defined by the users, data sources, and sometimes, their vendors.

Domain-Specific AI Assistant Implementation

In real life if you wanted to ask a question about the financial performance of a project, you’d speak to someone in finance. Then you might follow up with questions to the project manager or engineering team. This is exactly the kind of pattern that we need to achieve to make generic AI assistants work. We need to replicate the communication paths that exist in the real world.

This idea dramatically reduces the complexity of the problem. We can now focus on solving the problem of how to create an assistant that solves problems within a specific domain. This is much more tractable than trying to create a single assistant, because the use cases are much narrower and the data sources tend to be more homogeneous.

Once we have a set of domain-specific assistants, the next challenge is to replicate the communication paths that exist in the real world. I’m hesitant to prescribe a particular solution here, because I can imagine a few different approaches.

Hierarchical AI Assistant Architecture

The first is a hierarchical assistant, where you have a single global assistant that can access all of the domain-specific assistants. This is probably the most straightforward approach and probably the first thing I’d try. But it is limited by the prompting and routing that occurs in the parent assistant and the quality of the child assistants. For example, the parent might not know that the finance assistant needs information from the engineering assistant about current progress to answer a question about financial performance.

The main simplicity behind this is that the routing is located in one place, the parent assistant.

Federated AI Assistant Architecture

The second is a federated assistant. In this scenario, any assistant is allowed to communicate with any other assistant. So you could ask the engineering assistant about the current financial state of a project, which then routes the question to the finance assistant, which then re-asks a series of progress questions back to the engineering assistant, before finally answering the question.

This is far more flexible, but it is also more complex. You could imagine an infinite loop of assistants asking each other questions. You’d need a monitoring system to ensure that the assistants aren’t asking each other questions indefinitely.

You’d also need some way of keeping the routing information up to date in each assistant. A centralised system could be used to store the routing information, but this would be a single point of failure. A distributed system could be used to store the routing information, but this would be more complex to implement and maintain.

Decentralized AI Assistant Architecture

The third is a decentralized assistant. In this scenario, each assistant is responsible for its own data sources, like in the federated assistant. But a decentralized assistant is also responsible for ensuring that there is concensus on various aspects of the end result.

For example, you could imagine a scenario where a variety of “AI experts” are asked to come to a consensus on what is the best answer to a question. Or concensus on how to answer a question.

Concensus over routing is interesting too. You could imagine a “protocol” (basically just a question) that allows assistants to ask each other what their role is and what they’re best at. All assistants could then come to a concensus over what capabilities each assistant has and what they should be used for.

I’m unsure of the practicality and usefulness of this approach, but it’s quite fascinating.

A Note on Agents

“Agent” is a wildly overloaded term in the AI space. I consider an agent to be a language model that is empowered to call upon external systems in order to achieve a goal. This spans RAG, tools, automated assistants, and more. It is too broad to be useful in a technical context.

For example, if you speak to a travel agent, you are hiring a person to perform a multitude of tasks to achieve the goal of going on holiday. That includes a lot of different systems and decisions. To encode all of that in a single language model, with all the edge cases involved, is a huge challenge. I believe that a true AI travel agent is most likely to be a collection of systems and assistants, each of which is responsible for a subset of the overall goal.

None of this precludes the use of “agents,” however people define the term, in the previous patterns.

Access Control

These patterns do not consider access control. Data architectures typically compare authenticated users against a set of policies that define what data they have access to. All of the current language models and their APIs do not consider access control. This is a problem, because you need to ensure that the assistant is only accessing data that the user has access to. If this is important to you, then you need to design and build a data access control layer behind each agent.

Other Types of Domain

I talk about domain-specific assistants in the context of business domains. But you could slice domains further and consider domains specific to a particular sub-field or technology. For example, you could have a domain-specific assistant that connects to an esoteric database like the internet sacred text archive or MySQL (ha!).

One good example of this is in the context of using different assistants to connect to different types of RAG stores like Neo4j, MongoDB, Elastic, and, yes, MySQL. The main challenge here is that there is a significant amount of centralised coordination required to ensure that the assistants are working together. I envisage a future where assistants are able to communicate through the medium of natural language only.

Practical Enterprise AI Assistant Implementation Guide

Your unique contextual requirements determine how you build your solution. This requires a bespoke approach tailored towards your business’s needs. The following list is a brief roadmap to help you get started:

1. Review your data sources and determine the level of fragmentation
2. Evaluate whether vendor assistants provide sufficient coverage of your data
3. Evaluate whether the vendor assistant provides the features you need
4. If not, build a proof of concept for a domain-specific assistant (go to step 1, but with a subset of your data)
5. Build out logging, monitoring, and feedback features to ascertain how the assistant is being used. Analyse the data to identify issues and opportunities for improvement
6. Iteratively improve assistant(s) to address the issues identified

Conclusion: The Future of Enterprise AI Assistants

As businesses accelerate their adoption of AI Knowledge Management Systems, the ecosystem is becoming more fragmented, not less. A key decision for every organization is whether to consolidate data and functionality around a single Enterprise AI Assistant platform or to implement a Custom AI Assistant Development approach that orchestrates multiple Domain-Specific AI Assistants through a unifying interface.

The future of Enterprise AI Assistants lies in creating intelligent ecosystems where AI Data Integration Solutions enable seamless communication between specialized assistants. This new paradigm of Hierarchical AI Assistant Architecture promises to revolutionize how organizations access, analyze, and leverage their institutional knowledge.

Ready to transform your organization’s knowledge management with Enterprise AI Assistant solutions? Contact our team to discuss your specific requirements and discover how we can help you implement a custom AI assistant strategy that drives real business value.