As AI permeates our lives swiftly, finance and legal professionals are eagerly embracing its potential: GPUs are flying off shelves, investments are pouring in, and legions of AI innovators are poised to reshape the world. Artificial General Intelligence (AGI) has never been so close to us, yet still out of reach. What can we really do in the meantime? Twiddling our thumbs is hardly an appealing option. Amidst the flurry of excitement, there's a sobering reality for law, finance, accounting, and government professionals: how do we harness the power of AI without compromising the confidentiality of sensitive documents, many of which fall under Attorney-Client Privilege?

The most obvious choice seems to be hosting your own AI solution in your own guarded environment. Whether you're training your own model from scratch or fine-tuning existing open-source models like Mistral and Grok, the upside is clear: tailor-made performance to suit your unique needs, with no risk of sensitive documents slipping beyond your control. However, it's not all sunshine and rainbows. Developing and maintaining your own model is a pricey endeavor, not to mention the ongoing costs of fortifying your security measures. The Return on Investment (ROI) is murkier than a swamp on a foggy day. What if better models emerge down the line? What if AGI becomes a reality? Yet, for many large organizations, the sheer volume of documents necessitates hosting their own AI solutions, even if it means settling for slightly subpar performance. It's worth noting that even with a private AI deployment, document-level control remains crucial. For example, many client documents can't be shared across the whole organization (e.g., documents in the D.C. office should remain confidential within the D.C. office personnel only). This is currently achievable by combining Retrieval Augmented Generation (RAG) with enterprise search solutions such as Elasticsearch's built-in document-level access security, ensuring confidentiality isn't compromised beyond proper document-level security parameters.

Cloud-based AI ops innovations offer a second path for those wary of the hefty price tag of developing their own AI. Platforms like chatpdf.com allow users to upload documents or connect their drives, all while boasting certifications like SOC 2, HIPAA, FedRAMP, and ISO. However, before taking the path, ask yourself: will your clients be comfortable knowing their sensitive documents are entrusted to a third-party cloud? This question is especially thorny for law firms tasked with safeguarding attorney-client privileges but often lacking in cybersecurity prowess. While SOC 2-certified cloud solutions might meet the minimum requirements of state Bar Associations, they may fall short of client expectations.

The crux of the confidentiality issue in the AI age boils down to this:

There are always some documents you will never feel comfortable putting online.

Looking to the future, the author proposes a two-pronged approach for professional AI use with confidential documents. Firstly, in-house or via a third party, hosted AI solutions can be leveraged for firm-wide documents (typically less sensitive, such as policies and process guidance). Secondly, on-premise and self-managed AI solutions deployed on endpoint devices offer a localized solution.

I am definitely not alone in this position with the second prong. AI researchers have long recognized the importance of private and local documents in helping AI perform. We will dive deep into the second prong since we have discussed the first choice already. But before we proceed, let's examine the Retrieval Augmented Generation (RAG) concept first.

When an LLM is developed, generic and quality datasets are used to train and develop the model. The model developed thus has the baseline knowledge of the data fed to it but lacks special and private knowledge from your confidential documents because such data are not known nor exposed to the LLM developers (even if known, developers often choose to ignore specialized datasets because such datasets may skew the model performance unexpectedly for common usage scenarios).

To equip AI with private knowledge, RAG plays an important role (high-level explanation below):

Retrieval – local documents are extracted, parsed into digestible pieces, and converted into vectors that AI models can understand in the next step. Those vectors encode semantic information from the local documents and represent facts, concepts, or contextually relevant details from the local records.

Augmented – When a user queries the AI, AI models consider the vectors supplied to better understand the user query, its relevant knowledge, and context.

Generation – AI models generate responses based on the most relevant vectors, ensuring the generated texts align with the retrieved knowledge embedded.

RAG has become a staple in AI solutions, significantly enhancing performance when supplied with the right contextual knowledge. With RAG explained, let's explore two sub-choices for running AI solutions with RAG on endpoint devices:

The first choice is to run the entire AI model locally on the endpoint device. This is arguably the safest choice today to use AI with confidential documents because no internet is even required. For example, NVIDIA launched an experimental software called "NVIDIA Chat with RTX" to demonstrate how to run AI models locally on personal computers equipped with expensive NVIDIA RTX GPUs. This option is ideal for safeguarding confidentiality and maintaining the attorney-client privilege --- nothing leaves your local machine. However, there are obvious downsides: 1. Your machine must be super-powered with the most advanced GPUs on the market to support local AI models. 2. Although security and confidentiality are the best, the performance of the local AI is much lower than that of the best AI models on the market, which are often proprietary (e.g., OpenAI's GPT models).

The second choice is to run RAG and the AI models via APIs on the endpoint devices. This is arguably the second safest choice to use AI with confidential documents because although the documents are chunked and processed by the API providers via an internet connection, the documents do not live outside. Crucially, the vectors/embeddings processed via the API method are almost impossible to reverse-engineer back to the original texts by an outside party (including the API providers themselves), thanks to semantic compression and contextual nuances. The safety reasons here are muti-fold: 1. RAG is a semantic compression with many details that are lost without which humans can't comprehend. 2. Vectors represent the semantic relatedness of words, phrases, or concepts and carry context-specific nuances. 3. Vectors capture only word abstraction instead of a verbatim copy of the original content. 4. Original documents are chunked in pieces and processed by API providers' serverless functions, because of which API providers can't, don't, and probably will never want to (for cost reasons) save those chunks themselves. In addition, after the API processing of the local documents, vectors can be saved locally to avoid exposing them to outsiders.

The advantages of the second choice are apparent: 1. API providers provide the best-in-class embeddings/vectors with Retrieval. 2. API providers also provide the best-performing AI models for Augmented Generation. 3. Confidentiality and attorney-client privilege are also reasonably achieved via the on-prem local computer programs that can be self-managed.

Using this method, you can run AI solutions locally on your regular consumer-grade computers without worrying about uploading your documents to the cloud or a third party. You can chat with your private documents, such as diaries and personal logs, confidential client documents, tax returns, and anything you like. You can also create unlimited use cases (contexts) to make AI as versatile as possible (similar to OpenAI's GPT Stores; in our case, the whole program can be self-managed locally). This method is particularly useful for small organizations that don't have enough resources to invest in AI solutions themselves yet still want to harness AI to deal with local confidential documents.

Conclusion:

In closing, while AI holds immense promise, let's not forget the human element. Ultimately, humanity trumps technology, especially when dealing with matters requiring confidentiality and attorney-client privilege.

Sidenote:

If you are a CPA interested in becoming an IT professional, please check out the AICPA's CITP program (see the detailed information in my LinkedIn volunteer section).