Banking Perspectives: A spotlight on the technology testing process at Moody’s

In the Banking Perspectives blog series, we invite senior leaders from across Moody’s Banking to share their insights on technology at Moody’s and the wider banking industry. 

In the previous edition, Omar Akkor, Senior Director Manager for Banking Product Strategy at Moody’s, reflected on technology and innovation at Moody’s and in the banking industry as a whole.

In this blog, Omar discusses how Moody’s takes on the role of testing to deliver the latest innovative technology to our customers reliably and securely.
 

Q: Thank you for your time, Omar. What is Moody’s approach to strengthening new solutions based on new technologies?

O: Through our partnerships, we have developed an infrastructure that is secured and propriety to Moody’s, ensuring that both Moody’s and its customers’ data is protected.

As Moody’s provides both ratings and analytics, we take data security very seriously. We want to ensure that our systems are secure, and that our data is not used by third party organizations to train their models.

Through our partnerships such as with Microsoft, Google, and AWS, we're putting the right infrastructure and guardrails in place to ensure that the data and models we deploy are well protected. This is the same level of security that we make available to our customers.
 

Q: Tell us more about the testing process of applications and initiatives that we develop. How do Moody’s go through the fortification process once the prototype had been delivered?

O: The experimentation stage is usually fast, but things start to slow down when we go through due diligence.

As part of our process, the first step is to work with our Legal department to understand the implications of the initiatives that we are developing or releasing. We address questions such as what the implications may be on customers, the type of data we can use and not use, the type of sources we can use or not use – from trusted websites, types of attachments to customer proprietary data. We must justify and validate that with our Legal department first.

As a second step, we then go through information security. We need to ensure that the infrastructure we’re building is secure and not breakable. This is an area where we have zero tolerance.

The third step is what we call the ‘Red’ team, a cybersecurity team that tries to find vulnerabilities and break through our cybersecurity defenses.

Once we have done all of that, we then go back to Legal again. Ultimately, it’s a very thorough process to ensure that both Moody's and its customers’ data are protected.

We are also exploring information tokenization before sending Moody’s and its customers’ data to our partner large language models (LLMs). We are analyzing use cases where tokenization adds another security level without hurting the quality of the responses.
 

Q: Interesting – how do you mitigate hallucinations? 

O: We use the LLM as an orchestrator which leverages Moody's and its customers’ information. We, as a user, decide what information is going to be used – such as customer documents, Moody’s or customers’ data, websites and so on. We have also developed guardrails which guide the LLM on how and what information it retrieves.

This is how we control for hallucinations. And then, of course, we always have SMEs and specialists who go through the proof of concept (POC) to review the answers, provide feedback, and adjust the RAG process (Retrieval augmented generation).

A RAG is a code which engineers develop to give instructions to the LLM on what to use and what not to use. These are going to be based on guidelines and guardrails that we put in place as part of our objective, to ensure we’re giving clear instructions on what is acceptable and not acceptable in terms of response. 

We also leverage the LLM itself to recheck the answers and the tone to ensure it validates its own answers. In addition, we use tools that our partners like AWS, Google, and Microsoft make available to us to control those hallucinations.
 

In the next edition of the Banking Perspectives blog series, Omar will discuss Moody’s GenAI development process in more detail, from the experimentation stage to delivery.