CreditChat: The Benefits of Integrating Generative AI into a Credit Analysis and Automation Platform
Alexandre Marinho, Alexandre Fazolin e Joseph Guarese
The analysis and assessment of credit risk of legal entities plays a fundamental role in making decisions regarding the establishment of credit limits and durations values granted to these companies depending on their level of risk and the expected return depending on aspects such as reciprocity and relationship.
The use of artificial intelligence techniques in credit automation is not something new, dating back to the late 1980s, when the first expert systems for credit analysis began to emerge, based on technologies such as decision trees, logistic regression and neural networks.
Nevertheless, recent advances in Generative AI technology open up a wide range of new possibilities. Chatbots based on LLM (Large Language Models) models, such as ChatGPT, have been gaining great popularity due to their ability to carry out complex activities in an automated and personalized way, revolutionizing the way organizations operate.
As a result, in the coming years, customizations of LLM models will enable a dramatic increase in the productivity of organizations through the automation of complex day-to-day tasks. Areas such as Finance, Marketing & Sales, Programming, and many others will benefit profoundly from this evolution.
The Advent of Chat GPT
Recently, the introduction of the ChatGPT Platform by OpenAI revolutionized the world of Generative AI. ChatGPT uses Deep Learning techniques, natural language processing (NLP), and LLM technology to generate human-like responses to user queries.
As expected, the entire finance industry and particularly the areas of credit and risk management, analysis, approval and review will be profoundly impacted by the possibility of applying this technology to activities such as analysis reports generation, automated reviews of customer credit limits and dynamic approval workflows.
ChatGPT Applications in the Credit Area
There are numerous applications of ChatGPT in the Financial Area and particularly in the Credit Analysis Area. Among the various use cases in which Generative AI can be used, the following stand out:
• Balance Sheet Analysis
• Analysis of Credit Proposals
• Review of Credit Lines and Limits
• Credit Portfolio Analysis
• Litigation Analysis
• “Peer Analysis”
• Preparation of Credit Reports and Presentations
• Decision Support in Credit Committees
The analysis and compilation of large amounts of financial data can be time-consuming and subject to human error, especially if carried out by professionals with less experience.
ChatGPT is very efficient in analyzing large amounts of data, being able to simultaneously evaluate registration aspects, internal and external information, financial ratios and news about the Company, detecting trends and identifying potential credit risks.
Additionally, carrying out up-to-date analyzes based on trends, news and market sentiments adds substantial value to effective credit risk management.
By automating this process, Organizations will enable credit analysts to save valuable time used in preparing credit analysis reports and use it in activities such as visiting customers and improving the ability to detect credit default patterns to refine models. , in order to present accurate and contextually relevant forecasts.
CreditChat
Since its founding, 25 years ago, SIACorp has used Artificial Intelligence techniques such as decision forests, Bayesian networks and logistic regression in the rating calculation models and limit suggestions made by its CreditFlow Credit Automation Platform.
Envisioning the potential for using Generative AI technology in the credit analysis and granting process, SIACorp created CreditChat, a Chatbot for Credit Analysis fully integrated with the CreditFlow Credit Automation Platform.
This information includes customer's relevant data, their financial statements, punctuality information (dynamically accessed in the accounts receivable module of the ERP or Core Banking System), restrictions (accessed in credit bureaus) and behavioral information, which is automatically uploaded by the CreditFlow modules in ChatGPT dynamically building the prompt submitted to generate the Analysis.
A future version of CredtChat will incorporate recent news. As the knowledge base of the current ChatGPT version is trained with news until July 2023, we decided not to incorporate news processing at this time.
APIs, Prompt Engineering, and Model Training
Through advanced integration and prompt engineering techniques, we use ChatGPT to generate context in JSON format from information provided by the CreditFlow Platform and then use it as input for the Open.AI API, ultimately exporting the information generated by ChatGPT in JSON/XML format and rendering dynamic analysis reports.
Open.Ai's APIs and Tools can be used to customize ChatGPT regarding Text Generation, Prompt Engineering, Embeddings, Speech to Text and Text to Speech, Image Generation, Model Fine-Tuning and Vision. To implement CreditChat, SIACorp focused on the use of APIs in the context of Text Generation and Prompt Engineering.
Information Analyzed by CreditChat
Among the information submitted by CreditChat to ChatGPT we can highlight aspects such as:
• Master Data and Registration Information
• Internal Financial Information (ERP or Core Banking)
• External Financial Information (Credit Bureaus)
• Information and Debt (Brazil Central Bank SCR Sustem)
• Collateral and Gurantees Information
• Scoring and Credit Rating Criteria
• Regulatory Compliance
Furthermore, the system is capable of reading files and can be trained with large volumes of information contained in documents such as:
• Balance Sheets
• Industry Credit Standards
• Financial Health Indicators
• Previous Credit Reports
Based on information uploaded from CreditFlow through the Open.AI Library and automatically submitted to ChatGPT, CreditChat prepares detailed and complex credit analyses, which consider all aspects analyzed, obtained from the submission of questions (or prompts) to ChatGPT.
Prompt Engineering
Prompt Engineering is one of the most important activities within the scope of Generative AI.
This topic is so relevant that SIACorp created a Prompt Engineering Area, dedicated to the study and elaboration of ideal questions (in JSON and natural language) to obtain the best possible answers regarding topics related to credit analysis.
Writing data to the ChatGPT prompt screen works well in situations where you want to get answers to queries made on a relatively small data set.
However, it is not the best way to work with a large set of documents or web pages as input to LLM, due to the limitation on prompt size and the cost associated with passing large text to ChatGPT.
To implement CreditChat, SIACorp created customized prompts, in addition to training the ChatGPT knowledge base with data and documents stored in CreditFlow, as shown in the following figure.
Among the various Prompt Engineering techniques available, we can mention Chain-of-Thought (COT) Prompting (used by SIACorp in the implementation of CreditChat), Zero-Shot Prompting, Few-Shot-Prompting, Tree of Thoughts Automatic Reasoning and many others .
Through the use of the COT technique, ChatGPT can generate a series of intermediate steps to lead to the final answer, providing detailed information as well as reasons and justifications for your answers.
Embeddings
Embedding is a technique for representing information such as text, image or audio, in numerical form. The diagram below conceptually explains how embeddings are used to retrieve information from documents using LLM. Na técnica de Embedding os documentos passam por um modelo que cria pequenas estruturas de dados e as armazena em um banco de dados vetorial. Quando um usuário deseja consultar o LLM, os embeddings são recuperados do banco de dados vetorial e passados para o LLM.
Embeddings são muito utilizados na Solução CreditChat, que incorpora em suas análises informações contidas em documentos regulatórios, demonstrativos financeiros, relatórios de acionistas e portfolios da empresa.
The RAG (Retrieval Augmented Generation) technique is used to increase the accuracy and reliability of LLMs, expanding their knowledge base. Through this technique, it is possible to build solutions with private data and use the intelligence of an LLM without having to train it again, since they incorporate targeted information, which may be more up to date than the set used in its training or be specific to a context. particular.
Preparation of Presentations, Reports and Credit Assessments
CreditChat generates presentations for committees and richly designed credit reports, thanks to the integration between Generative AI and the entire collection of internal and external documents and information managed by CreditFlow Modules.
CreditChat accesses information on the CreditFlow Platform and submits it as input to the ChatGPT APIs. Based on the information uploaded from CreditFlow submitted to ChatGPT, CreditChat prepares detailed and complex credit analyses, which consider thousands of pieces of information simultaneously, in a manner that fully adheres to the policies and parameters defined by the company.
In addition to the texts automatically generated by Generative AI, the reports generated by CreditChat incorporate KPIs, KRIs, graphs, maps, traffic lights and extensive visual resources to present the credit situation of the company under analysis.
The reports and opinions issued by CreditChat are generated through robust LLM techniques integrated with financial statements, punctuality information (dynamically accessed in the accounts receivable module of the ERP System or Core Banking), debt (Bacen SCR), restrictions (accessed in credit bureaus) and behavioral information.
Technical Architecture of the Solution
Generative AI technology demands high-performance computing systems, aiming for faster processing to deal with the large amounts of data generated by applications.
The entire CreditFlow Platform, including CreditChat, is implemented in a cloud-native architecture with support for containers and microservices.
CreditChat was implemented in C#, on the Microsoft .NET 8.0 Platform. In its implementation, we adopted a wide variety of cutting-edge technologies to guarantee the Platform a very high level of performance, robustness and portability.
Among the technologies used, the following stand out.
• ASP.NET WebApi Core with JWT Bearer Authentication and ASP.NET Identity Core: We implement JWT Bearer authentication to ensure end-to-end security in our operations, simplifying identity management and enabling efficient authentication and authorization.
• .NET Core Native DI: We adopt native dependency injection to enable a more modular and easier to maintain architecture.
• AutoMapper and FluentValidator: We automate processes and validations, simplifying development and ensuring data consistency.
• MediatR: We facilitate communication between system components, promoting a decoupled and scalable architecture.
• Domain Driven Design and CQRS: We adopt a domain-driven design, separating read and write operations for greater efficiency and scalability.
• Event Sourcing and Unit of Work: Traceability and flexibility are guaranteed, allowing a historical view of data and modular maintenance.
• Application Insights: We use detailed metrics to quickly identify and resolve issues, ensuring a seamless experience for users.
Testing Methodology
The following figure presents the methodology used by SIACorp to test ChatGPT proposed by Xingzhi Wanga, Nabil Anwerb and Yun Daic Ang Liua in (1), in which we answered separate questions in six topics relating to the Credit Analysis and Review processes.
Topics relating to Credit Analysis cover aspects such as Registration Analysis (which involves the evaluation of a wide range of data relating to the company's situation), Financial Analysis (which considers the fundamental analysis of financial statements) and Behavioral Analysis (which considers payments of loans and accounts receivable).
Topics relating to Credit Review cover aspects such as Profitability Analysis, Reciprocity Analysis and Analysis of the Customer's Relationship with the Organization.
CreditChat's performance was evaluated based on answers to dozens of questions regarding registration, financial and behavioral aspects of the company, in addition to Central Bank regulations and sectoral analyses.
Furthermore, by modifying some standard questions, several trap questions were generated to test your cognitive flexibility.
Based on the Revised Bloom's Taxonomy (2), questions have been categorized into four difficulty levels:
• Level 1 - Basic Knowledge
Consists of understanding and explaining concepts relating to credit analysis;
• Level 2 - Application of Knowledge
It consists of choosing, employing, illustrating and demonstrating concepts to appropriately solve specific problems.
• Level 3 - Critical Analysis
It consists of dividing information into groups of components to understand its nature or determine characteristics.
• Level 4 - Extension of Knowledge
It consists of the synthesis of new knowledge and implementation of inferences.]
Then, the responses generated were evaluated in relation to four qualitative characteristics: correctness, relevance, clarity and comparability.
While correctness and relevance are two basic requirements, clarity and comparability represent desirable requirements.
In cases where ChatGPT did not meet basic requirements, its performance was considered low, indicating that it is not yet ready to perform relevant human work.
In cases where ChatGPT met the four requirements, its performance was considered high, that is, comparable to that of human analysts.
In other cases, its performance was considered average, which means it achieved acceptable performance, but there is still room for improvement.
The following figure shows an overview of the test results. In a set of questions divided into four categories, in turn divided into six topics, ChatGPT achieved high performance in 15 of the 24 categories and average performance in 9 categories.
Our tests were quite satisfactory and indicated that CreditChat was adequately trained with technical knowledge in credit analysis, and can be used to substantially boost productivity in the Organizations' Credit Areas.
As CreditChat is connected to CreditFlow, a conventional Credit Automation Platform, the knowledge encoded in policies, rules, logic and models, accessed by CreditFlow are made available to ChatGPT, increasing its knowledge and simplifying the tracking process for verification.
Our tests also show that the reliability of responses is closely related to the construction of the submitted prompts. For the same question, different answers can be obtained if we interact with ChatGPT differently.
CreditChat implementation
To fully leverage the benefits of ChatGPT while mitigating its limitations, SIACorp implemented a three-tier model for implementing CreditChat proposed by Xingzhi Wanga, Nabil Anwerb and Yun Daic Ang Liua in (1), as illustrated in the following figure .
In each layer, specific tasks must be performed according to their level of complexity or priority.
• Knowledge Layer
The knowledge layer is the core of the model and consists of loading the knowledge base with specialized technical knowledge in credit analysis, organization standards and procedures and regulatory documents, enabling the generation of robust and consistent analyses.
• Reasoning Layer
It consists of configuring the analysis models and inference engines used by CreditChat, based on methodologies such as forest trees, logistic regression and Bayesian networks.
• Collaboration Layer
This layer represents the biggest challenge for organizations, which will have to restructure the roles of credit professionals and operational procedures to allow more effective collaboration with ChatGPT, involving them in the process of verifying the models that are gradually being refined.
Conclusion
In this article, we present CreditChat, a Chatbot for Credit Analysis fully integrated with the CreditFlow Credit Automation Platform, which adds immense value to the process of producing credit analysis reports in a comprehensive, creative and objective way, thus showing its potential for use for this purpose.
We conclude that implementing Generative AI in credit risk assessment offers benefits such as a dramatic increase in the speed and efficiency of analyzing large data sets, minimizing the potential for bias and human error, and presenting objective and consistent risk assessments. Finally, it allows proactive risk management, identifying potential compliances or non-compliances at an early stage, allowing credit managers to take the necessary measures to mitigate risks.
Our conclusion is that ChatGPT is impressively efficient in providing information, generating coherent and structured content. We will soon publish studies on topics such as embedding, abductive reasoning and training in the knowledge and reasoning layers.
In the coming years, the evolution of LLM models will enable a dramatic increase in productivity in the credit automation and risk management segment and SIACorp is very prepared and available to your organization to assist you in this transition.
Published by:
Alexandre Marinho is President of SIACorp and has been working in Information Technology and Artificial Intelligence for over 45 years and implementing Credit Automation Systems for over 35 years. Alexandre Fazolin is a Partner at SIACorp, working in the IT area for 18 years, leading the implementation of CreditFlow in dozens of projects in Brazil and abroad. Joseph Guarese is a Partner at SIACorp and a specialist in distributed cloud architectures for Credit Automation.
About SIACorp
Working since 1998 in the Implementation of Credit Automation Systems, SIACorp is the Company with the highest number of implementations of Credit Analysis Systems in Brazil. To learn more about the CreditFlow Solution, a tool aimed at automating the Origination, Analysis, Approval and Credit Risk Management processes, click here to schedule an appointment demonstration!