Unveiling the blackbox within ESG ratings' blackbox: Toward a framework for analyzing AI adoption and its impacts

2.1 Sustainable development

Over the last four decades, the concept of sustainable development emerged and consolidated in response to pressing global challenges. The most widely adopted definition, presented in the Brundtland Report, conceives it as the development that meets today's needs without compromising the ability of future generations to meet their own (UN, 1987). Economic development, social development, and environmental protection are mutually dependent pillars of sustainable development (UN, 1997). The notion of sustainable development has been subject to considerable debate in academia (Osorio et al., 2005). It has been critiqued for being vague and its applicability has been questioned (Hopwood et al., 2005). However, the concept has become hegemonic over the last decade, and it has been integrated into international treaties, national constitutions, and corporate policies (Ruggerio, 2021).

The notion of sustainable development was further articulated with the adoption of the 2030 Agenda and its 17 Sustainable Development Goals (SDGs) by all members of the United Nations, in 2015. One of the most significant shifts in the global development agenda brought by the adoption of the SDGs was the increased emphasis on the role of the private sector (Scheyvens et al., 2016; Suárez Giri & Sánchez Chaparro, 2023). Businesses boast a series of strengths for delivering on the SDGs such as managerial capacity, technological innovation, know-how, and a higher willingness to take risks (Berrone et al., 2019; Scheyvens et al., 2016). The financial sector also has a pivotal role to play in advancing sustainable development by promoting a long-term investing paradigm and contributing to channel capital toward activities that support social and ecological systems (Busch et al., 2016; UN, 2015).

2.2 The ESG rating industry

The growing imperative for sustainable development has significantly transformed the finance and investment landscape in recent decades. Although the practice of incorporating sustainability factors into investing is not new, it was previously relegated to niche segments within the industry (Billio et al., 2022). Early approaches, such as ethical and socially responsible investing, were values-driven (Eccles et al., 2020). Investors generally accepted a certain trade-off between financial and sustainability objectives. A new paradigm emerged with sustainable finance. While there is not a universally agreed definition for this concept (Cunha et al., 2021), sustainable finance is commonly understood both in academic and practitioner literature as the practice of integrating environmental, social, and governance factors into investment decisions (Musleh Al-Sartawi et al., 2022; World Bank, 2021). This approach is value-driven, as the consideration of sustainability aspects is expected to drive long-term financial value (Clark et al., 2014; Eccles et al., 2020; Friede et al., 2015).

A pivotal element in enabling this paradigm shift has been the emergence and consolidation of the ESG rating industry (Berg et al., 2022). ESG raters are organizations that evaluate the environmental, social, and governance performance of companies, and quantify it in the form of standardized ratings (Pagano et al., 2018). Their metrics enable time-constrained asset managers to outsource much of the research and expertise needed to integrate sustainability factors into investment decisions (Hughes et al., 2021; Van Duuren et al., 2016). ESG ratings have been crucial for mainstreaming the integration of sustainability considerations in the investment industry (Clark & Dixon, 2024). The volume of assets managed following these metrics has surpassed $20 trillion in 2022 and projections indicate this volume will double by 2030 (Bloomberg, 2024).

The ESG rating industry has seen substantial transformations over the last three decades. Initially, ESG metrics were provided by small-scale organizations, including NGOs, specialized firms, and advocacy groups (Pagano et al., 2018). However, the rising demand for data driven by growing interest in sustainable investments during the 1990s and 2000s led to a proliferation of providers (Mooij, 2017). More recently, as the industry transitioned from its growth phase toward maturity, a notable consolidation process has occurred (Avetisyan & Hockerts, 2017; Mooij, 2017). Leading firms have acquired smaller raters to expand their geographic coverage and improve their competitive position (Antolín-López & Ortiz-de-Mandojana, 2023). Consequently, the sector has become dominated by a handful of the world's most prominent credit rating agencies and financial institutions, namely, Moody's, Bloomberg, LSEG-Refinitiv, Morningstar-Sustainalytics, MSCI, S&P Global, and Deutsche Börse-ISS-ESG (Antolín-López & Ortiz-de-Mandojana, 2023; Avetisyan & Hockerts, 2017). However, recent advancements in AI have paved the way for new entrants to the industry. Numerous tech-driven companies have emerged, producing ESG ratings leveraging AI (Cucari et al., 2023; Hughes et al., 2021).

A categorical distinction between the incumbents and the AI-driven new players in the ESG rating industry was introduced by Hughes et al. (2021). On the one hand, traditional raters have been defined as firms that formulate ratings through the expertise of human analysts, who process data such as corporate disclosures, surveys, and news articles employing proprietary methodologies (Ibid). Their use of AI has been described as minimal and restricted to data collection (Ibid). Traditional raters include all of the above-mentioned major rating agencies, which concentrate the vast majority of the ESG rating market. In contrast, firms that rely on AI for collecting data, analyzing it, and producing sustainability ratings have been described as alternative ESG raters (Ibid). Typically, alternative raters are much smaller and newer entities, with examples including RepRisk, TrueValue Labs, and Arabesque. These firms process data from sources such as news, journals, and NGO reports in near-real-time. The division between traditional and alternative ESG raters has been widely adopted in both academic and practitioner literature (Crona, 2021; Schmidt, 2022; Tominaga, 2022; Truant et al., 2023).

Despite the unprecedented evolution of AI in recent years, the landscape of empirical research on AI adoption within the ESG rating industry remains notably limited. One prominent exception is a recent study by Cucari et al. (2023), which maps the AI technologies adopted by a large sample of ESG data providers. These authors based their analysis on a sample of 32 companies, for which they analyzed their websites. While their study examined a broad range of ESG data providers, it notably excluded traditional rating agencies. Our research addresses this critical gap in the literature by investigating the adoption of AI among traditional ESG raters.

2.3 Artificial intelligence

There is not a universally agreed definition of AI. For this study, we adopted the recently updated OECD's definition, according to which “An AI system is a machine-based system that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments” (OECD, 2023).

Machine Learning (ML) and Deep Learning (DL) are other important terms in the field. ML is a branch of AI that focuses on enabling machines to learn from data and improve their performance over time on specific tasks, without being explicitly programmed to perform those (Brown, 2021). It involves collecting large volumes of data and feeding it to a model, which trains itself to find patterns or make predictions (Brown, 2021). DL, on the other hand, is a field within ML that involves training deep neural networks to recognize patterns and make sense of data (Kotu & Deshpande, 2018).

Natural Language Processing (NLP) is another prominent field of AI. It is concerned with giving computers the ability to understand language, mirroring human-like capabilities (Raina & Krishnamurthy, 2022). This is achieved by combining rule-based modeling of human language with statistical, ML, and DL models, including Large Language Models (LLMs) (IBM, 2023). Some of the most relevant applications of NLP in the context of ESG assessments include Name Entity Recognition (NER), Sentiment Analysis (SA), and Text Classification Algorithms (TCA).

NER algorithms are used for recognizing and extracting the names of organizations, locations, or individuals, from a body of text (Mansouri et al., 2008). SA deals with understanding the opinions, sentiments, attitudes, emotions, or evaluations of the author of a text (Liu, 2017). SA algorithms typically classify text according to positive, neutral, and negative categories (Tonkin & Tourte, 2016). TCA are tools for assigning categories or labels to textual data. Other relevant NLP technologies include text summarization, information extraction, and machine translation algorithms.

AI embodies the characteristics of general-purpose technologies (GPT), which are defined by their significant impact on productivity (Crafts, 2021), widespread application across industries, improvement over time, and generation of complementary innovations (Bresnahan & Trajtenberg, 1995; Lipsey et al., 2005). Classic examples include the steam engine, electricity, and information and communications technologies. As a GPT, AI transforms how companies operate by driving changes at the task, process, and business model levels (Brynjolfsson & Mcafee, 2017).

2.4 Technology adoption theories

Technology adoption at the organizational level can be understood as the integration of a new technology to improve performance, service, and/or delivery (Chakraborty et al., 2021; Tsumake & Swart, 2019). Over the last five decades, there has been considerable scholarly attention focused on understanding which factors influence technology adoption, which gave rise to a plethora of theoretical developments (Amini & Bakri, 2015). The most prominent include the Diffusion of Innovation Theory (DOI) (Rogers, 1995), the Technology-Organization-Enviroment Framework (TOE) (Tornatzky et al., 1990), the Technology Acceptance Model (Davis, 1986), the Theory of Planned Behavior (Ajzen, 1991), and the Unified Theory of Acceptance and Use of Technology (Venkatesh & Davis, 2003).

For the theoretical underpinning of this study, we adopted the DOI theory and TEO framework. On the one hand, these are the only established technology adoption theoretical frameworks applicable to the organizational, rather than the individual level (Oliveira & Martins, 2011; Schmitt et al., 2019). On the other, these two frameworks are considered highly compatible (Baker, 2012; Oliveira & Martins, 2011), including for the study of AI adoption (Alsheibani et al., 2020; Smit et al., 2023). Furthermore, these have been extensively employed in combination to guide empirical research on AI adoption, including several studies in the field of finance (See Table 1).

The DOI theory proposes five factors that influence the adoption of a new technology: its relative advantage, compatibility, complexity, trialability, and the observability of its results (Rogers, 1995). Relative advantage refers to the degree to which a new technology is perceived as better than previous ones. Compatibility is the degree to which a new technology is consistent with the needs of potential adopters. Complexity refers to the perceived difficulty of utilizing an innovation. Trialability is the degree to which a technology can be tested on a limited basis. Lastly, observability refers to the degree to which the effects of adopting a new technology are visible. Although the DOI theory remains highly influential, it overlooks how internal organizational attributes and external environmental factors influence adoption (Parker & Castleman, 2009).

The TOE framework posits that a firm's adoption of a new technology is influenced by its technological, organizational, and environmental context (Tornatzky et al., 1990). The technological context refers to all available technologies that are relevant to a firm (Baker, 2012). The organizational context refers to the characteristics and resources of the firm. It includes variables such as size, organizational structure, resource slack, and top-management support for the implementation of new technologies (Baker, 2012; Oliveira & Martins, 2011). The environmental context refers to the setting in which an organization operates. It includes factors such as industry structure, level of competition, and the regulatory environment (Oliveira & Martins, 2011). The TOE framework is regarded as highly complementary to the DOI (Baker, 2012; Oliveira & Martins, 2011). The factors influencing adoption in the DOI are encompassed within the technological context of the TOE framework (Oliveira & Martins, 2011).

Numerous empirical studies have combined the TOE and DOI frameworks to investigate the factors influencing business adoption of AI technologies. This body of research is comprehensive, encompassing a wide range of industries and regions, providing a robust foundation for understanding the drivers and barriers of AI adoption in firms. Table 1 synthesizes the most relevant studies, highlighting the key factors that influence business AI adoption.

Based on the drivers of AI adoption identified in the literature, our hypothesis is that major ESG raters have significant incentives to adopt AI. First, the field of ESG ratings is an especially fertile ground for AI, given its ability to process vast amounts of unstructured data in real-time, at a fraction of the costs of human analysts (Amundi & IFC, 2021). Moreover, leading ESG raters boast highly skilled workforces and sophisticated IT infrastructures. Additionally, these control considerable financial resources and huge volumes of valuable data. Finally, these face increasing competitive pressures from AI-driven new entrants to the sector (Hughes et al., 2021).

2.5 Shortcomings of the DOI theory and TEO framework for the study of AI adoption

The DOI theory and the TOE framework are valuable models for understanding why firms implement new technologies. However, they provide limited tools for examining how organizations integrate new technologies and for analyzing the impacts of their adoption. These shortcomings are particularly relevant for the study of AI adoption, for two key reasons. First, AI comprises a wide spectrum of technologies and has a wide range of business applications. Hence, unlike previous technologies (e.g., cloud computing), the specific features of AI adoption can vary radically from one firm to another. Second, due to its revolutionary capabilities and its nature as a GPT, the impacts of AI adoption are many and far-reaching (Brynjolfsson & Mcafee, 2017).

Despite these limitations, the DOI theory and the TOE framework are arguably the most widely employed models for studying business AI adoption. Consequently, empirical research has predominantly focused on the drivers and barriers of firms' AI adoption rather than on the specific technologies adopted and their applications (Kar & Kushwaha, 2023). Moreover, comprehensive empirical studies on the impacts of business AI integration are remarkably scarce (Cubric, 2020; Lee et al., 2023). Existing research primarily focuses on the positive outcomes of AI adoption (e.g., enhanced operational efficiency, firm growth, increased innovation) (Babina et al., 2024; Czarnitzki et al., 2023; Mishra et al., 2022), while largely overlooking its broader impacts on adopting firms and their stakeholders.

Given these gaps in the literature, it is crucial to extend established technology adoption models to support the analysis of the specific characteristics of firms' AI integration, and how these influence the impacts of adoption on companies and their stakeholders (Duan et al., 2019; Venkatesh et al., 2016). Although some previous studies have proposed extensions to the DOI Theory and TOE Framework to address the effects of technology adoption, these remain limited in scope.

For instance, Acheampong and Moyaid (2016) extended the TOE framework and DOI theory to examine how technology adoption affects firms' financial and operational performance, introducing the time since adoption as a moderating variable. Similarly, Aligarh et al. (2023) TOE framework extension focused on cloud computing adoption's impact on financial outcomes, incorporating staff computer confidence as a moderator. Finally, Nguyen et al. (2022) extension of the TOE framework for online retailing adoption considered firm age and employee count as moderating factors influencing the impacts of adoption on business performance.

While these studies emphasize the need to broaden the scope of established conceptual frameworks to explore the impacts of technology deployment, they all treat adoption as a dichotomous variable, overlooking the specific features of technology integration. Moreover, these have a narrow focus on firms' financial and operational performance. Importantly, none of these extensions were specifically developed for the study of AI adoption.

One notable exception in the reviewed literature is a recent study by Shahzadi et al. (2024), which developed a TOE framework extension tailored to examine AI integration in supply chain management. This extension is particularly significant for three reasons. First, it was specifically developed for the study of AI adoption. Second, it does not treat AI adoption as a binary variable. Instead, the authors define four AI-enabled supply chain practices that inform the exploration of adoption impacts. Third, while the proposed model encompasses the impacts of AI adoption on firms' operational and financial performance (as in previous studies), it also considers the impacts on firms' social and environmental performance. Although this framework was specifically developed for the study of supply chain management, it highlights that extending established frameworks to study AI adoption impacts requires developing a granular understanding of the specific uses of AI within firms.

2.6 Toward a drivers-dimensions-impacts model of AI adoption

To address the gaps identified in the technology adoption literature, we introduce the Drivers-Dimensions-Impacts (DDI) model. Building upon the foundational elements of the DOI theory and TOE framework (Panel A of Figure 1), the DDI model facilitates the analysis of the specific features of AI adoption and its resulting impacts (Panel B).

We posit that the impacts of AI deployment are shaped by the interplay between the four dimensions of adoption. The first step in understanding these impacts is to identify the specific AI technologies deployed and their key characteristics, as this determines the scope of potential effects. The impacts of AI adoption will be greatest when these technologies are applied to critical functions within a firm's business model, when AI is leveraged for developing industry-specific innovations, and when it is used pervasively throughout a firm's operations. Conversely, impacts will be lower when AI is applied to ancillary functions, when innovations are minimal, and when applications are isolated. To illustrate, in the case of a pharmaceutical company, the consequences of implementing an off-the-shelf AI resume screening tool to support recruiting will be significantly more reduced than that of deploying a custom AI system to accelerate drug discovery, optimize clinical trials, and enhance supply chain management.

To guide the exploration of the impacts of AI adoption the DDI model adopts the categories of the TOE framework due to their clarity, comprehensiveness, and extensive use in empirical literature. The impacts of adoption within the Technological context consist of the AI-powered innovations developed by firms. The impacts on the Organizational context include, among others, performance improvements (Mikalef & Gupta, 2021; Wamba-Taguimdje et al., 2020), development of new products and services (Goto, 2023; Zhang et al., 2021), substitution of labor Budhwar & Malik, 2023, introduction of new security vulnerabilities (Schreiber & Schreiber, 2024), and business model transformations (Corea, 2017). The impacts on the Environmental context include improvements in the coverage of client needs (Allioui & Mourdi, 2023), the replication of societal biases (Cubric, 2020), increased competitive pressures, and diminishing transparency and stakeholder trust (Chi et al., 2023). The DDI framework allows for flexible impact identification in each specific application, recognizing that the effects of AI adoption vary significantly across industries and contexts.

The four proposed dimensions of AI adoption exhibit varying degrees of relevance when examining impacts across Technological, Organizational, and Environmental contexts. For instance, studying a company's AI innovations is particularly pertinent to understanding impacts in the Technological context, while assessing adoption intensity across a firm's operations is key to comprehending implications in the Organizational context. However, to comprehensively explore the impacts of AI adoption, it is crucial to consider the interrelations among all four dimensions. Researchers must investigate the nuanced dynamics at play in each specific application, drawing on a deep understanding of the industry in question. For this reason, the graphic representation of the DDI framework does not include direct arrows between individual adoption dimensions and impact classes, emphasizing the importance of their collective consideration.

The DDI framework offers a dynamic perspective, as the static nature of traditional adoption models prevents them from adequately explaining the evolution of AI use over time (Mogaji et al., 2024; Polyportis, 2024). It can be applied iteratively, for understanding how the evolution of technological, organizational, and environmental factors affects the trajectory of AI use. Furthermore, it incorporates a feedback loop to capture how the impacts of AI deployment shape subsequent adoption cycles. Using an example from the banking industry to illustrate this, AI deployment in credit assessments might perpetuate biases against minorities, prompting regulatory changes that, in turn, reshape banks' adoption patterns.

Acknowledging the complex nature of AI and its diverse applications, the DDI framework is not intended as an all-encompassing explanatory model. Its primary goal is to assist researchers in navigating the intricacies of AI adoption by providing a structured method for organizing complex ideas and guiding empirical research endeavors. The framework was designed as a general and highly adaptable tool, following the spirit of the TOE framework and DOI theory. It can be applied using primary or secondary data. The level of access to data will be a key factor in shaping the empirical applications of the framework. Finally, it is important to note that this framework is not a static or definitive work. It should be expanded and refined as knowledge of AI adoption evolves. The next section presents the methodology designed to operationalize the application of this framework to study AI adoption in the ESG rating industry.

3.1 Data sources

This study pursued a data triangulation approach, a research strategy that combines data from diverse sources to enhance inference validity (Kern, 2018). First, we analyzed corporate communications. These are a key data source for studying AI adoption and its functional applications, as firms integrating cutting-edge technologies typically disclose such information (Calvino et al., 2022; Dahlke et al., 2024). We limited our analysis to publicly available information on ESG products from the sampled raters' websites.

Second, we analyzed job openings data. The demand for roles requiring AI-specific skills is a strong indicator of AI adoption, as deploying such technologies necessitates a highly specialized workforce (Alekseeva et al., 2021; Calvino et al., 2022). Consequently, the volume of AI-specific job postings serves as an indicator of the extent to which a firm is deploying AI technologies. Moreover, the skills required in job postings serve as a key data source for understanding the types of AI technologies that firms implement. Furthermore, job openings data complements the analysis of firms' official communications, which may be subject to bias (Pan et al., 2022).

We relied on job posting data provided by Lightcast™ (formerly Burning Glass Technologies). This private data provider gathers labor market data by scraping over 65,000 websites worldwide and claims to capture the near-universe of online job vacancies. Given that AI positions are predominantly posted online, their data boasts extensive coverage of this field (Squicciarini & Nachtigall, 2021). The database analyzed has global coverage and includes the following variables: job title, company, date, skill, skill category, and ISCO code.

Third, we analyzed patent documents, a well-established data source in technology adoption studies (Calvino et al., 2022). Patents offer a rich window into a company's technological capabilities, providing detailed descriptions of their innovations. Fourth, we analyzed research documents authored by scholars affiliated with the raters studied. Industry papers have recently proliferated in the field of AI (Hagendorff & Meding, 2023; Liang et al., 2024). These offer privileged insights into a firm's technological innovations. Furthermore, the contribution to prominent AI academic conferences and journals is an indicator that an organization is at the forefront of the development of the discipline. The data collection was performed in January 2024.

Our decision to rely exclusively on unobtrusive data sources was informed by three in-depth interviews with experts in AI and sustainability, who have extensive knowledge of the ESG rating industry. The sample included an AI researcher who is a former employee of one of the studied ESG raters, a sustainability manager from a multinational corporation, and a professor and researcher in the fields of business and AI. These experts indicated that due to the sensitivity surrounding AI use in the industry and the sector's low transparency, raters would likely be reluctant to disclose meaningful insights regarding their AI implementation through direct inquiries.

3.2 Sampling strategy

For this investigation, we decided to focus on the following seven traditional raters: Moody's, Bloomberg, LSEG-Refinitiv, Morningstar-Sustainalytics, MSCI, S&P Global, and Deutsche Börse-ISS-ESG. These firms were selected for three main reasons. First, the ESG rating market is highly concentrated among these companies (Antolín-López & Ortiz-de-Mandojana, 2023). Secondly, these are widely recognized for their global product distribution, extensive firm coverage, and robust research teams (Pagano et al., 2018). Finally, these raters act as de facto norm setters in the ESG rating industry (Pagano et al., 2018). Hence, we are confident that this sample allows us to offer a comprehensive outlook on the technological transformations unfolding in the sector. However, it is important to note that our bias toward industry leaders prevents us from adequately capturing AI adoption trends among smaller, niche, or local ESG raters.

For identifying relevant data on raters' corporate communications, we performed targeted searches on both their websites and Google. Our search string included the acronym ESG, raters' names, and a subset of the most common AI keywords detailed in the Supporting information (SI) 1. Subsequently, we scanned relevant documents and extracted passages referencing AI use, which were systematically recorded in a spreadsheet. To account for recent significant acquisitions of rating companies by the studied firms (Antolín-López & Ortiz-de-Mandojana, 2023), we integrated the acquired companies into our analysis. Specifically, Refinitiv data was integrated for LSEG, VigeoEiris data for Moody's, and Sustainalytics data for Morningstar-Sustainalytics.

Regarding job postings, our initial database contained over 16 million entries spanning January 2016 to March 2024. While the raw data encompassed all postings from the firms in our sample, it also included openings from irrelevant entities with similar names. To detect these, we manually examined company names. In most cases, we could filter unrelated firms based on their trade names (e.g., Morningstar Senior Living, S&P Landscaping). In those cases in which we could not know by a firm's denomination whether they were or not an ESG rater, we visited their websites. If we could not find a website, we checked the ISCO codes of their job postings, which allowed us to have a reasonably good idea of their field of activity. After filtering the observations corresponding to the raters studied, our final database contained 163.176 unique job postings, and 3.495.208 individual skills associated with these. To identify AI-specific job openings, we utilized a taxonomy of 250 AI skills developed by Lightcast and the OECD. All postings containing at least one of these skills were categorized as AI-specific. This approach allowed us to pinpoint 8.907 AI openings.

The patent search was conducted using Scopus since this database provides direct access to the world's five major patent offices (Scopus, 2023). For filtering AI-specific patents we followed the criteria developed by (Baruffaldi et al., 2020). According to their approach, a patent is considered AI-specific if it fulfills any of the following conditions: (1) It falls within the taxonomy of core-AI IPC codes detailed in the SI-2. (2) It falls within the taxonomy of AI-related IPC or CPC codes on SI-2, and it's abstract or claims sections contain at least one of the terms in the keyword list presented on SI-1. (3) It contains at least three of the keywords on SI-1 on its abstract or claims. We identified 52 AI-related patent documents, of which 33 correspond to granted patents, and 19 to pending applications.

To identify research publications authored by researchers affiliated with the firms sampled we combined three databases. First, we searched in Scopus, as this is one of the most comprehensive repositories of academic research (Singh et al., 2020). Second, we searched the arXiv repository, which has become one of the main outlets for the dissemination of cutting-edge AI research in recent years (Stanford HAI, 2023). Finally, we conducted searches on the ACL Anthology, a highly comprehensive repository of NLP research, hosting over 90.000 articles (ACL, 2024). The search string employed contained the name of each firm, its possible variations, and all keywords in SI-1. The criteria for including research documents in our sample was to include three or more AI keywords in their title, abstract, or body. Our search resulted in the identification of 124 research documents. We limited our search of both research documents and patents to 2019 since we are particularly interested in recent developments in the industry.

3.3 Text mining

Our study involved examining large amounts of unstructured textual data for filtering relevant documents and analyzing them. To perform these tasks we employed keyword search and keyword frequency analysis, two basic text mining techniques that have been extensively used to study corporate technology adoption (Cruciata et al., 2022; Gök et al., 2015; Héroux-Vaillancourt et al., 2020), including several AI-specific studies (Calvino et al., 2022; Dahlke et al., 2024; Zhong et al., 2023). First, we scanned the documents retrieved against our extensive AI keyword list, employing a custom Python script. This allowed us to filter those documents that met our selection criteria. Second, we retrieved the frequencies of AI keywords in the selected documents. This was essential for classifying them based on both their technological and functional domain.

3.4 Qualitative content analysis

This work employed the structuring content analysis technique developed by Mayring (2015) for filtering, synthesizing, and analyzing the particular aspects of the sampled documents relevant to answering our research questions. The categories of data to be extracted from the documents were defined a priori, as it is standard practice in structuring content analysis (Kohlbacher, 2006; Mayring, 2015). These categories included: mentions of AI use, leveraged technologies, and functional applications. The extracted data was organized on a spreadsheet to facilitate its analysis. Relevant passages were subsequently analyzed, and classified based on the specific technologies mentioned, their functionality, and their relevancy to sustainability assessments.

3.5 Quantitative analysis

We employed two complementary quantitative analysis approaches. First, we conducted a comprehensive statistical analysis of the collected data using Python, to identify patterns and trends. Second, we developed novel indexes to quantify two key constructs central to our research: adoption intensity and level of innovation. The indexes allowed us to operationalize these abstract concepts into measurable variables for further analysis.

4.1 Analysis of corporate communications

4.1.1 Overview

This sub-section presents the insights on raters' AI adoption obtained by analyzing publicly available documentation published on their websites. This key data source informed our understanding of the specific technologies employed by raters, their functional applications, and the intensity of adoption across firms. For our analysis, we mapped raters´ self-reported AI use to the core functions of the production of ESG ratings as described by (Hughes et al., 2021), namely, data collection, processing, analysis, and rating generation, plus the related function of controversy monitoring. Table 2 summarizes our findings, illustrating the extent of AI integration within each of the core ESG assessment functions across the examined firms.

TABLE 2. Reported AI use in core ESG assessment functions.

Firm	Data collection	Data processing	Analysis	Rating generation	Controversy monitoring	Intensity index
Morningstar-Sustainalytics	NLP (1)a		ML, NLP (1)	ML (1)	ML, NLP (1)	0.80
LSEG-Refinitiv	AI, NLP (2, 3)		NLP (2, 3)	AI, NLP (2, 3)	AI, NLP (2, 3)	0.80
MSCI	NLP (4)	AI (4)	AI (4)			0.60
Bloomberg		ML (5)	AI (5, 6)			0.40
S&P Global		ML, NLP (7, 8, 9)			AI, ML (7, 8, 9)	0.40
ISS-ESG					AI (10)	0.20
Moody's				ML (11)		0.20
Total	3	3	4	3	4	-

• ^a The documents referenced by the numbers in brackets are listed in SI-3.

The most relevant finding of this analysis is that all traditional raters report employing AI in core functions of their ESG assessments. Furthermore, five raters report employing AI in more than one core function. The use of AI is relatively homogeneous across the different functions of the rating process. However, as evident in Figure 2, the intensity of adoption varies considerably among firms.

4.2 Analysis of job posting data

4.2.2 Evolution of raters' AI talent demand

Our analysis evidences a turning point in the industry's demand for AI expertise in 2021. As shown in Figure 3, the number of AI openings increased gradually between 2016 and 2019—from 166 to 493. While 2020 saw a notable rise, driven primarily by Bloomberg's recruitment efforts, 2021 marked a true inflection point. The number of AI job postings tripled, soaring from 760 in 2020 to 2313 in 2021, and further peaked at 2595 the following year. Notably, this surge in AI talent demand from 2021 onward was pervasive across all rating agencies.

AI openings are a robust proxy of AI adoption since deploying these technologies requires a highly specialized workforce (Alekseeva et al., 2021; Calvino et al., 2022). Hence, the booming demand for AI expertise signals a substantial rise in raters' adoption of these technologies. In the first quarter of 2024, rater's AI job postings amounted to 483, indicating that the industry's demand for AI talent remains strong.

Unfortunately, accurately determining the number of AI openings destined for raters' ESG business lines is challenging, as this information is typically not specified in job descriptions. A total of 218 AI openings mentioned the acronym ESG in the job title, and 22 mentioned the terms Sustainable Finance or Sustainable Investment. However, these figures are likely to significantly underestimate the number of AI positions concerned with ESG assessments.

The volume of AI job postings varies considerably among firms. S&P Global and Bloomberg account for 33% and 28% of the total openings respectively. These are followed distantly by Moody's (15%), LSEG-Refinitiv (11%), Morningstar-Sustainalytics (7%), MSCI (4%), and ISS-ESG (1%). Interestingly, the companies leading the demand for AI talent also exhibit the highest proportion of AI openings relative to their total job listings. Bloomberg stands at 8%, followed by S&P (7%), Morningstar-Sustainalytics (6%), Moody's (4%), MSCI (4%), LSEG-Refinitiv (3%), and ISS-ESG (2%).

4.2.3 AI-specific skills required in raters' job postings

To achieve a nuanced understanding of the AI skills demanded by raters we combined Lightcast's skill clusters, which correspond to broad areas within AI expertise, alongside their granular AI skill taxonomy. The most remarkable aspect of the analyzed data is the prominence of ML skills in raters´ job postings. As illustrated in Figure 4, 58% of the skills demanded by raters fall within the ML cluster. The AI cluster comes second at 19%, followed by Neural Networks (13%), NLP (8%), CV (1%) and Robotics (1%). The skills most frequently required in raters' AI postings are broadly defined capabilities (e.g., ML, NLP), followed by popular AI/ML frameworks (e.g., Tensorflow), software libraries (e.g., Pythorch ML Library, Sci-Kit, Dask, Keras), functionally-defined AI subfields (e.g., text mining, SA, recommender systems), development platforms (e.g., AWS SageMaker, MLflow), algorithms types (e.g., decision trees, support vector machines, XGBoost), work methodologies (e.g., MLOps), and model training approaches (e.g., supervised/unsupervised learning).

4.3 Patent data analysis

4.3.1 Overview

This sub-section presents the insights on raters' AI adoption obtained from the analysis of their patent filings. Patent documents serve as a vital data source for gauging raters' innovativeness and understanding the practical applications of their AI-powered inventions. As presented in Table 3, our patent search resulted in the identification of 52 AI-related documents. Thirty-three correspond to granted patents, and 19 to pending applications. To structure our analysis, we classified patents into ESG-Specific, ESG-Relevant, and ESG-Unrelated. Additionally, we categorized the documents in our sample based on the core technologies they primarily employ, classifying these into three broad categories: ML, NLP, and CV.

TABLE 3. Raters' AI patent filings.

Company	ESG-specific	ESG-relevant	ESG-unrelated	Total	Granted/pending
S&P Global	5	20	3	28	16/12
LSEG-Refinitiv	1	10	4	15	10/5
MSCI	-	5	1	6	5/1
Bloomberg	-	-	2	2	2/0
Morningstar-Sustainalytics	-	1	-	1	0/1
Total	6	36	10	52	33/19

• Note: All patent documents analyzed are listed on SI-4.

As evident in Figure 5, S&P Global is the leading firm on AI patents, with 32 documents, followed by LSEG, with 15. The rest of the firms combined account for 8 documents. No AI patents from Moody's or ISS-ESG were identified in our search.

4.3.2 Technologies leveraged

The majority of the patents studied fall within the ML category (31). These are followed by a group of 20 documents that primarily leverage NLP, and by one CV patent. The prominence of ML in raters' patents is evident in Figure 6. The word cloud shows that the most frequent keywords in the documents examined include common ML algorithms (e.g., decision trees, support vector machines) and approaches for training ML models (e.g., Supervised/Unsupervised learning). These are followed distantly by common NLP functions (SA, NER) and neural network architectures. Succinct descriptions of the applications of these technologies in noteworthy patents are provided in the following sub-sections.

4.4 Analysis of research publications

4.4.1 Overview

This sub-section presents insights on raters' AI adoption based on the analysis of their research publications. These are a crucial data source for assessing their level of AI innovation, and for gaining insights into the functional applications of their R&D efforts. As presented in Table 4, our search of research publications resulted in the identification of 124 documents. Following the criteria used to classify patents, we categorized research documents according to their relevance for ESG assessments, and their AI subfield.

TABLE 4. Raters' AI research publications.

Company	ESG-specific	ESG-relevant	ESG-unrelated	Total	Innovativeness index
Bloomberg	1	81	19	101	0.43
S&P Global	2	4	2	8	0.31
Moody's	1	4	1	6	0.02
LSEG-Refinitiv	1	4	-	5	0.17
MSCI	1	1	-	2	0.07
Morningstar-Sustainalytics	-	1	-	1	0.01
ISS-ESG	-	-	-	0	0
LSEG-Refinitiv & S&P	-	-	1	1	-
Total	6	95	23	124	-

• Note: All research publications analyzed are listed on SI-5.

One of the most salient features of our sample is the prominence of Bloomberg in the studies retrieved. As shown in Figure 7, the overwhelming majority of the papers in the sample were published by this firm (101), followed distantly by S&P (8), Moody's (6), and LSEG-Refinitiv (5). No AI research publications from ISS-ESG were identified in our search.

4.4.2 Technologies leveraged

Most of the papers sampled fall within the field of NLP (92). These are followed by a heterogeneous group of ML documents (25) and by smaller clusters of speech recognition (4), and CV studies (3). The prominence of NLP is evident in Figure 8, the word cloud elaborated with the AI keywords in the research publications studied. Notably, common NLP functions (e.g., SA, NER, MT) and popular LLMs (e.g., BERT, T5) were mentioned in numerous publications.

4.5 Self-reported AI adoption intensity, talent demand, and innovation

Our analysis evidence that all firms studied have adopted AI technologies for producing their ESG ratings. However, our results show a significant heterogeneity among raters' reported intensity of AI integration, their AI talent demand, and their innovative output. The scatterplots in Figure 9 show the relationship between these variables.

Unsurprisingly, the firms boasting the highest innovative output tend to demand more AI talent (Plot 1). However, the relationship between raters' self-reported AI use, their talent demand, and innovation intensity is more nuanced. Intriguingly, the firms leading the AI talent demand (Plot 2) and innovative output (Plot 3) reported comparably low use of AI in their ESG ratings. Conversely, those who reported the highest intensity of AI use exhibit only moderate levels of AI labor demand and innovative output. This apparent disconnect could respond to several factors, including limitations in the definition and scope of our indicators, or differing corporate strategies. However, it could also be a sign of the lack of methodological transparency that characterizes the industry.

5.1 Characterizing AI adoption in the ESG rating industry

Our results reveal a significant and growing level of AI adoption in the ESG rating industry, challenging previous academic (Hughes et al., 2021; Telukdarie et al., 2024) and practitioner literature (Briere et al., 2022; Venkatesh, 2022) that characterized traditional ESG raters as having minimal AI use. The evidence from multiple data sources converges to show the increasing pervasiveness of these technologies in the sector. Raters have not only integrated AI into their ESG rating processes but have also emerged as sophisticated innovators, developing cutting-edge AI solutions tailored to their sustainability assessments.

The analysis of the DDI framework's four dimensions of AI adoption offers a comprehensive perspective on the evolving technological landscape of the ESG rating industry. Beginning with the Intensity dimension, our examination of raters' websites reveals that all studied firms employ AI in core functions of their ESG rating process, with the majority leveraging AI technologies across multiple core functions. This intense integration is further corroborated by our analysis of job postings, which shows a surge in raters' demand for AI talent since 2021.

The analysis of the Specific Technologies and Functional Applications dimensions evidenced that raters extensively leverage NLP to improve the efficiency of key functions of their ESG assessment processes. In particular, the documents retrieved from raters' websites show that these employ name entity recognition algorithms to support data collection, and classification and sentiment analysis algorithms to power their data analysis. These technologies have enabled huge improvements in data timeliness, allowing raters to monitor corporate controversies in near real-time. Additionally, the patents and papers examined reveal that raters have developed NLP innovations to expand the scope of their analysis to new data sources, with a focus on social media. Furthermore, raters have pioneered promising machine translation innovations applied to local and minority languages, which could be leveraged to expand their coverage to new geographies.

These results are in line with Hughes et al. (2021), who highlighted AI's potential to broaden the scope of the data that goes into ESG ratings beyond the content of corporate reports and mainstream media outlets. This expansion could lead to a more diverse range of perspectives being represented within these metrics. Moreover, our results are consistent with recent studies that showed the potential of NLP technologies to improve the efficiency of complex environmental and social assessments (Adelakun et al., 2024; Curmally et al., 2022; York & Bamberger, 2024).

Yet, AI is not only contributing to streamlining traditional sustainability assessment methods. Our analysis of the Innovation dimension suggests that raters are extensively leveraging the predictive power of ML, significantly expanding their risk assessment capabilities. The ML patents and papers reviewed showcase innovations that enable raters to perform novel types of risk analysis. Notable examples include models for forecasting facility-level climate risks, estimating the effects of supply chain disruptions, or predicting the impact of corporate controversies. Furthermore, raters are developing ML applications with the potential to increase the accuracy of their assessments substantially. Prominent examples include models for estimating missing emissions data or detecting unknown supply chain relationships. Raters' increasing reliance on these technologies is reflected in the prominence of ML-related skills within their job openings. These findings are in line with previous works that highlighted the positive impacts of ML adoption on the accuracy of complex sustainability (Bassey et al., 2024; Ye et al., 2020) and financial assessments (Perumalsamy et al., 2022; Tolulope & Opeyemi, 2024).

We attribute the significant divergence of our findings from prior literature, which portrayed traditional ESG raters as lacking significant AI integration, to two key factors. First, the timing of our study is crucial. Our job opening analysis reveals a surge in AI integration within the industry from 2021 onwards. Consequently, studies conducted before this pivotal year encountered a vastly different technological landscape. Second, we are confident that our triangulation of diverse data sources enabled us to gain more in-depth insights into raters' AI adoption, as compared with earlier works that relied exclusively on corporate disclosures.

5.2 Examining the impacts of AI adoption in the ESG rating industry

The DDI framework's impact categories provide a valuable lens for exploring the potential and realized effects of AI integration, drawing on the insights gained from analyzing its four adoption dimensions. In the Technology sphere, the AI innovations developed by raters could catalyze further advancements in climate risk prediction, supply chain analysis, and social and environmental data collection. At the Organization level, NLP integration is already enabling raters to collect and analyze data more efficiently and agilely, while ML is facilitating the development of new products. Regarding the Environment, the AI-driven improvements in rating accuracy, timeliness, and coverage could directly benefit investors by enhancing their risk management capabilities. Indirectly, these advancements could also benefit the communities affected by rated companies and the natural environment through increased data representativeness and enhanced corporate accountability.

However, AI adoption in the ESG rating industry might also have negative consequences. At the Organization level, the integration of NLP could lead to significant levels of job displacement, as these technologies automate tasks traditionally performed by ESG analysts, such as the analysis of corporate reports and news. At the Environment level, the potential negative impacts are many. To begin, the biases of ESG ratings related to company size, industry, and geography are well documented (Drempetic et al., 2020; Gyönyörová et al., 2023; Liang & Renneboog, 2020). Raters tend to be more favorable toward larger firms, which have more resources to invest in their sustainability reporting, as well as firms operating from developed countries (Liang & Renneboog, 2020). Without appropriate mitigation strategies in place, these biases will likely be perpetuated through the training of algorithms.

Furthermore, AI models have built-in blind spots. For instance, these might overlook some of the ESG risks associated with companies operating in underrepresented industries or regions. If left unchecked, these factors could undermine ratings' fairness and accuracy. While precise data about ESG ratings' AI biases and blindspots remains limited, asset managers should exercise caution when employing these metrics to evaluate potentially affected investments, such as companies based in developing countries or operating in underrepresented industries. To mitigate these issues asset managers could resort to supplementary sources of information (e.g., direct engagement with companies and their stakeholders).

Additionally, AI-powered sustainability assessments are likely to influence the behavior of rated companies, similar to how listed firms have modified their communication strategies in response to NLP-supported trading (Cao et al., 2023). This could lead to changes in sustainability reporting and new forms of greenwashing. For instance, companies could try to game the system by crafting reports designed to be scored positively by algorithms. Furthermore, if sustainability reports become increasingly optimized for machine analysis, these may become less accessible to human readers, a trend already observed in financial reporting (Call et al., 2023). To address these concerns, leading institutions developing sustainability reporting standards, such as the Global Reporting Initiative, could play a crucial role in establishing guidelines for safeguarding the human readability of reports while accommodating AI-driven analysis.

Finally, If the pattern found in previous studies persists in this new context (Christensen et al., 2022), the increased data volume enabled by AI could lead to greater ESG rating divergence. This could contribute to exacerbating the existing confusion among rated companies and investors about these metrics.

5.3 A blackbox within a balckbox: The urgent need for AI transparency in ESG ratings

This study highlights a critical need for increased transparency on AI usage. While all raters studied mentioned the adoption of AI in their websites, the information provided lacked in most cases sufficient depth and specificity. Moreover, the firms leading the demand for AI talent, patent filings, and research publications, reported a comparably low level of AI use in their ESG ratings. This discrepancy raises concerns about the industry's transparency concerning its implementation of AI technologies.

Previous literature has characterized ESG ratings as a blackbox, due to raters' little transparency regarding their data sources and methodologies (Abhayawansa & Tyagi, 2021). The increasing deployment of AI in the industry, and the limited information disclosed about it threaten ESG ratings to become a blackbox within a blackbox. Stakeholders could find it increasingly difficult to understand what data goes into AI models, and how these analyze it and process it into ratings. The addition of a further layer of opacity to the production of ESG ratings could exacerbate confusion around these metrics and undermine stakeholders' trust.

Fostering trust in ESG ratings will require an increased degree of openness from raters. Stakeholders need to comprehend the specific technologies employed and their applications within the rating process. Furthermore, transparency around the data used to train AI models, their biases, and the mitigation strategies adopted by raters is crucial. Moreover, the explainability of the technologies used is fundamental to the industry's legitimacy and credibility. Stakeholders must be able to understand how different features influence the outputs of the models employed. Finally, transparency about the extent of human oversight in AI applications is vital.

Regulations introducing industry standards for the disclosure of information on AI usage will be instrumental in advancing transparency. The EU's provisionally agreed “Proposal for a Regulation (…) on the Transparency and Integrity of ESG Rating Activities” takes a positive step by requiring raters to disclose “the use of AI in the data collection or rating/scoring process including information about current limitations or risks of using AI” (PE759.068, 2024, 77). Yet, it remains uncertain whether the extent of detail in raters' disclosures will be sufficient to provide meaningful insights into their AI use.

Direct engagement of key stakeholders with ESG raters could also be a crucial driver for enhancing the industry's AI transparency. Rated corporations, as the primary data providers, and asset managers, as the main end-users of ESG ratings, are uniquely positioned to advocate for this change. Networks and organizations aimed at convening investors and corporations to foster sustainable development, such as the UN Global Compact, the World Business Council for Sustainable Development, and the Principles for Responsible Investment's signatories, could act as catalysts for initiatives seeking increased AI transparency.

5.4 The DDI framework: A versatile tool for AI adoption research

Our analysis highlights the value of the DDI framework for characterizing AI adoption and gaining insights into its potential impacts within a specific sector. Despite the opaque nature of the ESG rating industry, the proposed operationalization allowed us to comprehensively assess the current state of AI adoption among sustainability raters. The examination of the four dimensions of AI adoption allowed us to identify multiple potential impacts affecting raters' Technological, Organizational, and Environmental contexts. Furthermore, this approach facilitated the exploration of implications for policy and practice stemming from the identified impacts.

The DDI framework is highly adaptable and can be applied across diverse sectors. However, its implementation will vary significantly depending on researchers' access to data. Our operationalization based on unobtrusive research methods is particularly well-suited for contexts of limited access to primary data. This approach is particularly suitable for highly concentrated industries like banking, aerospace, and pharmaceuticals. Given the high sensitivity of AI use in these sectors, direct engagement with firms is challenging. Hence, the examination of publicly available data sources is essential for comprehending the technological transformations unfolding within these industries. Our framework can also be applied to more fragmented industries (e.g., hospitality, construction). In these cases, researchers must employ rigorous sampling methods to ensure representativeness when selecting study participants. Nevertheless, direct data collection techniques such as surveys and in-depth interviews will be feasible, which can lead to more in-depth insights.

We believe the multi-dimensional lens offered by the DDI framework is a powerful tool for both academics and policymakers, helping to structure the complexity of AI adoption across industries and analyze its implications. Moreover, we are confident that the proposed model offers a strong complement to the DOI theory and the TOE framework, by expanding the analytical focus beyond the drivers of adoption, facilitating the characterization of AI integration and the analysis of its impacts. However, we recognize that the DDI framework is not a finished work, and should be further refined and expanded upon in future studies.