[Japan] Differentiation Points in Disclosure Documents Revealed by Natural Language Processing
Building a Scoring Model Based on Metrics such as ROIC Disclosure and Management Commitment
Executive Summary
Investor relations (IR) materials that investors and analysts find useful tend to share common characteristics. However, the evaluation criteria are diverse, and it is difficult to clearly define the truly important ones. To address this challenge, we conducted a natural language processing (NLP) analysis of earnings presentation materials. The analysis was performed at three levels of granularity: word-level, slide-level, and document-level. For each, we examined the data against evaluation axes relevant to IR disclosure quality.
Our findings suggest that three factors are particularly important in the evaluation of IR materials:
Disclosure of key KPIs such as ROIC
Detailed and comprehensive segment-level disclosures
Clear articulation of management goals and commitment
Based on this analysis, we are offering companies free consultation and scoring services for their IR materials. If interested, please contact us via the link below.
Data Coverage
The dataset includes 472 companies from the TOPIX 500 index for which we were able to extract text from the most recent full-year earnings presentation documents (Japanese only). Companies without such disclosures or whose PDFs did not allow text extraction were excluded.
Dependent Variable: “High-Quality IR Company”
Our dependent variable is a dummy indicator showing whether a company is regarded as having high-quality IR practices. Specifically, it takes the value of 1 if:
The company has been recognized in Extel IR awards (when disclosed), or
It was selected as a “Best Disclosure Company” by the Securities Analysts Association of Japan (one per sector, 2024).
Otherwise, the variable is 0. For simplicity, we refer to these groups as “high-quality IR companies” and “other companies.”
While the evaluation of IR quality is not based solely on earnings presentation materials, this analysis assumes that companies recognized as “high-quality IR firms” will also be evaluated positively from the perspective of their presentation materials. Moreover, by focusing on these materials, we expect to uncover practices that are distinctive to high-quality IR firms.
Of the 472 companies analyzed, 17 were classified as high-quality IR firms. These are: Hitachi, Mitsui & Co., Sumitomo Mitsui Financial Group, Tokio Marine Holdings, SoftBank, Ajinomoto, Chugai Pharmaceutical, Bridgestone, Nippon Steel, TDK, Mitsui Fudosan, Isetan Mitsukoshi Holdings, Mitsui Chemicals, Nomura Research Institute, Idemitsu Kosan, Kawasaki Kisen Kaisha, and Konami Group.
It should be noted that due to data constraints, some companies recognized by the Securities Analysts Association of Japan as “Best Disclosure Companies” were not included in this analysis simply because they were not part of the 472-company universe. In addition, Extel award results are included only when the company itself disclosed this information and it was identifiable through an online search.
Explanatory Variable Group 1: Word-Level TF-IDF Data
For the word-level analysis, the text contained in each company’s presentation materials was segmented into morphemes (the smallest units of meaning, hereafter simply referred to as “words”). This segmentation was performed using the morphological analysis engine MeCab.
We then applied a two-stage filtering process to the extracted words:
Frequency filter: Only words that appeared in at least 10% of the 472 companies’ presentation materials were retained. This yielded 1,568 words.
Manual filter: Three categories of words were excluded:
Words with no analytical significance (e.g., “various,” “main”)
Sector-specific terms (e.g., “resin,” “store”)
Words tied to particular business exposures that cannot easily be changed by corporate effort (e.g., “Vietnam,” “Americas”)
Since the purpose of this study is to provide insights that issuers can use to improve disclosure through their own efforts, we excluded not only meaningless words but also those tied to factors beyond a company’s control. After this filtering, 387 words remained.
For each issuer’s presentation text, we applied TfidfVectorizer, which vectorizes based on word frequency and rarity. The resulting TF-IDF values serve as proxy variables, indicating the relative importance of each word within a company’s presentation materials.
Explanatory Variable Group 2: Quantitative Data
In addition, we calculated basic quantitative measures for the 472 companies’ presentation materials, such as total character count. This allowed us to test whether companies with more slides or longer text tended to be evaluated more positively. Since preliminary hand checks suggested potential significance, we also included the ratio of English characters within Japanese presentation materials.
Explanatory Variable Group 3: Slide Classification Data
Next, we constructed slide-level analytical data to examine whether the types of slides included in an issuer’s presentation are linked to IR evaluation. For example, we wanted to verify whether including slides on ESG topics contributes to stronger evaluations.
To categorize slides systematically, we vectorized the text from each slide using the model oshizo/sbert-jsnli-luke-japanese-base-lite. This model has 768 dimensions, 133M parameters, and achieves a relatively high score of 0.811 on the “JSTS valid-v1.1” benchmark dataset.
We then applied hierarchical clustering based on cosine similarity, treating the 0–1 distance between sentence vectors as the metric. This enabled us to group slides according to similarity. As with the word-level analysis, however, certain slides were highly idiosyncratic to individual companies. Because such slides are of limited analytical value, we excluded slide groups that appeared in 40 or fewer companies’ presentations (i.e., if a group was represented in the decks of fewer than 41 companies, it was dropped from analysis).
As a result, 38 slide groups remained. We reviewed samples from each group and assigned descriptive labels.
The largest group was “Positive-tone performance and plans” (6,871 slides).
The second largest was “Tables showing performance figures” (5,789 slides).
The third was “Disclaimers, cover pages, and tables of contents.”
We then applied manual filtering to remove:
Slide groups clearly irrelevant to IR evaluation (e.g., tables of contents, slides with only company logos)
Slide groups tied too strongly to specific sectors or companies (e.g., pharmaceutical-related slides, groups containing only Asics’ slides)
After this filtering, 20 slide groups remained. For each of the 472 companies, we created dummy variables indicating the presence (1) or absence (0) of each slide group in their presentation materials.
Explanatory Variable Group 4: Qualitative Question Scores
Finally, we constructed a dataset to capture elements not fully addressed by the previous explanatory variables. These were based on three qualitative questions that we considered important:
Change over time: Do the presentation materials make it possible to discern changes relative to past disclosures?
Management accountability: Do the materials convey a sense of responsibility if targets are not achieved?
Granularity of targets: Are indicators and their targets presented at a detailed level, such as by segment?
Using these three evaluation axes, we applied Google’s “gemini-2.5-flash” model to the full text of each company’s presentation materials, assigning a score from 0 to 100.
For questions 1–3, Gemini’s scoring tended to produce high averages (93.2, 81.1, and 90.3, respectively). However, we also observed occasional extreme outliers, with scores as low as 0 or 5. To address this issue, we converted each company’s score into a percentile value by ranking the scores across the 472 companies and dividing by the universe size.
Summary of All Explanatory Variables
We then compiled summary statistics for each group of explanatory variables, as outlined below.
The high-quality IR firm variable is defined as a dummy, taking the value of 1 for high-quality IR companies and 0 otherwise. Similarly, the slide classification variables take a value of 0 if a given slide type is not included in the materials. The word-level TF-IDF variables also default to 0 when the relevant term does not appear. Consequently, these three groups of variables contain a larger proportion of zeros compared to others, and their medians are therefore 0.
Analysis and Results
Methodology
We applied L1-regularized logistic regression to the dataset described above. L1 regularization simultaneously performs variable selection and prevents overfitting. By adding an “L1 penalty term” to the regression, unnecessary explanatory variables are forced to have coefficients of zero, simplifying the model and enhancing interpretability. In our case, the dataset included 472 companies and 413 explanatory variables (features), which is relatively large compared to the sample size. Thus, incorporating an L1 penalty was deemed appropriate.
Moreover, because the dependent variable is binary (0 = other companies, 1 = high-quality IR companies), logistic regression was chosen. This approach allows us to estimate the probability of a company being classified as high-quality and to score their earnings presentation materials accordingly.
The regularization parameter (C) in the L1 regression was set at 1.1. Smaller values of C yield simpler models but reduce explanatory power, while larger values increase the risk of overfitting. Following the purpose of this study, we adjusted the parameter such that 15 explanatory variables remained, which led to the final choice of C = 1.1.
Results
The explanatory power of the model, as indicated by its scoring metrics, was as follows:
The most intuitive metric is Nagelkerke R², which is an extension of the coefficient of determination (R²) for logistic regression. Interpreting this value broadly, our model can explain roughly 20% of the probability that a company will be classified as a high-quality IR firm based on its earnings presentation materials.
As a result of the analysis, 15 explanatory variables (features) retained non-zero coefficients. Among these, seven variables contributed positively, which we denote as f1–f7 for convenience.
Despite the considerable data constraints, the results highlighted the importance of ROIC, and most of the explanatory variables selected were highly convincing.
While the analysis also identified eight negatively contributing variables, we chose not to present them here, in line with the purpose of this paper: to clarify the elements that make for effective earnings presentation materials. Unlike the positively contributing variables, the negatively contributing ones offered little practical insight.
In this analysis, only 17 out of 472 companies were classified as high-quality IR firms. Accordingly, the dependent variable took the value of 1 for just 17 cases, and 0 for the remaining 455. As a result, the negatively contributing variables tended to be those that simply had non-zero values on average, without providing meaningful differentiation. Put differently, because most companies were classified as 0, the negative contributors merely reflected features that are “commonplace” in presentation materials, rather than attributes that distinguish high-quality IR practices.
Interpretation of Results
ROIC and Segment-Level Performance Disclosure: As indicated by explanatory variables f1 and f2, companies where the term ROIC appears with high importance in presentations, as well as those that include slides disclosing segment-level performance figures, were evaluated more positively.
Bridgestone ranked highest in terms of the TF-IDF score for ROIC. In its relatively compact 22-page presentation, the word ROIC appeared nine times, demonstrating a clear focus on this key KPI—an approach that left a favorable impression.
Kawasaki Kisen Kaisha (K Line) was another example of a high-quality IR firm. Its presentation included slides disclosing segment-level performance data, with quarterly figures and key points clearly summarized.
Management and Accountability: As indicated by explanatory variables f3 and f4, earnings presentation materials that emphasize rigorous management of performance risks and KPIs, and that clearly convey a sense of managerial accountability, are characteristic of high-quality IR companies.
(Note: In this context, “management” refers not to “executives” as individuals but to the act of managing itself—suggesting that both thorough oversight and executive commitment are key factors.)
Mitsui & Co. is a strong example on these points. Its full-year presentation, titled “Medium-Term Management Plan 2026 Progress / FY2026 Business Plan,” communicates a strong intent to manage various aspects such as risk, portfolio, and balance sheet. In addition, its section on “Initiatives for Sustainable Enhancement of Corporate Value” demonstrates concrete actions toward improving ROE and reducing the cost of capital, aligned with stated objectives. Based on such content, it is likely that Gemini scored the company highly in terms of “conveying management accountability when targets are not achieved.”
Comprehensiveness of Materials and Use of English: As shown by explanatory variables f5 and f6, companies with more slides or more comprehensive appendices also tend to be recognized as high-quality IR firms.
An especially interesting finding comes from variable f7: despite analyzing Japanese-language presentation materials, the proportion of English text within them was positively associated with high-quality IR firms.
This likely reflects two patterns:
Intentional visibility for overseas investors (e.g., adding English for accessibility), and
Routine use of English in KPIs or job titles (e.g., referring to “CFO” instead of “Chief Financial Officer” in Japanese).
In other words, companies that are consistently mindful of communication with overseas investors and global offices may naturally end up with a higher proportion of English in their materials.
While simply adding more English text does not necessarily improve presentation quality, the fact that this variable emerged as significant is noteworthy.
Mitsui Fudosan exemplifies this characteristic. Its 92-page presentation provides rich information, including long-term performance trends and market data. Particularly rare and well-received is the inclusion of extended time-series data on non-financial indicators. Although the company separately prepares a full English version of its presentation, even the Japanese version incorporates bilingual elements—presumably to facilitate understanding by overseas investors.
Conclusion and Future Challenges
In conclusion, this analysis was able to extract the key characteristics of earnings presentation materials prepared by high-quality IR companies. Specifically, such firms tend to:
Disclose ROIC and other critical KPIs,
Provide detailed segment-level information, and
Convey management’s clear commitment to their goals.
In addition, the sheer number of slides and the presence of English text also emerged as meaningful features.
That said, the study has several limitations. First, the dependent variable—whether a company is recognized as a high-quality IR firm—was somewhat subjective, as it reflects the judgment of a limited number of analysts and investors. Second, the use of text analysis meant that the number of candidate explanatory variables was very large (413 features). Nevertheless, the fact that the seven positively contributing variables identified were almost all intuitively convincing is a surprising and encouraging outcome.
At the same time, a number of challenges remain. For example, the qualitative question scores in Explanatory Variable Group 4, which relied on Gemini’s assessments, proved problematic—two out of the three questions were ultimately discarded by the model. Furthermore, important KPIs such as WACC and Cost of Equity, which we regard as critical, did not remain as explanatory variables. This may be because disclosure on these metrics is still insufficient.
While this analysis focused on 472 companies within the TOPIX 500, we successfully built a system that could be applied across all listed companies to generate insights on how IR materials might be improved. Looking ahead, we also aim to extend the same framework by using valuation indicators as dependent variables, in order to visualize how qualitative aspects of a company’s disclosure impact valuation.
Services We Can Provide in Relation to This Study
Based on the findings of this report, we are able to offer the following services to both issuers and investors:
For Issuers
Free services: We provide issuers with evaluation scores of their presentation materials as well as consultations regarding IR practices.
Paid services: Beyond scoring, we can give concrete recommendations on slides that should be added to presentation materials and assist in calculating KPIs that should be highlighted.
For Investors
While this study focused on companies within the TOPIX 500, we are preparing to build a database covering all TSE-listed companies.
In the next phase, we plan to analyze the impact of qualitative disclosure features on valuation by using sector-relative PBR as the dependent variable.
We can also conduct customized screening on a paid basis. For example:
Identifying companies with slides that communicate a message similar to a particular page of another company’s presentation.
Searching for companies whose slides emphasize themes such as “management focused on ROIC.”
We have constructed a database of the vectorized slide data used in this study. The database itself can be made available for purchase upon request.
We would be delighted if you subscribed for updates.
Note: The services described above may be discontinued without prior notice. Our role is strictly limited to providing information. We do not engage in activities that would constitute solicitation under Japan’s Financial Instruments and Exchange Act, such as making investment recommendations or presenting target stock prices.