Please rotate your device to landscape mode to view the charts.

A Picture Is Worth a Thousand Words: Measuring Investor Sentiment by Combining Machine Learning and Photos from News

Journal: Journal of Financial Economics

Date: 20220401

Author: Obaid, Khaled; Pukthuanthong, Kuntara

Abstract:
By applying machine learning to the accurate and cost-effective classification of photos based on sentiment, we introduce a daily market-level investor sentiment index (Photo Pessimism) obtained from a large sample of news photos. Consistent with behavioral models, Photo Pessimism predicts market return reversals and trading volume. The relation is strongest among stocks with high limits to arbitrage and during periods of elevated fear. We examine whether Photo Pessimism and pessimism embedded in news text act as complements or substitutes for each other in predicting stock returns and find evidence that the two are substitutes.

Link: Google Scholar

Background and Context

Research Problem

Traditional investor sentiment measures rely on text analysis, but visual content in news may provide additional insights about market sentiment that text alone cannot capture.

Methodology

The study applies machine learning (Google Inception v3 model) to classify sentiment in over 148,823 Wall Street Journal news photos between 2008-2020, creating a daily Photo Pessimism Index.

Innovation

This is the first study to use machine learning for large-scale classification of news photo sentiment to predict market returns, achieving 87.1% accuracy in photo classification.

Impact of Photo Pessimism on Market Returns Shows Clear Pattern of Reversal

  • High photo pessimism predicts lower next-day market returns (-4.2 basis points)
  • The effect reverses over the following days, showing complete reversal within a week
  • Pattern supports behavioral theory that sentiment temporarily moves prices away from fundamentals

Performance of Portfolio Strategies Based on Photo Pessimism

  • Trading strategy based on photo sentiment outperforms text-based and market index strategies
  • PhotoPes strategy generates 5.8 basis points in daily excess returns
  • Shows economic significance of photo sentiment in predicting market movements

Impact Strengthens During High Fear Periods

  • Photo pessimism has 2.8x stronger effect during periods of market fear
  • Impact reaches -10.3 basis points during high fear vs -3.7 basis points in normal times
  • Suggests photos are particularly effective at conveying negative sentiment during market stress

Sharpe Ratios of Different Trading Strategies

  • Photo-based strategy achieves highest risk-adjusted returns with Sharpe ratio of 0.052
  • Outperforms both text-based strategy (0.032) and market index (0.036)
  • Demonstrates practical value of photo sentiment for investment decisions

Out-of-Sample Predictive Performance

  • PhotoPes shows significant out-of-sample predictive power across major market indices
  • Highest R² of 0.302% for SPY ETF demonstrates robust predictability
  • Results hold after controlling for transaction costs and market conditions

Contribution and Implications

  • First study to demonstrate that machine learning can effectively extract market-relevant sentiment from news photos
  • Shows photos and text are substitutes in conveying market sentiment, with photos being particularly effective during periods of market stress
  • Provides practical trading strategies that generate significant risk-adjusted returns using photo sentiment

Data Sources

  • Returns impact chart based on Table 2 Panel A coefficients for VWRETD index
  • Strategy performance chart based on Table 7 Panel A summary statistics
  • Fear period impact chart based on Table 6 coefficients comparing high/low fear periods
  • Sharpe ratio chart based on Table 7 Panel A performance metrics
  • Out-of-sample R² chart based on Table 9 predictive performance statistics