Revolutionising Financial Portfolio Management: The Non-Stationary Transformer’s Fusion of Macroeconomic Indicators and Sentiment Analysis in a Deep Reinforcement Learning Framework

Abstract

In the evolving landscape of portfolio management (PM), the fusion of advanced machine learning techniques with traditional financial methodologies has opened new avenues for innovation. Our study introduces a cutting-edge model combining deep reinforcement learning (DRL) with a non-stationary transformer architecture. This model is designed to decode complex patterns in financial time-series data, enhancing portfolio management strategies with deeper insights and robustness. It effectively tackles the challenges of data heterogeneity and market uncertainty, key obstacles in PM. Our approach integrates key macroeconomic indicators and targeted news sentiment analysis into its framework, capturing a comprehensive picture of market dynamics. This amalgamation of varied data types addresses the multifaceted nature of financial markets, enhancing the model’s ability to navigate the complexities of asset management. Rigorous testing demonstrates the model’s efficacy, highlighting the benefits of blending diverse data sources and sophisticated algorithmic approaches in mastering the nuances of PM.

1. Introduction

Portfolio management (PM) is a multifaceted domain where investors strive to optimise financial asset returns to achieve long-term goals. It can be differentiated into passive or active management, whereby each branch adopts unique strategies that fulfil these objectives. Active PM [1,2], characterised by dynamic trading to secure higher profits, contrasts with the conservative nature of passive PM, which aims to mirror market indices for steady, long-term gains. Prevailing practices in active PM often rely on predefined trends, which increasingly fall short against the complexities of volatile financial markets. This limitation has catalysed a shift toward incorporating advanced machine learning and artificial intelligence technologies [3,4]. Reinforcement learning (RL), renowned for its dynamic adaptability to market changes, has been applied in various PM models, such as iCNN [5], EIIE [6], SARL [7], AlphaStock [8], and GPM [9]. These models utilise historical asset prices and external information such as financial news, extracting features to guide portfolio rebalancing and decision-making. However, PM still grapples with challenges that include the extraction of non-stationary temporal sequential features, integrating macro- and microeconomic knowledge, and diversifying portfolios to minimise risk exposure.

To tackle these challenges, our study introduces a unique algorithmic approach that combines a non-stationary transformer with deep reinforcement learning algorithms (NSTD). This innovative approach is designed to reconstitute non-stationary features post-stationary processing of time-series data, merge multimodal information from diverse sources, and enrich portfolio diversity to mitigate concentration risks. The proposed model integrates macroeconomic indicators and sentiment scores from financial news into the non-stationary transformer framework, enhancing its ability to navigate the complexities of PM with data heterogeneity and environmental uncertainty. The model’s enhanced performance, validated through rigorous experimentation, demonstrates the efficacy of combining diverse knowledge sources for PM.

This paper is structured as follows: Section 2 reviews related work in PM, focusing on the evolution of active and passive strategies and the role of machine learning in this domain. Section 3 elaborates on our methodology, detailing the integration of macroeconomic features and sentiment scores from news data into the non-stationary transformer and its synergy with DRL. Section 4 and Section 5 present our experimental setup and findings, offering a comparative analysis of the NSTD model against traditional and modern deep-learning financial strategies. Finally, Section 6 concludes the paper, summarising our key findings and suggesting avenues for future research in PM.

2. Related Work

Portfolio management (PM) is primarily divided into passive and active strategies. The passive approach focuses on replicating the performance of specific indices. This approach is exemplified in studies such as [10], which explore integrating environmental, social, and governance (ESG) factors into passive portfolio strategies. Additionally, [11] discusses dynamic asset allocation strategies for passive management to achieve benchmark tracking objectives, whereas [12] emphasises the evaluation methodologies crucial for aligning passive portfolios with their target indices. This contrasts with the work on active portfolio management, which seeks maximal profit independent of any specific index. Traditional active PM strategies, as categorised by [13], include equal fund allocation, follow-the-winner, follow-the-loser, and pattern-matching approaches. However, these strategies need more adaptability to accommodate real-world financial markets’ unpredictable dynamics.

The evolving financial landscape and increasing data availability have highlighted the limitations of traditional PM methods. This realisation has spurred a transition towards advanced machine learning technologies. The emergence of DRL, combining deep learning (DL) and reinforcement learning (RL) [14,15,16], represents a significant advancement in addressing PM’s inherent challenges. DRL synergises the representational capabilities of DL with RL’s strategic decision-making, which is ideal for the volatile finance sector. Early explorations showcased neural networks’ potential in predicting market behaviour (e.g., [4,17]). The subsequent integration of RL, as seen in [5,18], enhanced decision-making capabilities in PM models. Furthermore, the time-series nature of financial data has led to the adoption of transformer mechanisms, as illustrated by [19], to unravel temporal information. This approach has been refined in subsequent studies that aim to reduce the computational complexity of self-attention mechanisms (such as [20,21]). Further advancements, including Autoformer [22] and Pyraformer [23], have significantly contributed to robust time-series analysis and forecasting.

The non-stationary transformer framework, introduced in [24], tackles the challenge of non-stationarity in time-series data, a critical limitation of conventional Transformers. The framework, integrating series stationarisation and de-stationary attention modules, is specifically designed to enhance the handling of non-stationary data. This unique module optimises the processing by dynamically adjusting its focus, enabling the transformer to adapt more effectively to varying data patterns, thereby enhancing transformer performance across diverse forecasting scenarios. This novel approach aligns well with reinforcement learning algorithms, offering enhanced capabilities for PM. The framework’s ability to adapt to non-stationary data makes it particularly relevant to the challenges and complexities encountered in passive and active PM.

3. Methodology

This study enhances portfolio management by integrating DRL with macroeconomic modelling, utilising insights from macroeconomic factors and sentiment analysis from news data. The proposed method employs a non-stationary transformer, as shown in Figure 1, further augmented by a model-free, off-policy actor-critic algorithm - the deep deterministic policy gradient (DDPG) algorithm [25]. In this framework, “X_data” represents the historical stock data, “X_news” denotes the sentiment-analysed news information, and “X_macro” signifies macroeconomic indicators, such as GDP, CPI, and unemployment rates. These diverse data streams are concatenated and normalised before input into the transformer. Within the transformer, “Q”, “K”, and “V” denote the transformed “queries”, “keys”, and “values”, respectively, in the self-attention mechanism. These components are critical for adapting the model to the non-stationarity in financial time series by recalibrating attention in response to the dynamic changes in the data. The actor and critic modules in the DDPG framework use the transformer’s output to make decisions about asset allocation, optimising the portfolio performance.

3.1. Incorporating Macroeconomic Analysis through Composite Indicators

3.1.1. Theoretical Framework for Financial Metrics in Macroeconomics

The interconnection between macroeconomic conditions and financial markets plays a crucial role in shaping investment strategies. Macroeconomic analysis, as a tool, reflects the broader economic states and influences market trajectories, thus informing financial decision-making in an augmented DRL environment [26,27]. This symbiosis between economic indicators and market dynamics forms the theoretical basis for integrating macroeconomic insights into financial metrics.

3.1.2. Data Collection and Preprocessing

The selection of macroeconomic data from 2012 to 2016 was intentional, catering to the study’s aim to analyse market behaviour under various economic backdrops. This period was marked by significant global events affecting market stability, such as the European debt crisis stabilisation efforts and the U.S. Federal Reserve’s tapering of quantitative easing, which offer a rich context for assessing the robustness of the proposed model. Monthly GDP data from BBKI, together with CPI and Unemployment Rate data from the FRED database, were incorporated due to their comprehensive reflection of economic health and their influence on market movements.

3.1.3. Impact of Macroeconomic Indicators on Market Movements

This section outlines the effect of key macroeconomic indicators on market dynamics:

• Real GDP: Real GDP, sourced from the Bureau of Economic Analysis (BEA), represents the total economic output, calculated as $GDP=C+G+I+NX$, where C is consumer spending, G is government expenditures, I is investment, and $NX$ is net exports.
• Consumer Price Index (CPI): Provided by the Bureau of Labor Statistics (BLS), the CPI reflects changes in the price level of a consumer goods and services market basket, calculated through a two-stage process involving fundamental indexes for each item–area combination and aggregate indexes.
• Unemployment Rate: Reported by the BLS, the unemployment rate is based on projections from an interindustry model and the National Employment Matrix, assuming full employment conditions.

These indicators, essential in assessing the economy’s health, significantly impact forecasting market trends. Incorporating macroeconomic indicators into portfolio management provides a nuanced understanding of market movements, essential for informed investment decisions. Portfolio managers can align their strategies with economic trends by analysing indicators such as GDP growth, inflation data, and employment statistics. This process involves using these indicators as predictive tools for market trajectory, aiding in risk mitigation and opportunity identification, especially in the context of major stock indices like the S&P 500.

3.2. Incorporating Sentiment Analysis for Stock Insight from News Data

3.2.1. Framework for Sentiment Analysis

In the realm of stock market analysis, the integration of sentiment analysis has emerged as a pivotal tool for deriving insights from news articles related to individual stocks. This methodology utilises a sentiment analysis framework, which assigns a sentiment score to each news headline. These scores range from −1, denoting a strongly negative sentiment, to 1, indicating a strongly positive sentiment, with 0 representing a neutral stance. This quantification of news sentiment is instrumental in evaluating its potential impact on the microeconomic variables associated with stocks, as highlighted in the works by Devlin et al. [28] and Yang et al. [29].

3.2.2. Data Collection and Preprocessing

The initial phase of this methodology involves procuring a comprehensive dataset of stock-related news articles. The Daily Financial News dataset [30], which is comprised of a substantial collection of news headlines, provides the foundation for the subsequent sentiment analysis. The preprocessing of this dataset includes a thorough cleansing process to eliminate inconsistencies, converting date formats into a standardised form, and organising the data chronologically based on the date and the stock symbol. This meticulous preprocessing ensures that the dataset is primed for accurate sentiment analysis.

3.2.3. Sentiment Analysis and Lexicon Enhancement

The sentiment analysis uses the VADER (Valence Aware Dictionary and sEntiment Reasoner) dictionary [31], a tool recognised for its proficiency in analysing texts from social media and news headlines. Its effectiveness stems from its ability to discern the polarity and the intensity of emotions conveyed in textual data. To refine the sentiment analysis further, the lexicon is enhanced by incorporating the 25 most frequently occurring words in the dataset’s headlines. Each word is assigned a specific sentiment value, calibrated on a scale from −1 to 1. This augmentation of the lexicon is aimed at bolstering the precision of the sentiment analysis.

3.2.4. Sentiment Scoring and Data Structuring

Upon completion of the sentiment analysis, each news headline is ascribed a sentiment score based on the augmented VADER lexicon. The outcome of this analysis is a structured dataset, which includes the original news headlines and their corresponding sentiment scores. This dataset is uniquely formatted to encapsulate the date of the news article, the sentiment score, and the associated stock symbol. Such a structured arrangement of data is specifically designed to facilitate its seamless integration into the non-stationary transformer portfolio management model. Including this sentiment-based information is anticipated to enhance the model’s ability to make more informed and accurate predictions regarding stock market trends and behaviours.

3.3. Non-Stationary Transformer with Deep Deterministic Policy Gradient

The advancement in portfolio management techniques has led us to develop the “Adaptive Transformer” (non-stationary transformer) within the DDPG framework. This innovative element is tailored to recognise and adapt to the dynamic statistical characteristics of financial time-series data, thereby enhancing the model’s predictive accuracy. The architecture of our “Adaptive Transformer” comprises four key components, each intricately designed to work in harmony, ensuring the model’s adeptness in navigating the ever-changing landscape of financial data. We delve into the specifics of these components.

3.3.1. Projector Layer

At the core of the “Adaptive Transformer” resides the projector layer, an intricate multilayer perceptron (MLP). It recognises and adapts to non-stationary elements within sequential data streams. The procedure initiates with a global average pooling operation to reduce temporal complexity, thereby isolating pivotal attributes:

$$x_{\mathrm{pooled}}=\mathrm{GlobalAvgPooling}\left(x\right)$$

(1)

After this data reduction, dense layers (${\mathrm{Dense}}_{\mathrm{hidden}}$) ensue. These layers are enhanced with ReLU activation functions and L2 regularisation, a strategy that aids in distilling and integrating intricate data features. The process also benefits from the application of a hyperbolic tangent function, culminating in an output that encapsulates the identified non-stationary elements:

$$y=tanh\left({\mathrm{Dense}}_{\mathrm{hidden}}\left(x_{\mathrm{pooled}}\right)\right)$$

(2)

3.3.2. Transformer Encoder Layer

The transformer encoder layer, an integral part of the “Adaptive Transformer”, has a self-attention mechanism that is particularly proficient in meticulously examining and interpreting financial data sequences. Embedded within this layer are two pivotal adaptive elements, termed tau_learner (${\mathrm{Projector}}_{\tau }$) and delta_learner (${\mathrm{Projector}}_{\mathsf{\Delta }}$). These components are instrumental in learning temporal scaling and shifting factors, endowing the model with a refined acuity for adjusting its self-attention outputs to the present non-stationary conditions:

$$\tau =exp\left({\mathrm{Projector}}_{\tau }(x_{\mathrm{raw}},{\sigma }_{\mathrm{enc}})\right),\mathsf{\Delta }={\mathrm{Projector}}_{\mathsf{\Delta }}(x_{\mathrm{raw}},{\mu }_{\mathrm{enc}}),$$

(3)

where ${\sigma }_{\mathrm{enc}}$ and ${\mu }_{\mathrm{enc}}$ denote the standard deviation and mean of the input sequences, respectively. The recalibrated multi-head attention (MHA) output is thus articulated as follows:

$${\mathrm{Attn}}_{\mathrm{output}}=\mathrm{MHA}\left(x\right)·\tau +\mathsf{\Delta }$$

(4)

To promote robustness and prevent overfitting, layer normalisation and a dropout strategy are applied:

$$x=\mathrm{LayerNorm}(x+\mathrm{Dropout}\left({\mathrm{Attn}}_{\mathrm{output}}\right))$$

(5)

3.3.3. Policy Network with Transformer

The policy network with transformer is designed to formulate a strategic policy for decision-making processes. It integrates state representations with the robust architecture of the transformer, succeeded by batch normalisation and a series of dense layers, thereby creating a refined state-to-action mapping function, represented as:

$$\mathrm{Net}={\mathrm{Dense}}_{\mathrm{ReLU}}\left(\mathrm{BatchNorm}\left(\mathrm{Transformer}\left(x_{\mathrm{state}}\right)\right)\right)$$

(6)

Following this, a Softmax function, tempered by a specified parameter, delineates the action space. This process is essential in balancing the imperative dichotomy between exploratory and exploitative behaviours:

$$\mathrm{Action}={\mathrm{Softmax}}_T\left(\mathrm{Dense}\left(\mathrm{Net}\right)\right)\times \mathrm{action}\_\mathrm{bound}$$

(7)

(Equation (7)) signifies the culmination of the network’s computation, where the Softmax function, adjusted by a temperature factor, contributes to a probabilistic selection of actions constrained within the defined bounds of possible actions.

3.3.4. Q-Value Network with Transformer

Simultaneously, incorporating a transformer architecture, the Q-value network calculates the expected returns for various state-action pairs. A concurrent transformer encoder framework facilitates these calculations. Initially, the inputs representing the states and actions are processed independently, followed by normalisation. They are then concatenated to combine the normalised state and action inputs into a single tensor. This combined tensor then passes through a series of dense layers, ultimately leading to the extraction of the Q-value. The Q-value represents a unified assessment of the anticipated reward for a given state–action pair:

$$Q(s,a)=\mathrm{Dense}\left(\mathrm{Concat}\right(\mathrm{BatchNorm}\left(s\right),\mathrm{BatchNorm}\left(a\right)\left)\right)$$

(8)

The resultant Q-value is then articulated as:

$$Q_{\mathrm{value}}=\mathrm{Dense}\left({\mathrm{Net}}_{\mathrm{combined}}\right)$$

(9)

This expression encapsulates the network’s computation, where the Q-value embodies the expected utility of adopting a specific action in a given state, as per the model’s learned policy.

3.3.5. Optimisation and Policy Learning within DDPG Framework

The DDPG algorithm lies at the heart of our optimisation technique. It is a model-free, off-policy actor-critic algorithm using deep function approximators that can learn policies in high-dimensional, continuous action spaces. Our implementation encapsulates an actor network designed to map states to actions, and a critic network that evaluates the action given the state.

The initialisation of these networks is performed with random weights, which are subsequently refined through training iterations. The actor network proposes an action given the current state, while the critic network appraises the proposed action by estimating the Q-value. The policy, parameterised by $\theta$, generates actions predicated on the current state, aiming to maximise the cumulative reward:

$$J\left({\mu }_{\theta }\right)={\int }_S{\rho }_{\mu }\left(s\right)r(s,{\mu }_{\theta }\left(s\right))ds={\mathbb{E}}_{s\sim {\rho }_{\mu }}\left[r(s,{\mu }_{\theta }\left(s\right))\right]$$

(10)

where ${\mu }_{\theta }$ is the deterministic policy, $Q^{\mu }$ is the action-value function according to policy $\mu$, and ${\rho }^{\mu }$ is the state distribution under policy $\mu$. Considering the dynamic realm of portfolio management, we recalibrate this function to account for the sequential product of portfolio values:

$$J_T\left({\mu }_{\theta }\right)=J_0\prod _{t=1}^T(1+r(s_t,{\mu }_{\theta }\left(s_t\right)))$$

(11)

where $J_0$ represents the initial portfolio valuation.

To ensure stability and promote convergence, we employ target networks for the actor and critic, which are slowly updated to track the learned networks. The update for the target networks is governed by a soft update parameter, $\tau$, ensuring that the targets change gradually, which helps with the stability of learning:

$${\theta }_{\mathrm{target}}\leftarrow \tau \theta +(1-\tau ){\theta }_{\mathrm{target}}$$

(12)

The policy is refined through gradient ascent on the expected cumulative reward, while the critic’s parameters are adjusted based on the temporal difference error. To maintain exploration, a noise process $\mathcal{N}$ is added to the actor’s policy action:

$$a_t={\mu }_{\theta }\left(s_t\right)+{\mathcal{N}}_t$$

(13)

where $a_t$ is the chosen action, ${\mu }_{\theta }$ is the current policy, $s_t$ is the current state, and ${\mathcal{N}}_t$ is the noise term at time t, which decays during training to allow the policy to exploit the learned values as it matures.

This methodical learning process culminates in developing a policy that maximises expected returns, aligning with our objective of enhancing portfolio management strategies.

3.3.6. Action and Reward Formulation

At any discrete time step t, the action $a_t$ is identified with the portfolio’s asset allocation vector $w_t$. The policy endeavours to optimise the allocation for the forthcoming period $w_{t+1}$, maintaining the constraint of unitary sum across all asset weights. The updated allocation, which adapts to price fluctuations $y_t$, is formalised as:

$$w_{t+1}=\frac{y_t\odot w_t}{y_t·w_t}$$

(14)

Here, ⊙ denotes the Hadamard product and · represents the conventional dot product.

The reward function, a pivotal determinant of the portfolio’s efficacy, encompasses transaction costs denoted by $\beta$, and is thus defined to capture the realised profit at time t:

$$r_t=w_t·y_t-\beta \sum _{i=1}^n|w_{i,t+1}-w_{i,t}|$$

(15)

Enhancing the practical applicability and differentiability of the reward function, we integrate fixed commission rates $c_b$ for buying and $c_s$ for selling. The reward function is consequently defined as:

$$r_t=ln({\beta }_t·(w_t·y_t))$$

(16)

The ultimate objective is to amplify the cumulative return R across a designated interval T, expressed as:

$$R(s_1,w_1,\dots ,s_T,w_T)=\frac{1}{T}\sum _{t=1}^{T}ln({\beta }_t·(w_t·y_t))$$

(17)

The optimisation conundrum thus revolves around ascertaining the optimal policy ${\mu }_{\theta }$ that maximises this return:

$${\mu }_{\theta }^{*}=arg\underset{{\mu }_{\theta }}{max}\frac{1}{T}\sum _{t=1}^{T}ln({\beta }_t·(w_t·y_t))$$

(18)

The policy gradient with respect to the parameters $\theta$ is meticulously derived:

$${\nabla }_{\theta }{\mu }_{\theta }\left(\tau \right)=\frac{1}{T}\sum _{t=1}^T{\nabla }_{\theta }ln\left({\mu }_{\theta }(w_t,s_t)\right)$$

(19)

and the parameters are then conscientiously updated with a learning rate $\lambda$:

$$\theta \leftarrow \theta +\lambda {\nabla }_{\theta }{J}_T\left({\mu }_{\theta }\right)$$

(20)

The normalisation by T is crucial, as it appropriately scales the gradient irrespective of batch sizes, an essential factor for effective mini-batch training in a stochastic environment.

Overall, this structured approach in the methodology section ensures a comprehensive understanding of the sophisticated non-stationary transformer model and its integration with the DDPG framework, which is crucial for advanced portfolio management techniques.

4. Experiment Setup

4.1. Dataset

Our analysis focuses on S&P 500 data from 2012 to 2016 (Table 1), chosen for their market stability and diverse economic conditions. For efficiency, we selected the top 100 stocks by trading volume from the 2012 S&P 500 index. This selection means our stock portfolio is based on these 100 stocks, allowing for a targeted and effective training approach for our deep reinforcement learning models. To ensure robust model training and validation, we divide our dataset into distinct subsets:

• Training Set (2012–2014): The initial phase of our analysis utilises data from 2012 to 2014 as the training set. This period is crucial for training our models, tuning hyperparameters, and optimising algorithms based on historical market trends.
• Validation Set (2015): Data from 2015 are used as the validation set. The primary purpose of this phase is to compare different time window hyperparameters and select the optimal ones for application in our models. This step is critical for assessing the performance of our models under different market conditions and ensuring that they can generalise well to new data.
• Testing Set (2016): Finally, the 2016 data serve as our testing set. In this phase, we apply the models refined through training and validation to this unseen dataset. This allows us to compare the performance of various models and configurations, providing insights into their effectiveness and robustness.

The rationale behind this structured approach to dataset splitting is to ensure that our models are well-trained on historical data and thoroughly validated before being tested. By comparing the performance of models trained and validated on different subsets, we can more accurately assess their predictive capabilities and adaptability to market changes. This systematic approach significantly enhances the reliability and validity of our findings.

Table 1. Statistics of training, validation and testing datasets.

Period	Date Range	#Examples
Training	2 January 2012 to 31 December 2014	754
Validation	4 January 2015 to 31 December 2015	252
Testing	2 January 2016 to 30 December 2016	252

Table 1. Statistics of training, validation and testing datasets.

Period	Date Range	#Examples
Training	2 January 2012 to 31 December 2014	754
Validation	4 January 2015 to 31 December 2015	252
Testing	2 January 2016 to 30 December 2016	252

4.2. Compared Models

Our empirical evaluation contextualises the performance of the NSTD model by benchmarking it against a selection of advanced deep-learning models and traditional financial strategies. The models in this comparative analysis include the following:

• Deep Learning Approaches:
  • Transformer: [19] Employs self-attention mechanisms for long-range dependencies in time-series data.
  • LSTM (Long Short-Term Memory): [32] Excels in learning from sequences with long temporal dependencies.
  • GRU (Gated Recurrent Unit): [33] Designed to capture dependencies of varying time scales in sequential data.
  • RNN (Recurrent Neural Network): [34] Processes temporal sequences through a directed graph architecture.
  • EIIE (Ensemble of Identical Independent Evaluators): [6] Processes time series for each asset independently while maintaining shared network parameters.
  • EI${}^3$ (Ensemble of Identical Independent Inception): [35] Extracts and aggregates multiscale price movement information, integrated with a recurrent reinforcement learning framework.
• Traditional Financial Strategies:
  • OLMAR (Online Moving Average Reversion): [36] Utilises mean reversion effect based on moving average.
  • UBAH (Uniform Buy and Hold): [37] Allocates funds uniformly across assets, maintaining them without further trading actions.

4.3. Evaluating Investment Strategies with Financial Metrics

We employ several financial metrics to evaluate the efficiency and risk of investment portfolios, as discussed below:

4.3.1. Accumulative Return (AR)

Accumulative return (AR) [38] is a crucial metric that quantifies the total return generated by an investment relative to its initial capital. It is calculated as the ratio of the final portfolio value $p_t$ to the initial portfolio value $p_0$, directly measuring the overall return on investment. Mathematically, it is expressed as:

$$\mathrm{AR}=\frac{p_t}{p_0}$$

(21)

where $p_t$ represents the portfolio’s value at the end of the investment period, and $p_0$ is the portfolio’s initial value. This metric is particularly useful for assessing an investment’s growth or decline over a specified time frame. A higher AR indicates a greater return on the initial capital invested, signifying effective portfolio management and investment strategy. Conversely, an AR of less than one means a decline in the value of the initial investment, indicating potential areas for strategic adjustment in portfolio management. AR is thus an integral component of a comprehensive investment performance analysis, providing a straightforward yet powerful tool for investors to gauge the efficacy of their investment decisions.

4.3.2. Sharpe Ratio

The Sharpe ratio [39] measures risk-adjusted returns:

$$\mathrm{Sharpe}\mathrm{Ratio}=\frac{R_p-R_f}{{\sigma }_p}$$

(22)

where $R_p$ is the return of the portfolio, $R_f$ is the risk-free rate, and ${\sigma }_p$ is the standard deviation of the portfolio’s excess return. A Sharpe ratio greater than one is generally considered good, indicating adequate returns relative to the risk taken.

4.3.3. Maximum Drawdown (MDD)

MDD [40] assesses the largest drop in portfolio value:

$$\mathrm{MDD}=\frac{\mathrm{Trough}\mathrm{Value}-\mathrm{Peak}\mathrm{Value}}{\mathrm{Peak}\mathrm{Value}}$$

(23)

A lower MDD is preferable, indicating less potential loss.

4.3.4. Sortino Ratio

The Sortino ratio [41] differentiates harmful volatility from overall volatility:

$$\mathrm{Sortino}\mathrm{Ratio}=\frac{R_p-R_f}{{\sigma }_d}$$

(24)

where ${\sigma }_d$ is the standard deviation of negative asset returns. A higher Sortino ratio indicates better performance concerning downside risk.

These metrics offer valuable insights for strategic portfolio positioning and serve as benchmarks for evaluating post-investment performance. When integrated with DRL strategies, they optimise investment strategies, adapting to changing economic landscapes.

5. Experiment and Analysis

Our experiment used the S&P 500 data from 2012 to 2014 as our training dataset. This selection was instrumental in training our models, providing a comprehensive representation of market activities over this period. To circumvent memory overflow issues commonly encountered in prolonged training sessions, we adopted a strategy of fine-tuning model parameters after every set of 200 episodes. The training process was executed in a phased manner, with each model undergoing training ten times, each consisting of 200 episodes, amounting to a total of 2000 training rounds. The learning rate was subsequently reduced to 90% of its value from the previous phase. This progressive reduction in the learning rate facilitated nuanced model tuning and optimisation. The optimisation was conducted using the Adam optimiser.

In addition to the primary training process, we conducted comprehensive comparative experiments to determine the most effective hyperparameters for our model. These experiments focused on two key hyperparameters: the temperature coefficient and window length. The window length, a crucial hyperparameter in time-series analysis, was also rigorously examined. To ascertain the most effective window length for capturing market dynamics, we considered a range of durations, specifically 3, 5, 7, 10, 15, 20, 30, and 50 days. For the temperature coefficient, to control the portfolio diversity, we explored various settings, including 500, 1000, 3000, 5000, and 10000, to observe the impact on model performance. Each model, configured with a distinct set of these hyperparameters, was initially trained on our training dataset (S&P 500 data from 2012 to 2014). Following the training phase, we saved the model parameters and then validated their performance on a separate validation dataset from 2015. This approach allowed us to thoroughly assess the influence of each hyperparameter on the model’s efficacy, enabling us to fine-tune our model for optimal performance.

Upon identifying the most effective hyperparameters through our rigorous evaluation process, we applied these optimised settings to a backtest dataset from 2016. This phase of our research involved conducting comparative ablation studies using four distinct dataset compositions, each offering unique insights into the model’s performance. The primary dataset encompassed a comprehensive blend of macroeconomic indicators, sentiment scores from news articles, and historical stock trading data, providing a holistic view of market influences. The second dataset was curated by excluding macroeconomic indicators while maintaining sentiment scores and historical trading data, allowing us to assess the impact of macroeconomic factors on model performance. In contrast, the third dataset excluded news sentiment scores but included macroeconomic indicators alongside historical data, enabling us to isolate and evaluate the influence of sentiment analysis. The fourth and final dataset was streamlined to include only historical stock trading data, offering a baseline to gauge the added value of macroeconomic and sentiment data.

Moreover, to contextualise our model’s performance within the broader landscape of financial analytics, we conducted comparative analyses against other leading deep learning models and established traditional financial models. This comprehensive approach allowed us to validate our model’s efficacy and understand its standing relative to the current state-of-the-art methodologies in financial portfolio management.

5.1. Evaluation on Validated Dataset

Upon training our models with the dataset from 2012 to 2014, we evaluated the efficacy of various hyperparameters—specifically, the window length and temperature coefficients—using the 2015 dataset as our validation set. This step was crucial to calibrate the models effectively before final testing.

In Figure 2, the analysis of cumulative returns for the validation dataset across varying window lengths reveals the strategic influence of the chosen time frame. An intriguing periodic decline in $P_t/P_0$ at the end of each month is observed, which may reflect routine market occurrences such as settlement cycles. The 10-day window (NSTD-10), marked in red, consistently surpasses other strategies from mid-year to year-end, indicating that a 10-day span effectively leverages historical data to inform current market predictions. Shorter windows like the 3-day (NSTD-3) and 5-day (NSTD-5) show early promise but later falter, suggesting they may be too narrow to grasp the larger cyclical movements of the market that a 10-day window encapsulates. Conversely, the longer window strategies—NSTD-30 and NSTD-50—demonstrate smoother return trajectories but do not achieve the higher returns seen with NSTD-10. This implies that while longer windows provide a broader historical context, potentially capturing more extended market cycles, they may lack the agility to capitalise on immediate market shifts. The 10-day strategy emerges as the most effective, balancing immediate market reactivity with understanding broader market movements, highlighting its efficiency for this period.

In Figure 3, the cumulative returns for the validated dataset are mapped out across different temperature coefficients (T) on portfolio performance. The parameter T influences the dispersion of stock weights within the portfolio, with higher T values promoting diversification and lower values indicating concentration. Portfolios with T = 5000 and T = 10,000, respectively, demonstrate a marked increase in cumulative return, especially noticeable from mid-year onward, with T = 10,000 displaying a pronounced uptick as the year concludes. This trend suggests that higher temperature settings, while promoting a more balanced weight distribution across the portfolio, may also enable the capture of growth opportunities in a rising market, contributing to higher overall returns. Conversely, the T = 500 and T = 1000 settings indicate a preference for a more concentrated portfolio. While this strategy may yield higher short-term gains by focusing on a limited number of stocks, it also introduces heightened risk, as the portfolio is more vulnerable to the volatility of its few constituents. These lower T values produce more modest growth throughout the year, with occasional dips reflecting the increased risk exposure. The T = 3000 setting balances the two extremes. Portfolios with this temperature setting deliver steady, consistent growth over the year, suggesting that a moderate T value can effectively balance the benefits of diversification with the potential gains from a more focused investment strategy. Overall, the temperature parameter’s calibration is crucial for managing risk and capturing growth within a portfolio.

5.2. Ablation Study for Features

After obtaining the optimal hyperparameters, we tested our model based on different features in 2016 (test set). The comprehensive NSTD-10 includes macroeconomic indicators, sentiment-derived news scores, and historical stock trading data. The NSTD without Macro excludes macroeconomic indicators while retaining news scores and historical data. The NSTD without News removes news scores featuring macroeconomic indicators and historical data. The NSTD Pure is the most pared-down, solely comprising historical stock trading data.

Figure 4 shows the year-long trajectory of cumulative returns for each NSTD variant. The NSTD-10 model sets a high standard with an impressive performance, indicating that a 10-day time window robustly captures market trends when combined with a full feature set. Meanwhile, the NSTD without the macro variant experiences a moderate decline in return. This suggests that while macroeconomic indicators contribute to performance, their absence does not drastically diminish the model’s efficacy. Conversely, the model without news sentiment maintains a competitive AR close to the full model’s, suggesting that excluding sentiment indicators might not significantly impact the model’s performance in the presence of other data types. Yet, as shown in

Table 2. Different metrics’ performance of models.

Model	AR	MDD	Sharpe Ratio	Sortino Ratio
NSTD 10	3.353	0.277	1.859	282.140
NSTD without News	3.000	0.182	1.273	78.099
NSTD without Macro	2.311	0.158	1.668	0.000
NSTD Pure	1.801	0.237	0.443	3.101
Transformer	1.460	0.231	1.119	7.823
RNN	1.953	0.282	1.303	29.622
LSTM	1.921	0.248	1.824	0.000
GRU	1.622	0.251	1.772	158.717
EI${}^3$	2.196	0.291	1.446	27.188
EIIE	2.174	0.298	1.773	0.000
OLMAR	2.349	0.221	1.517	98.337
UBAH	2.349	0.221	1.517	98.337

, the altered ratios of Sharpe and Sortino indicate that news scores play a role in refining risk-adjusted returns. Notably, the NSTD Pure model demonstrates a significant decrease in AR and the Sharpe ratio, emphasising the collective value of integrating both macroeconomic and sentiment indicators into the model. This variant’s underperformance highlights the complexity of market dynamics and the importance of a diversified information set for effective market analysis. It is evident that the comprehensive NSTD-10 model, with its full suite of indicators, excels, underscoring the importance of a multifaceted approach to capturing the full spectrum of market signals. However, the model does not predict the recurrent end-of-month drops in $P_t/P_0$, which indicates a potential area for model refinement to accommodate such cyclical patterns.

5.3. Comparison with Other Deep Learning and Traditional Financial Methods

In demonstrating the superior performance of the NSTD-10 model, we conducted extensive backtesting analysis over 2016 and compared it with deep learning architectures, such as the transformer, RNN, LSTM, and GRU, as well as state-of-the-art EIIE and EI${}^3$, together with traditional financial algorithms like online moving average reversion (OLMAR) and the universal portfolio algorithm (UBAH). The evaluation hinged on critical financial metrics: annual return (AR), maximum drawdown (MDD), Sharpe ratio, and Sortino ratio. The analysis depicted in Figure 5, when considered alongside the quantitative metrics from Table 2, offers a nuanced assessment of various model performances. The NSTD-10 model emerges as the frontrunner, boasting an annual return (AR) of 3.353, indicative of its effective market trend utilisation for optimal capital increase. Its Sharpe ratio of 1.859 and a notably high Sortino ratio of 282.140 reflect its adeptness at securing returns per unit of risk, with a pronounced efficiency in mitigating downside risk. Delving into the NSTD model variations, the omission of news sentiment analysis (NSTD without News) results in a slight decrease in AR to 3.000 and a marked reduction in the Sortino ratio to 78.099, highlighting the significant contribution of sentiment data to forecasting precision and risk-adjusted returns. Excluding macroeconomic indicators (NSTD without Macro), we see a reduced AR of 2.311. However, it attains the lowest maximum drawdown (MDD) at 0.158, signifying macroeconomic insights’ protective role against market downturns. The NSTD Pure variant, which solely incorporates historical trading data, reiterates the value of a diversified data approach, as evidenced by its lower metrics across the spectrum. Compared with established deep learning models such as the transformer, RNN, LSTM, and GRU, these models exhibit commendable results but do not match the NSTD-10’s superior standards. Specifically, the LSTM model’s high Sharpe ratio of 1.824 underscores its strength in handling time-series data. Nevertheless, this does not correlate with a high AR or Sortino ratio, pointing to potential limitations in market dynamics adaptation. The ensemble approaches EI${}^3$, and EIIE offers more consistent returns. However, their Sharpe and Sortino ratios fall short of those achieved by NSTD-10, suggesting that while reliable, they may not optimise the balance between risk and return. Traditional financial methods such as OLMAR and UBAH maintain stable risk management but lack NSTD-10’s efficacy in capitalising on market conditions. Overall, the NSTD-10 model’s unmatched performance in terms of overall returns and risk management sets a new standard in portfolio management. It underscores the strategic benefit of integrating comprehensive market data, from economic indicators to sentiment analysis, within a sophisticated algorithmic framework.

6. Conclusions

This study introduces the non-stationary transformer with the deep deterministic policy gradient (NSTD) model, a novel approach in portfolio management that leverages macroeconomic insights and sentiment analysis. Our empirical evaluation suggests that the NSTD model, particularly with a 10-day window, outperforms various financial strategies by effectively capturing market trends and balancing short-term reactivity with long-term insights. While integrating many data sources has significantly augmented the decision-making process, we acknowledge the model’s current limitation in predicting specific cyclical market movements. In future work, we aim to enhance the NSTD model’s efficacy by incorporating a broader spectrum of external economic indicators and multimodal data, which will improve its predictive capabilities for cyclical patterns. This development is expected to refine our understanding of complex market dynamics and support the creation of more informed and adaptable investment strategies. The NSTD model represents a significant step forward in applying machine learning to portfolio management. With these enhancements, it has the potential to set a new benchmark for technological innovation in financial analysis.