Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Time-aware deep reinforcement learning with multi-temporal abstraction

Time-aware deep reinforcement learning with multi-temporal abstraction Deep reinforcement learning (DRL) is advantageous, but it rarely performs well when tested on real-world decision-making tasks, particularly those involving irregular time series with sparse actions. Although irregular time series with sparse actions can be handled using temporal abstractions for the agent to grasp high-level states, they aggravate temporal irregularities by increasing the range of time intervals essential to represent a state and estimate expected returns. In this work, we propose a general Time-aware DRL framework with Multi-Temporal Abstraction (T-MTA) that incorporates the awareness of time intervals from two aspects: temporal discounting and temporal abstraction. For the former, we propose a Time-aware DRL method, whereas for the latter we propose a Multi-Temporal Abstraction mechanism. T-MTA was tested in three standard RL testbeds and two real-life tasks (control of nuclear reactors and prevention of septic shock), which represent four common contexts of learning environments, online and offline, as well as fully and partially observable. As T-MTA is a general framework, it can be combined with any model-free DRL method. In this work, we examined two in particular: the Deep Q-Network approach and its variants, and Truly Proximal Policy Optimization. Our results show that T-MTA significantly outperforms competing baseline frameworks, including a standalone Time-aware DRL framework, MTAs, and the original DRL methods without considering either type of temporal aspect, especially when partially observable environments are involved and the range of time intervals is large. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Applied Intelligence Springer Journals

Time-aware deep reinforcement learning with multi-temporal abstraction

Kim, Yeo Jin; Chi, Min
Applied Intelligence , Volume 53 (17) – Sep 1, 2023
Download PDF
27 pages

Loading next page...
 
/lp/springer-journals/time-aware-deep-reinforcement-learning-with-multi-temporal-abstraction-B6HfEGZq0m
Publisher
Springer Journals
Copyright
Copyright © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
ISSN
0924-669X
eISSN
1573-7497
DOI
10.1007/s10489-022-04392-5
Publisher site
See Article on Publisher Site

Abstract

Deep reinforcement learning (DRL) is advantageous, but it rarely performs well when tested on real-world decision-making tasks, particularly those involving irregular time series with sparse actions. Although irregular time series with sparse actions can be handled using temporal abstractions for the agent to grasp high-level states, they aggravate temporal irregularities by increasing the range of time intervals essential to represent a state and estimate expected returns. In this work, we propose a general Time-aware DRL framework with Multi-Temporal Abstraction (T-MTA) that incorporates the awareness of time intervals from two aspects: temporal discounting and temporal abstraction. For the former, we propose a Time-aware DRL method, whereas for the latter we propose a Multi-Temporal Abstraction mechanism. T-MTA was tested in three standard RL testbeds and two real-life tasks (control of nuclear reactors and prevention of septic shock), which represent four common contexts of learning environments, online and offline, as well as fully and partially observable. As T-MTA is a general framework, it can be combined with any model-free DRL method. In this work, we examined two in particular: the Deep Q-Network approach and its variants, and Truly Proximal Policy Optimization. Our results show that T-MTA significantly outperforms competing baseline frameworks, including a standalone Time-aware DRL framework, MTAs, and the original DRL methods without considering either type of temporal aspect, especially when partially observable environments are involved and the range of time intervals is large.

Journal

Applied Intelligence – Springer Journals

Published: Sep 1, 2023

Keywords: Time-aware; Temporal abstraction; Deep reinforcement learning; Irregular time series; RL for real-world applications; Nuclear reactor control; Sepsis treatment

References

  • Hyperbolically discounted temporal difference learning
    Alexander, W; Brown, J
  • Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path
    Antos, A; Szepesv̀ari, C; Munos, R
  • Recent advances in hierarchical reinforcement learning
    Barto, A; Mahadevan, S
  • The arcade learning environment: an evaluation platform for general agents
    Bellemare, MG; Naddaf, Y; Veness, J; Bowling, M
  • Predictors of patients who present to the emergency department with sepsis and progress to septic shock between 4 and 48 hours of emergency department arrival
    Capp, R; Horton, C; Takhar, S; Ginde, A; Peak, D; Zane, R; Marill, K
  • Recurrent neural networks for multivariate time series with missing values
    Che, Z; Purushotham, S; Cho, K; Cho, K; Sontag, D; Liu, Y
  • Reinforcement learning in continuous time and space
    Doya, K
  • Doubly robust policy evaluation and optimization
    Dudík, M; Erhan, D; Langford, J; Li, L
  • Efficient estimation of average treatment effects using the estimated propensity score
    Hirano, K; Imbens, GW; Ridder, G
  • Condition-based probabilistic safety assessment for maintenance decision making regarding a nuclear power plant steam generator undergoing multiple degradation mechanisms
    Hoseyni, SM; Maio, FD; Zio, E
  • Double reinforcement learning for efficient off-policy evaluation in Markov decision processes
    Kallus, N; Uehara, M
  • Algorithm for autonomous power-increase operation using deep reinforcement learning and a rule-based system
    Lee, D; Arigi, AM; Kim, J
  • Prediction of peak values in time series data for prognostics of critical components in nuclear power plants
    Liu, J; Zio, E
  • The timing of early antibiotics and hospital mortality in sepsis
    Liu, V; Fielding-Singh, V; Greene, J
  • Bias and variance approximation in value function estimates
    Mannor, S; Simester, D; Sun, P; Tsitsiklis, JN
  • Management of severe sepsis: advances, challenges, and current status
    Levy, MM; Artigas, A
  • Human-level control through deep reinforcement learning
    Minh, V; Kavukcuoglu, L; Silver, D; Rusu, AA; Veness, J; Bellemare, MG
  • Policy gradient in continuous time
    Munos, R
  • Discouting of delayed rewards: models of individual choice
    Myerson, J; Green, L
  • Learning dexterous in-hand manipulation
    Open, AI
  • Epidemiology and costs of sepsis in the united states-an analysis based on timing of diagnosis and severity level
    Paoli, C; Reynolds, M; Sinha, M
  • Providing support to operators for monitoring safety functions using reinforcement learning
    Park, J; Kim, T; Seong, S
  • Reinforcement learning of motor skills with policy gradients
    Peters, J; Schaal, S
  • Nucler energy: between global electricity demand, worldwide decarbonisation imperativeness and palnetary environmental implications
    Pràvàlie, R; Bandoc, G
  • A note on measurement of utility
    Samuelson, PA
  • Individualized sepsis treatment using reinforcement learning
    Saria, S
  • Mastering the game of go with deep neural networks and tree search
    Silver, D; Huang, A
  • The third international consensus definitions for sepsis and septic shock (sepsis-3)
    Singer, M; Deutschman, C; Seymour, C
  • Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning
    Sutton, R; Precup, D; Singh, S
  • Grandmaster level in StarCraft II using multi-agent reinforcement learning
    Vinyals, O
  • Reinforcement learning in continuous time and space: a stochastic control approach
    Wang, H; Zariphopoulou, T; Zhou, X
  • An accident diagnosis algorithm using long short-term memory
    Yang, J; Kim, J
  • Discounting of reward sequences: a test of competing formal models of hyperbolic discounting
    Zarr, N; Alexander, WH; Brown, JW