Reinforcement Learning (RL) has emerged as a promising solution in many network and service management research applications. Despite its growing popularity, the adoption of RL in Network and Service Management (NSM) is still in its early stages. While we have seen encouraging applications of RL, several fundamental questions remain unanswered.

What follows is the RL Manifesto for Network and Service Management, born out of a series of international conferences and exchanges aimed at assessing the potential of RL in the NSM field.

Click here to subscribe to the Google Group!

1. Beyond “Point” Decision Making: Unlocking the Sequential Potential of RL

Reinforcement Learning (RL) fundamentally addresses sequential decision-making problems. However, its current applications in network and service management often focus on “point” decision-making—using RL as a tool for feedback-based control loops to restore a desired state. This approach limits RL’s true potential. Are we underutilizing RL’s ability to model trajectories within state spaces? A shift in perspective could redefine how RL problems are formulated for network and service management, enabling long-term optimization strategies. Promising approaches include trajectory-centric RL formulations and leveraging temporal abstraction techniques.

2. Expanding the Interaction Models Beyond MDPs

The Markov Decision Process (MDP) remains the dominant model for RL applications, but network and service management problems often involve more complex interaction dynamics. Models such as Partially Observable MDPs (POMDPs), Decentralized POMDPs (Dec-POMDPs), or even hierarchical interaction models offer untapped opportunities. The tradeoff between model accuracy and complexity must be carefully managed, especially given challenges like the curse of dimensionality. Research should focus on scalable methodologies, such as abstraction techniques, to enable richer interaction models without overwhelming computational resources.

3. Rethinking Reward Models: Scalar vs. Multidimensional Approaches

Reward models are a cornerstone of RL but are often oversimplified. Scalar rewards might fail to capture the multifaceted objectives in network and service management. Alternatives, such as vectorized or hierarchical reward models, and reward-free RL paradigms, are worth exploring. How can these models be tailored to the specific needs of network management tasks, and what methodologies could validate their effectiveness?

4. The Potential of Model-Based RL and Hybrid Approaches

Model-based RL approaches, such as combining Model Predictive Control (MPC) with RL, present significant opportunities. These methods can improve sample efficiency, enhance safety, and stabilize learning. Recent debates, including Yann LeCun’s suggestion to prioritize MPC over RL, highlight the need to critically evaluate the utility of these hybrid approaches. Identifying methodologies that seamlessly integrate model-based reasoning with RL’s adaptability is crucial for real-world deployment.

5. RL and Digital Twins: Synergistic Development

The interplay between RL and digital twins represents an exciting frontier. Should digital twins be designed specifically to facilitate RL, or vice versa? Collaborative development could lead to breakthroughs in both fields, with digital twins providing rich simulated environments for RL training and RL enhancing the operational insights generated by digital twins.

6. Exploring Offline RL for Practical Applicability

Offline RL, trained on pre-collected datasets, offers a promising path to improve sample efficiency and address the challenges of real-world RL deployment. However, the performance gap between offline and online RL remains an open question. Future research should quantify this gap and develop methodologies to minimize it, particularly in scenarios where real-world interactions are expensive or disruptive.

7. Improving Interpretability and Robustness

RL solutions often face criticism for their lack of interpretability. Approaches such as explainable AI (XAI), symbolic reasoning, and logic-based integration (e.g., DeepProbLog) could make RL more accessible for network and service management. Furthermore, RL must become more robust to context changes and adversarial attacks. Developing holistic metrics to evaluate robustness across both RL-specific KPIs and system-level KPIs is essential for building trustworthy RL systems.

8. Transfer Learning and Generalizability

Transfer learning can play a vital role in reducing training overhead and improving adaptability in RL applications. However, techniques to generalize policies across different environments, network topologies, and workload scenarios remain underexplored. Research should focus on designing algorithms capable of robust transfer learning and metrics to evaluate generalizability effectively.

9. Leveraging Generative AI for RL Advancement

Generative AI could expand RL’s training capabilities through synthetic trajectory generation and multi-modal policy design. Exploring these synergies may lead to breakthroughs in sample efficiency and policy quality. Generative models can also assist in approximating complex environment dynamics, offering new tools for RL algorithm development.

10. RL in the Data Plane: New Frontiers

Emerging applications of RL in the data plane, as seen in recent work, offer intriguing opportunities. RL algorithms must address unique challenges in this domain, such as meeting stringent latency requirements. Future research should explore trade-offs in algorithm complexity and inference time to satisfy quality-of-service (QoS) constraints in real-time operations.

11. Identifying Suitable Use Cases and Formulating Appropriate KPIs

RL is not universally optimal. Use cases where RL’s sequential optimization capabilities are indispensable need clear identification. For instance, scenarios with irreversibility or significant delayed rewards are well-suited to RL. Conversely, problems solvable through simpler heuristics should avoid unnecessary RL complexity. Developing frameworks to match RL algorithms with problem types and complexity levels is a priority.

12. Advancing Sampling Efficiency through Model-Based RL

Model-based RL can leverage existing knowledge about network dynamics to accelerate convergence and improve decision-making quality. By integrating predictive models, RL can reduce reliance on direct interaction, minimizing disruptions during training and enhancing policy stability.

13. Addressing Challenges in Generalizability and Partial Observability

The ability of RL to cope with partially observable environments and generalize across diverse scenarios remains a challenge. Research must prioritize the development of algorithms that address these gaps, incorporating methods such as belief state approximation and state space abstraction.

14. Trustworthy and Secure RL

The network and service management community must address RL’s susceptibility to adversarial attacks. Investigating robust RL methodologies and fostering the development of trustworthy frameworks will be crucial to RL adoption in mission-critical applications.

15. Pathways to Broader Adoption

To enable RL’s widespread adoption in network and service management, key challenges must be addressed, including:

• Enhancing sample efficiency through hybrid and offline approaches.

• Designing interpretable and robust RL solutions.

• Identifying use cases where RL provides significant advantages.

• Addressing operational constraints, such as QoS and scalability.

RL’s transformative potential lies in its ability to optimize complex, dynamic systems over time. However, achieving this vision requires a concerted effort to address the theoretical and practical challenges that remain.

• Mauro Tortonesi, University of Ferrara, Italy

• Guillaume Fraysse, Orange Labs, France

• José Santos, University of Ghent, Belgium

• Filippo Poltronieri, University of Ferrara, Italy