## Mean Field Stochastic Games

We introduce evolving stochastic games with finite number players, in which each player in the population interacts with other randomly selected players (the number of players in interaction evolves in time). The states and actions of each player in an interaction together determine the instantaneous payoff for all involved players. They also determine the transition probabilities to move to the next state. Each individual wishes to maximize the total expected payoff over an infinite horizon. Under restricted class of strategies, the random process consisting of one specific player and the remaining population converges weakly to a jump process driven by the solution of a system of differential equations. We characterize the solutions to the team and to the game problems at the limit of infinite population. We prove that the large population asymptotic of the microscopic model is equivalent to a (macroscopic) stochastic evolutionary game in which a local interaction is described by a single player against a population profile. A new class of differential games called differential population games is obtained. In these games, an individual optimizes its expected fitness during its sojourn time in the system under population dynamics.

[audio: Mean field stochastic games, around 48mn]. Over 900 views from sep 26, 2011 to december 2012. Talk at Mean Field Games and related topics, 1st Workshop, Rome, May 12-13, 2011. [Slides].

Validity domain of Replica methods, Propagation of Chaos, Fixed-Point Analysis, Decoupling Assumption

Bellman-Shapley mean field optimality

Spatial non-reciprocal interactions and their asymptotics

Random number of interacting players, non-convergent dynamics and non-equilibrium behaviors

** **

**Mean field game dynamics**

We introduce a class of game dynamics obtained from asymptotic of dynamic games with variable number of interacting players. These dynamics cover the standard dynamics known in evolutionary games and population games. In the case of multi-class of players, the dynamics describe both the intra-population interaction and inter-population interaction. The stochastic mean field game dynamics are in general non-linear (partial or not) differential equations or inclusions. This last class occurs when the mean process converges weakly to another process described by a drift plus a noise. Keywords: Ito's formula, Kolmogoroff backward/forward equations, partial differential equations (PDE) or inclusions (PDI), stochastic integro-differential equations (SIDPE) etc.

- Spatial mean field game dynamics for hybrid networks
- Mobility-based dynamics, multicomponent dynamics for non-pairwise interaction
- Non-convergent dynamics and limit cycle in communication systems

References:

**Hybrid mean field game dynamics in large populations**, American Control Conference, pp.5109 - 5114 ACC 2011, San Francisco, California, US.- H. Tembine, J. Y. Le Boudec, R. ElAzouzi, E. Altman,
**From mean field interaction to evolutionary game dynamics**, IEEE/ACM Proc. WIOPT'2009, June 2009. - H. Tembine, E. Altman, R. ElAzouzi and W. H. Sandholm,
**Evolutionary game dynamics with migration for hybrid power control in wireless communications**, IEEE Proc. of 47th IEEE Conference on Decision and Control (CDC'2008), pages 4479 - 4484 December, 2008.

Below we list part of the content of the notes on **mean field stochastic games:**

**Mean Field Markov Games**

The central mean field issue is the development of low complexity solutions so that each player may implement a strategy based on local information in large populations. For models with mean field coupling, recent advances have been made in effectively addressing the complexity issue. In the mean field Markov game modeling, there must be an equation to express the optimization problem of each player. Usually this involves one equation for each player. If players are classified together by similar player classes, there is one equation per class. This equation is generally a Bellman equation, since a large proportion of optimization problems fall within this framework. An equation is also needed to express the class' behavior. When players are atomized, the group is represented in the modeling by a distribution on the state space of individual player; or, if there are several classes of players, by a distribution for each class. The dynamics of the distribution is governed by Kolmogorov-Chapman equation. In the Kolmogorov equation, the optimal behaviors of the players occur as data, since it is the infinite collection (the continuum) of individual behaviors that is aggregated and constitutes collective behavior. Thus, the modeling of the behavior of a group of players naturally leads to a Backward-Forward equation. The discrete time version of this problem have been studied by Jovanovic & Rosenthal in the eighty's. Our goal is to understand such games in both transient and asymptotic regimes.

References:

**Optimum and equilibrium in assignment problems with congestion: mobile terminals association to base stations,**IEEE Transactions on Automatic Control, accepted, 2012.**Mean field games in cognitive radio networks**, ACC 2012.**Energy-constrained mean field games in wireless networks**, Strategic Behavior and the Environment, Special Issue on: "ICT-based strategies for environmental conflicts, accepted 2012.**Mean field interaction in Chemical Reaction Networks**, Allerton Conference on Communication, Control and Computing, 28-30 Sept. 2011.**Mean field stochastic games: convergence, Q/H learning, optimality**, American Control Conference, pp. 2423 - 2428, ACC 2011, San Francisco, California, US.**Noisy Mean Field Stochastic Games with Network Applications**, ICIAM 2011 -- 7th International Congress on Industrial and Applied Mathematics, ICIAM Vancouver, MS403 Multiscale Approximations Of Kinetic Monte Carlo Simulations.- H. Tembine, M. Huang:
**Mean field difference games: McKean-Vlasov dynamics**. Proceedings of the 50th IEEE Conference on Decision and Control and European Control Conference, CDC-ECC 2011, Orlando, FL, USA, December 12-15, 2011. - H. Tembine, P. Vilanova and M. Debbah,
**Noisy mean field game model for malware propagation in opportunistic networks**, 2nd International ICST Conference on Game Theory for Networks, Gamenets, April 16-18, 2011 - Shanghai, China. - H. Tembine, P. Vilanova, M. Assaad, M. Debbah,
**Mean field stochastic games for SINR-based access control,**Gamecomm, 4th International Workshop on Game Theory in Communication Networks, 2011. - H. Tembine,
**Large-scale games in large-scale systems,**Valuetools, 5th International ICST Conference on Performance Evaluation Methodologies and Tools, May 16-20, 2011, ENS, Cachan, France. - H. Tembine,
**A mean field stochastic game approach to power management,**18th World Congress of the International Federation of Automatic Control (IFAC), 2011 **Markov Decision Evolutionary Games with Individual Energy Management,**in Annals of the International Society of Dynamic Games,Advances in Dynamic Games: Theory, Applications, and Numerical Methods, Editors: Michele Breton,Krzysztof Szajowski, pages 313-335, 2010.- E. Altman, Y. Hayel, H. Tembine, and R. El-Azouzi,
**Markov Decision Evolutionary Games with Expected Average Fitness**., Evolutionary Ecology Research Journal, Special issue in honour of Thomas Vincent (Joel S. Brown & Tania L. S. Vincent, Eds), May 2009. - H. Tembine, J. Y. Le Boudec, R. ElAzouzi, E. Altman,
**Mean Field Asymptotic of Markov decision evolutionary games**, International IEEE Conference on Game Theory for Networks, Gamenets 2009. - H. Tembine, Eitan Altman, Rachid El-Azouzi, Yezekael Hayel:
**Battery State-Dependent Access Control in Solar-Powered Broadband Wireless Networks,**Lecture Notes in Computer Science Volume 5425, 2009, pp 121-129

** Risk-Sensitive Mean Field Games**

The classical utility maximization, also referred as risk-neutral approach, consists to optimize the expected payoff, loss or performance of the stochastic dynamic system. However, not all behaviors can be captured the risk-neutral approach. In this project we aim to develop a larger class of players behaviors within the framework of mean field games in large-scale systems, including both risk-sensitive and risk-neutral players. We will distinguish two type of payoffs:

- Exponentiated cost integral risk-sensitive mean field games and

in which the objective of each generic player is to find a tradeoff between the mean and the variance of the losses. This is of importance in many applications including financial markets, economics of cloud computing as well as economics of the smart grid.

References:

- H. Tembine, Q. Zhu, T. Basar,
**Risk-sensitive mean field games,**IEEE Transactions on Automatic Control, conditionally accepted, 2012. - H. Tembine,
**Risk-sensitive mean field stochastic games**, CDC-ECC, 50th IEEE Conference on Decision and Control and European Control Conference, December 12-15, 2011, Orlando, Florida. - Q. Zhu, H. Tembine, T. Basar,
**Hybrid Risk-Sensitive Mean-Field Stochastic Differential Games with Application to Molecular Biology**, 50th IEEE Conference on Decision and Control and European Control Conference, December 12-15, 2011, Orlando, Florida. - H. Tembine,
**Risk-sensitive mean field stochastic games**, Workshop mean field games and related topics, Rome, May 12-13, 2011. - H. Tembine, Q. Zhu, T. Basar,
**Risk-sensitive mean field stochastic differential games**, 18th World Congress of the International Federation of Automatic Control (IFAC), 2011.

**Robust Mean Field Games**

Robust games is a branch of game theory that explicitly deals with uncertainty in the payoff function and eventually in the state. Robust games methods are designed to function properly so long as uncertain parameters or disturbances are within some appropriated set. Based on H-infinity control and robust control, one can establish a relation between risk-sensitive games and risk-neutral games via robust methods in the context of large number of players i.e. mean field games under uncertainty. Robust mean field games aim to achieve robust performance and/or stability in the presence of errors/disturbance with large number of players.

References:

- D. Bauso, H. Tembine, T. Basar,
**Robust Mean Field Games with Application to Production of an Exhaustible Resource,**7th IFAC Symposium on Robust Control Design, (ROCOND'12), 2012. - D. Bauso, H. Tembine, T. Basar,
**Robust mean field games with application to resource extraction**, 12th Viennese Workshop on Optimal Control, Dynamic Games and Nonlinear Dynamics, 2012

** Anomalous Mean Field Games**

An increasing number of natural phenomena do not fit into the description of diffusion developed by Bachelier, Einstein a century ago. The fractional Brownian motion (fBm) process, that is a stochastic process which would be almost surely continuous, Gaussian with mean 0 and have a non-linear covariance function. The concepts of anomalous dynamical properties, such as long-range spatial or temporal correlations manifested in power laws, stretched exponentials, fractional-noises, or non-Gaussian probability density functions, have been predicted and observed in numerous systems from various disciplines including astrophysics, physics, networks, chemistry, engineering, geology, biology, economy, meteorology and others. Based on these observations one has attempted to say that ''Anomalous is normal''. Based on empirical measurement and observations of Nile flows in 622-1469, Hurst discovered fractional scale. Motivated by these empirical observations, Mandelbrot has given the name of Hurst parameter to the parameter H of fBm. Anomalous diffusion also called S-diffusion (sub- or super-diffusion) have been observed in 1926 by Richardson. Anomalous diffusion creates a novel class of games called anomalous mean field games. We analyze large classes of stochastic games with individual states driven by controlled fractional Brownian motions. Under specific structure of transition kernels (satisfying the asymptotic indistinguishability property), mean-field limit arises because of the large number of players. Due to non-Markovian nature of anomalous diffusion, the mean field dynamics is expressed in terms fractional Fokker-Planck-Kolmogorov equation using fractional derivative framework.

References:

- H. Tembine,
**Anomalous mean field games**, working paper, 2011.

** Non-Asymptotic Mean Field Games**

Mean field games have been studied under the assumption of very large number of players. For such large systems, the basic idea consists to approximate large game by a stylized game model with a continuum of players. The approach has been shown to be useful in many applications in economics and engineering. However, the stylized game model with continuum of decision-makers assumption is rarely observed in practice and the approximation proposed in the asymptotic regime is meaningless for networks with few entities. This project aims to develop a mean-field framework that is suitable not only for large systems but also for a small world with few number of entities.

References:

- H. Tembine,
**Non-asymptotic mean-field games**, presented at NetGCOOP 2012.

** Mean Field Learning**

The goal is to develop model-based (but still with less information) and non-model-based learning schemes for games with continuous action space and large number of players. Each player will update her learning strategies based on an aggregative term, formed from the distribution of action of the other players. Each player will be influenced by the aggregate, and the mean field behavior is formed from the contributions of each player.

- Fully distributed mean-field learning
- No-feedback mean-field learning
- Hybrid mean-field learning

References:

**Mean-field learning**, work in progress, 2012**Cloud Networking Mean Field Games**, IEEE CloudNet'12.**Hybrid Mean Field Learning in Large-Scale Dynamic Robust Games**, International Conference on Control and Optimization with Industrial Applications, COIA 2011.

** Applications: Incentives against Theft in Power Grid**

Saurabh Amin, G. Schwartz, H. Tembine,**Incentives and security in electric distribution networks**, Jens Grossklags, Jean C. Walrand (Eds.): Decision and Game Theory for Security - Third International Conference, pp. 264-280, Lecture Notes in Computer Science 7638 Springer 2012, ISBN 978-3-642-34265-3

** Peaks reduction in power grid**

Galina Schwartz , H. Tembine, Saurabh Amin, Shankar Sastry,**Electricity demand response via randomized rewards: A Mean Field Game Approach,**50th Annual Allerton Conference on Communication, Control, and Computing, Allerton Retreat Center, Monticello, Illinois, October 2012.

** **Cloud Networks

Ahmed Farhan Hanif, H. Tembine, Mohamad Assaad, Djamal Zeghlache,**Mean****field games for Resource Sharing in Cloud Networks, working paper, 2012**Ahmed Farhan Hanif, H. Tembine, Mohamad Assaad, Djamal ZeghlacheIEEE CloudNet'12.**,****Cloud Networking Mean Field Games**,

** Reacting to the Interference Field in Wireless Networks**

**M. Debbah and Tembine H.,**Mechanisms and Games for Dynamic Spectrum Allocation: Cambridge University Press, 2012.**Reacting to the Interference Field,**H. Tembine,**Energy-constrained mean field games in wireless networks**, Strategic Behavior and the Environment, Special Issue on: "ICT-based strategies for environmental conflicts, accepted 2012, In Press.H. Tembine, R. Tempone, P. Vilanova,2012 American Control Conference, June 27 - 29, Montreal, Canada.**Mean field games in cognitive radio networks,**H. Tembine, P. Vilanova, M. Assaad, M. Debbah,**Mean field stochastic games for SINR-based access control,**Gamecomm, 4th International Workshop on Game Theory in Communication Networks, 2011.-
H. Tembine,**Large-scale games in large-scale systems**, Valuetools, 5th International ICST Conference on Performance Evaluation Methodologies and Tools, May 16-20, 2011, ENS, Cachan, France.

** Power management and quality of service:**** **

H. Tembine**,****A**18th World Congress of the International Federation of Automatic Control (IFAC), 2011**mean field stochastic game approach to power management,**H. Tembine, S. Lasaulce, and M. Jungers,**Joint power control-allocation for green cognitive wireless networks using mean field theory,**IEEE Proc. of the 5th Intl. Conf. on Cognitive Radio Oriented Wireless Networks and Communications (CROWNCOM), Cannes, France, Jun. 2010.- Debbah, M. and Lasaulce, S. and LeTreust, M. and Tembine, H. ,
**Controle de puissance dynamique pour les communications sans fils,**2010, Actes Societe Francaise de Recherche Operationnelle et d'Aide a la Decision (ROADEF), Toulouse, France H. Tembine, Eitan Altman, Rachid El-Azouzi, Yezekael Hayel:**Battery State-Dependent Access Control in Solar-Powered Broadband Wireless Networks,**Lecture Notes in Computer Science Volume 5425, 2009, pp 121-129Tembine, H. and Altman, E. and ElAzouzi, R. and Hayel, Y.:Proceedings of the 3rd International Conference on Performance Evaluation Methodologies and Tools, Athens, Greece, October 2008**Stochastic population games with individual independent states and coupled constraints,**

** **