Venue : CGFB
Defended in english
Bhargav Teja Nallapu
IMN / Labri
Thesis supervisors: Frédéric ALEXANDRE, Thierry VIEVILLE
A closed loop framework of decision making and learning in primate prefrontal circuits: Computational modeling and Virtual experimentation
Abstract
This thesis attempts to build a computational systems-level framework that would help to develop an understanding of the organization of the prefrontal cortex (PFC) and the basal ganglia (BG) systems and their functional interactions in the process of decision-making and goal-directed behaviour in humans. A videogame environment, Minecraft is used to design experiments to test the framework in an environment that could be more complex and realistic, if necessary. The framework, along with virtual experimentation forms a closed-loop architecture for studying the high-level animal behavior. The neural systems framework in this work rests on the network dynamics between the subsystems of PFC and BG. PFC is believed to play a crucial role, in executive functions like planning, attention, goal-directed behavior, etc. BG are a group of sub-cortical nuclei that have been extensively studied in the field of motor control and action selection. Different regions in the PFC and structures within BG are anatomically organized, in parallel and segregated loops (each of them referred as a CBG loop). These loops can be, on a high level, divided into 3 kinds : limbic loops, associative loops and sensori-motor loops.
First, a comprehensive framework with the above mentioned parallel loops is implemented. The emphasis rests on the limbic loops. Therefore the associative and sensori-motor loops are modeled algorithmically, taking help of the experimentation platform for motor control. As for the limbic loops, the orbitofrontal cortex (OFC) is the part of a loop for preferences and the anterior cingulate cortex (ACC), for internal needs. These loops are formed through their limbic counterpart in BG, ventral striatum (VS). VS has been widely studied and reported to be encoding various substrates of value, forming an integral part of value-based decision making. Simplistic scenarios are designed in the virtual environment using the agent and some objects and appetitive rewards in the environment. The limbic loops have been implemented according to existing computational models of decision making in the BG. Thus the framework and the experimental platform stand as a testbed to computational models of specific processes that have to fit in a bigger picture. Next, we use this framework to study more closely, the role of OFC in value-guided decision making and goal-directed behavior. As part of this thesis, several outstanding observations about the role of OFC in behavior have been summarized by consolidating numerous experimental evidences and reviews.
Lastly, to explain the findings of different roles of lateral and medial regions of OFC, existing computational architecture of CBG loops, pavlovian learning in amygdala and multiple evidences of amygdala-OFC-VS interactions are put together into a single model. The learning rules of reinforcement have been adapted to accommodate the appropriate credit assignment (correct outcome to correct chosen stimulus) and the value difference of the choice options. As a result, several findings from animal experiments studying the separable roles, were replicated. Difference in choice impairments depending on the value difference between the best and the second best option is one of them. Dissociable roles in Pavlovian Instrumental Transfer were also observed. The investigations into the observed evidences around OFC offer great insight into understanding the very process of decision-making, value computation in general. By venturing into a realm of bio-inspired adaptive learning in an embodied virtual agent, describing the principles of motivation, goal-selection and self-evaluation, it is highlighted that the field of reinforcement learning and artificial intelligence has a lot to gain from studying the role of prefrontal systems in decision-making.
Keywords: decision-making, learning, prefrontal cortex, virtual experimentation, goal-directed behaviour, computational modeling
Publications
Published pesearch :
Nallapu, B. and Alexandre, F. (2018). Cognitive Architecture and Software Environment for the Design and Experimentation of Survival Behaviors in Artificial Agents. In Proceedings of the 10th International Joint Conference on Computational Intelligence – Volume 1: IJCCI, ISBN 978-989-758-327-8, pages 152-159. DOI: 10.5220/0006920301520159
Nallapu B.T., Alexandre F. (2018) A Survival Task for the Design and the Assessment of an Autonomous Agent. In: Vouloutsi V. et al. (eds) Biomimetic and Biohybrid Systems. Living Machines 2018. Lecture Notes in Computer Science, vol 10928. Springer, Cham
Article in preparation :
Nallapu B.T., Alexandre F. Interacting roles of lateral and medial Orbitofrontal Cortex in decision-making and learning : A systems-level computational model.
Jury
NIKOLSKI Macha – CNRS – Président
CANAMERO Lola – University of Hertfordshire, U.K – Rapporteur
BOURET Sebastien – CNRS – Rapporteur
KHAMASSI Mehdi – CNRS – Examinateur
CHAKRAVARTHY Srinivasa – I.I.T Madras, India – Examinateur
ALEXANDRE Frédéric – INRIA – Directeur de thèse
VIEVILLE Thierry – INRIA – Co-directeur de thèse