Logo Flanders AI Research Program
NL
Overview

Dual Bandit algorithm

‘Dual Bandit’ algorithm, as presented in the publication ‘Joint Policy-Value Learning for Recommendation’

Visual of a bandit
Description

Conventional approaches to recommendation often do not explicitly take into account information on previously shown recommendations and their recorded responses. One reason is that, since we do not know the outcome of actions the system did not take, learning directly from such logs is not a straightforward task. Several methods for off-policy or counterfactual learning have been proposed in recent years, but their efficacy for the recommendation task remains understudied. Due to the limitations of offline datasets and the lack of access of most academic researchers to online experiments, this is a non-trivial task. Simulation environments can provide a reproducible solution to this problem.

We conducted the first broad empirical study of counterfactual learning methods for recommendation, in a simulated environment. We consider various different policy-based methods that make use of the Inverse Propensity Score (IPS) to perform Counterfactual Risk Minimisation (CRM), as well as value-based methods based on Maximum Likelihood Estimation (MLE). We present a “Dual Bandit” algorithm that combines both the MLE and CRM objectives, using that the value- and policy-based methods have an identical parameterisation under certain assumptions. Extensive experiments show that this “Dual Bandit” approach achieves stateof-the-art performance in a wide range of scenarios, for varying logging policies, action spaces and training sample sizes.

More information
Publications
  • Olivier Jeunen, David Rohde, Flavian Vasile, and Martin Bompaire. “Joint Policy-Value Learning for Recommendation”. Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovey & Data mining, 2020.