Lecture 6: UCB1

Popular Bandit Algorithm: Upper Confidence Bound Algorithm

There are K arms, when pulled we get a payoff in some distribution. The distribution is stationary and has an expectation of q*(a)

n: denotes discrete times

Initialization: Pull each arm at least once

Loop:

Play arm j that maximizes

Q(j) + (2ln(n) / n_j)^0.5

Theorem:

For all K > 1, if UCB1 is run on K arms, having arbitrary reward distribution with support in [0, 1] then its expected regret after any number of plays n, is at most

Capital delta = expected loss when played i and not a* (a* is the optimal action)

Chernoff bounds -

Let X1, ….. Xn be random variables, in the range [0, 1] st, E[xt | x1 … xt-1] = u

Let Sn = (X1, ….. Xn )/n