Learning Models

Individual Learning: Reinforcement

（权重）In reinforcement learning, an individual chooses actions based on the weights of those actions. Actions with a lot of weight are chosen more often than actions with little weight.

（奖励）The weight assigned to an action depends on the reward (payoff) that a person has received from taking that action in the past. This reinforcement of high-reward payoffs leads to better actions being taken.

（问题）The question we explore is whether reinforcement learning converges to only choosing the alternative with the highest reward.

（奖励数值化与记忆机制）At first, it may seem that to choose the most rewarding alternative is a trivial task. If the rewards are expressed in numerical form, such as money or time, we would expect people to choose the best. If rewards do not take numerical form, which is generally the case, people must rely on memory.

Edward Thorndike
（重复）law of effect - Repetition of an activity builds neurological pathways that induce that same behavior in the future.
（惊喜）surprise principle - more surprising rewards, rewards that far exceeded past or expected outcomes, produced faster learning in people.

（预期）In our reinforcement learning model, the weight assigned to a chosen alternative is adjusted based on how much the reward from that alternative exceeds our expectations (our aspiration level). This construction embeds both the law of effect (we take actions that produce higher rewards more often) and the surprise principle (the amount of weight we add to a choice depends on how much its reward exceeds the aspiration level).

A Reinforcement Learning Model

A collection of alternatives {A, B, C, D,…, N} have associated rewards {π(A), π(B), π(C), π(D),…,π(N)} and a set of strictly positive weights {w(A), w(B), w(C), w(D),…,w(N)}. The probability of choosing K is as follows:
$$
P(K)=\frac{w(K)}{w(A)+w(B)+w(C)+\dots+w(N)}
$$
$$
After\ choosing\ K, w(K)\ increases\ by\ γ · P(K) · (π(K)− A),
$$
$$
where\ γ > 0\ equals\ the\ rate\ of\ adjustment\ and\ A<max_{K} π(K)\ equals\ the\ aspiration\ level.
$$

γ - rate of adjustment
A - aspiration level

（收敛时间）Even with a low aspiration level, the alternatives with the highest rewards increase in weight the fastest, so the best alternative wins out in the long run. However, the time required for convergence on the best alternative may be long. It will also be true that as we add more alternatives, time to convergence also increases.

（动态参数）To avoid these complications, we can build in endogenous aspirations. We emend the model so that the aspiration level adjusts over time by setting it equal to the average reward.

In the learning-the-best-alternative framework, reinforcement learning with the aspiration level set equal to the average earned reward (eventually) almost always selects the best alternative.

（观察与模仿）Reinforcement learning assumes an individual acting in isolation. Social learning models assume that individuals see the actions and rewards of others. This can speed the rate of learning.

（奖励与从众）Replicator Dynamics, assumes that the probability of taking an action depends on the product of its reward and its popularity. We can think of the former as a reward effect and the latter as a conformity effect.

（行动与概率分布）Most often replicator dynamics models assume an infinite population. We can then characterize the actions taken as a probability distribution across the various alternatives. In the standard construction, time advances in discrete steps so that we can capture learning by changes in the probability distribution.

Replicator Dynamics

A collection of alternatives {A, B, C, D,…, N} have associated rewards {π(A), π(B), π(C), π(D),…, π(N)}. The actions of a population at time t can be written as a probability distribution across the N alternatives:
$$
(P_{t}(A), P_{t}(B),\dots,P_{t}(N)).
$$
The probability distribution changes according to the replicator equation:
$$
P_{t+1}(K)=P_{t}(K)\times(\frac{\pi(K)}{\bar{\pi_{t}}}),where\ \bar{\pi_{t}}\ equals\ the\ average\ reward\ in\ period\ t.
$$

As noted above, replicator dynamics includes a conformity effect (more popular alternatives are, more likely to be copied) as well as a reward effect. In the long run, the reward effect dominates, because high-reward alternatives always grow in proportion to lower-reward alternatives.

（路径依赖）In replicator dynamics, the average reward performs a function similar to that of the aspiration level in reinforcement learning when the aspiration level adjusts to equal the average reward. The only difference is that in replicator dynamics, we calculate the average reward for a population. In reinforcement learning, the aspiration level equals an individual’s average reward. That distinction matters insofar as a population provides a larger sample. Thus, replicator dynamics produce less path dependence than reinforcement learning.

（初始信息完备）In our construction of replicator dynamics, we assume that every alternative exists in the initial population. Given that the highest-reward alternative always has a higher-than-average reward and its proportion increases in every period,(eventually) replicator dynamics converge to the entire population choosing the best alternative.7 Thus, in a setting of learning the best alternative, both individual and social learning converge to the alternative with the highest reward.That will not be true in games.

In learning the best from a finite set of alternatives, replicator dynamics with an infinite population converges to the entire population choosing the best alternative.

Learning in Games

in a game, a player’s payoff depends both on her own action and on the actions of the other players. The payoff from a given action, such as cooperating in the Prisoners’ Dilemma, could be high in one period and low in the next depending on the action of the other player.

The Generous/Spiteful Game

（绝对or相对）The Generous/Spiteful Game, builds on a much-analyzed question about human behavior: Do we care more about our absolute or relative payoffs?

⚠️：和我的毕业论文思路非常相似

（国际政治中的新自由主义&新现实主义）A similar tension exists in foreign affairs. Neoliberals believe that countries want to maximize absolute payoffs measured by military power, economic prosperity, and domestic stability. Another camp, known as neorealists, believes that countries value relative payoffs. A country would rather have a lower absolute payoff but be stronger than its enemies. Kenneth Waltz, a neorealist, wrote at the height of the Cold War, “The first concern of states is not to maximize power but to maintain their positions in the system.“

Each of N players chooses to be generous G or spiteful S.
$$
Payoff(G,N_{G})=1+2\times N_{G}
$$
$$
Payoff(S,N_{G})=2+2\times N_{G}
$$
If we apply reinforcement learning in the Generous/Spiteful Game, the players learn to be generous.
If we apply replicator dynamics, the population learns to be spiteful.

（个体-绝对 vs 社会-相对）These findings highlight a key difference between individual and social learning. Individual learning leads people to choose the better action, so people learn a dominant action if one exists. Social learning leads people to choose actions that perform well relative to other actions.

“if people learn individually, they learn to act more generously than if they learn socially.”