Consider a mdp with reward function r(s) and transition model p(s0 j s; a). instead of a deterministic policy (s) = a, which assigns a single optimal action a for each state s, consider allowing probabilistic policies (s) = p(a j s), where p(a j s) is a probability distribution over possible actions. write the bellman equation for this formulation keeping in mind the de nition of the utility of a state.

Assume that two parallel arrays have been declared and initialized: healthoption an array of type char that contains letter codes for different healthcare options and annual cost an array of type int. the i-th element of annual cost indicates the annual cost of the i-th element of healthoption. in addition, there is an char variable, best2.write the code necessary to assign to best2 the health option with the lower annual cost, considering only the first two healthcare options. thus, if the values of healthoption are 'b', 'q', 'w', 'z' and the values of annualcost are 8430, 9400, 7050, 6400 your code would assign 'b' to best2 because 8430 is less than 9400 and is associated with 'b' in the parallel array. (we ignore 'w' and 'z' because we are considering only the first two options.)