If the expression ax^4 + bx^3 – x² + 2x + 3 has a remainder 3x+5
when it is divided x^2-x-2 find the values a and b

There's probably a more efficient way to do it, but the following method works.

First, note that

x² - x - 2 = (x + 1) (x - 2)

Now use synthetic division to divide

a x⁴ + b x³ - x² + 2x + 3

by the two factors above.

Division by x + 1 :

-1   |   a           b               -1                   2                3

…   |               -a          a - b      -a + b + 1      a - b - 3

= = = = = = = = = = = = = = = = = = = = = = = = = = = = =

…   |   a      b - a     a - b - 1      -a + b + 3           a - b

which is to say,

(a x⁴ + b x³ - x² + 2x + 3) / (x + 1)

= a x³ + (b - a) x² + (a - b - 1) x + (-a + b + 3) + (a - b) / (x + 1)

Division by x - 2 :

2   |    a        b - a           a - b - 1          -a + b + 3

…   |                 2a          2a + 2b       6a + 2b - 2

= = = = = = = = = = = = = = = = = = = = = = = = = = =

…   |    a       a + b        3a + b - 1       5a + 3b + 1

meaning,

(a x³ + (b - a) x² + (a - b - 1) x + (-a + b + 3)) / (x - 2)

= a x² + (a + b) x + (3a + b - 1) + (5a + 3b + 1) / (x - 2)

Putting everything together, we have

(a x⁴ + b x³ - x² + 2x + 3) / (x² - x - 2)

= a x² + (a + b) x + (3a + b - 1) + (5a + 3b + 1) / (x - 2) + (a - b) / (x² - x - 2)

Combine the remainder terms:

(5a + 3b + 1) / (x - 2) + (a - b) / (x² - x - 2)

= ((5a + 3b + 1) x + (6a + 2b + 1)) / (x² - x - 2)

We're given that the remainder is 3x + 5, so

5a + 3b + 1 = 3

6a + 2b + 1 = 5

Solve this system for a and b :

5a + 3b = 2

6a + 2b = 4

5a + 3b = 2

3a + b = 2

5a + 3b = 2

-9a - 3b = -6

(5a + 3b) + (-9a - 3b) = 2 + (-6)

-4a = -4

a = 1

5 • 1 + 3b = 2

3b = -3

b = -1