Given the function f(x)=3x+5a / x^2-a^2 find the value of a for which f'(12) = 0

The function, as presented here, is ambiguous in terms of what's being deivded by what.  For the sake of example, I will assume that you meant

           3x+5a
 f(x)=
          x^2-a^2

You are saying that the derivative of this function is 0 when x=12.  Let's differentiate f(x) with respect to x and then let x = 12:

             (x^2-a^2)(3) -(3x+5a)(2x)
f '(x) = = 0 when x = 12
                [x^2-a^2]^2

(144-a^2)(3) - (36+5a)(24)
 =  0
               [   ]^2

Simplifying,

(144-a^2) - 8(36+5a) = 0

144 - a^2 - 288 - 40a = 0

This can be rewritten as a quadratic in standard form:

-a^2 - 40a - 144 = 0, or a^2 + 40a + 144 = 0.

Solve for a by completing the square:

a^2 + 40a + 20^2 - 20^2 + 144 = 0
        (a+20)^2    = 400 - 144 = 156

        Then a+20 = sqrt[6(26)] = sqrt[6(2)(13)] = 4(3)(13)= 2sqrt(39)

         Finally, a = -20 plus or minus 2sqrt(39)

You must check both answers by subst. into the original equation.  Only if the result(s) is(are) true is your solution (value of a) correct.