After applying the composition / to the point (–5, 3), in which quadrant does the image lie?

the first one, all those points on the line just make a strait line.

check above link.

graphically?   well, you graph them both, the solutions is where they both intersect each other, since there are two solutions, meaning they will meet twice, or namely the linear will run into the quadratic curve twice, and where they meet, those x,y coordinates are the solutions to the system.