Which of the following is most likely the next step in the series?

A.
This is because in each circle the lines increase by one, so from the triangle (3 lines) to the rectangle (4 lines) it only makes sense to put a pentagon (5 lines)

No a triangle cannot be formed because when you add the two smallest number together (which would be 11) the have to be bigger than the number it is "against" ( which would be 12)