You need to compute the second derivative in the point x=0x=0, and check its sign. If it's positive, the function is convex, otherwise it's concave. So, we have
f(x) = xe^x-x^2f(x)=xex−x2
f'(x) = e^x + xe^x-2x
f''(x) = e^x + e^x + xe^x-2 = e^x(2+x)-2
So, f''(0)=e^0(2+0)-2 = 2-2=0.
Since the second derivative is negative immediatly before 0, and positive after, x=0 is an inflection point, in which the function passes from being concave to be convex, as you can see in this graph, in which the scales of x and y axes are changed in order to emphatize the behaviour of the curve.
