You need to compute the second derivative in the point #x=0#, and check its sign. If it's positive, the function is convex, otherwise it's concave. So, we have
#f(x) = xe^x-x^2#
#f'(x) = e^x + xe^x-2x#
#f''(x) = e^x + e^x + xe^x-2 = e^x(2+x)-2#
So, #f''(0)=e^0(2+0)-2 = 2-2=0#.
Since the second derivative is negative immediatly before #0#, and positive after, #x=0# is an inflection point, in which the function passes from being concave to be convex, as you can see in this graph, in which the scales of #x# and #y# axes are changed in order to emphatize the behaviour of the curve.
