The simplest means of doing this is to graph the function.
graph{x^5lnx [-10, 10, -5, 5]}
The function exists on the interval from #(0,oo)#. On this interval #x^5# is always positive, and #ln(x)# is negative until #x=1#. Looking at the graph, we know that the function will be concave upwards and increasing after #x=1#, but via taking the derivatives we can find when exactly the change from decreasing to increasing occurs.
We must use the product rule for this, which states that for #f(x)=g(x)h(x), f'(x) = g'(x)h(x)+g(x)h'(x)#. Thus here...
#f(x) = x^5ln(x) -> f'(x) = 5x^4lnx + x^5/x = 5x^4lnx + x^4 = x^4(5lnx +1)#.
We can find where the change from #f'(x)<0 # to #f'(x)>0# occurs by finding where f'(x)=0. From our equation, this cannot happen at #x=0# because the natural log is undefined there. Therefore, setting the equation equal to zero...
#f'(x) = x^4(5lnx +1) = 0 -> 5lnx +1 = 0 -> lnx = -1/5 -> x = e^(-1/5)# is our critical point. Plugging in values slightly larger and slightly smaller than this for x reveals that for #x < e^(-1/5), f'(x)<0#, and for #x>e^(-1/5), f'(x)>0#. Therefore the function is decreasing on #(0,e^(-1/5))#, and increasing on #(e^(-1/5), oo)#
To find the concavity, we must instead find #f''(x)# in the same way as we found the first derivative:
#f''(x) = d/dx(5x^4lnx + x^4) = (20x^3lnx + 5x^3 + 4x^3 = 20x^3lnx + 9x^3 = x^3(20lnx + 9)#.
The concavity can only switch at a point where this second derivative is equal to zero. As above, we know that x=0 is not a zero of this second derivative because the natural log is undefined there...
#f''(x) = 0 = x^3(20lnx + 9) = 20lnx + 9 -> 20lnx = -9 -> lnx = -9/20 -> x = e^(-9/20)#
This is our zero for the second derivative. As above, plugging in a slightly smaller x (i.e. #e^(-1/2)#) yields a negative value (#lnx = -1/2< -9/20 -> 20lnx+9<0 -> x^3(20lnx+9)<0#), and using a larger x yields a positive value, i.e. #x=e -> ln e = 1 >-9/20 -> e^3(20lne+9) = e^3(20+9) = e^3(29)>0#. Thus, the function is concave down for #(0,e^(-9/20))#, and concave up for #(e^(-9/20),oo)#