Simply replace the #x# in the function's formula with whatever you see in the parentheses to evaluate #g# at that point. We get:
#g(5n) = (5n)^2 - 5n#
Recall that #(ab)^2 = a^2b^2#, so:
#g(5n) = 25 * n^2 - 5n#
The tricky part here is that parentheses like #(5n)^2# are often omitted (which is obviously a mistake) and are often interpreted like #5n^2#, which just means #5# times #n^2#, or #25n#, which means #5^2 * n# (again, those are just common mistakes). However, the correct way is to always use them when substituting a "bigger" term in place of a smaller one, for example when using #abc# instead of #d# in the formula #d^3#, the correct substitution is #(abc)^3 = a^3b^3c^3#, and none other.