Recall the conservation of momentum, which states that the total momentum #m_iv_i# remains constant. Thus, it can be deduced that the initial and final momentums of the ball are conserved (since we assume that no momentum is transferred to the surroundings). Then, if #v_1# and #v_2# are the final of the two balls, we know that
#6*1+1*-3=1*v_1+1*v_2#
#3=v_1+v_2#
where the left-hand side is the sum of the momentum of the balls before collision and the right-hand side is the sum after collision.
Now, let us also calculate the kinetic energy (#1/2m_iv_i^2# before and after collision.
Before collision: #1/2*1*6^2+1/2*1*(-3)^2=45/2#
After collision: #1/2*1*v_1^2+1/2*1*v_2^2=1/2v_1^2+1/2v_2^2#
We know that #20%# of the kinetic energy is lost due to the collision. Therefore, the kinetic energy after collision is #80%# of the kinetic energy before:
#45/2*0.8=1/2v_1^2+1/2v_2^2#
#36=v_1^2+v_2^2#
So now we have two equations:
#{(v_1+v_2=3\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (1)),(v_1^2+v_2^2=36\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ (2)):}#
From equation #(1)#, we know that #v_1=3-v_2#. Substitute this into equation #(2)# to get
#(3-v_2)^2+v_2^2=36#
#9-6v_2+2v_2^2=36#
#2v_2^2-6v_2-27=0#
Use the quadratic equation to get that
#v_2=(6+-sqrt((-6)^2-4*2*(-27)))/(2*2)=(3+-3sqrt(7))/2#
However, if you look back at the problem, the second ball originally has #-3m/s# velocity before colliding. It would be impossible
and thus
#v_1=3-(3+-3sqrt(7))/2=(3\ \bar"+"\ 3sqrt(7))/2#
It seems like there are two answer options. One where #v_1# is negative while #v_2# is positive, and the other one where #v_1# is positive and #v_2# is negative.
However, if #v_1# is positive and #v_2# is negative, it would mean that the two balls both move in the same direction even after collision, which is absurd (in fact, this solution corresponds to a theory of waves, which can pass through each other).
Thus, we have
#v_1=(3-3sqrt(7))/2#
and
#v_2=(3+3sqrt(7))/2#