Newton's law of gravitation states that any two objects with mass attract each other with equal forces parallel to the line connecting their centers. The value or strength of these forces given by formula below.
where #m_1# ad #m_2# are masses of these objects, #r# is distance between them, and #G=6.67*10^-11 ("N m"^2)/"kg"^2# is a constant found experimentally by Cavendish (assuming that at least one of the objects is small relative to distance #r#).
From the picture we see that forces are equal even if masses are not.
Now we're ready to tackle this problem.
All masses are equal to #m#.
Calculating values of all 3 forces using our magic formula:
#F_1=F_2=G(m*m)/a^2=Gm^2/a^2#
The diagonal of this square has length #asqrt2#, so
#F_3=G(m*m)/(asqrt2)^2=Gm^2/(2a^2)#
Since #F_1=F_2# and #vec(F_1)# is perpendicular to #vec(F_2)#, the vector #vec(F_(12))=vec(F_1)+vec(F_2)# is parallel to the diagonal and #vec(F_3)# and #F_(12)=sqrt2F_1#.
Because of it, the net force is parallel to the diagonal and
#F_"net"=sqrt2F_1+F_3=sqrt2Gm^2/a^2+Gm^2/(2a^2)
=(sqrt2+1/2)Gm^2/a^2#
Plugging values in gives us
#F_"net"=(sqrt2+1/2)(6.67*10^-11("N m"^2)/"kg"^2)(800"kg")^2/(0.1"m")^2#
#F_"net"=(1.41+0.5)(6.67*10^-11("N"cancel("m"^2))/cancel("kg"^2))(64*10^4cancel("kg"^2))/(10^-2cancel("m"^2))#
#F_"net"=(1.91)(6.67*10^-11)6.4*10^5*10^2"N"#
#F_"net"=1.91*6.67*6.4*10^-4"N"#
Grabbing a calculator
#F_"net"=81.5*10^-4"N"# (truncating, because #sqrt2# is already truncated)
#F_"net"=8.15*10^-3"N"=8.15"mN"# (millinewtons)