Newton's law of gravitation states that any two objects with mass attract each other with equal forces parallel to the line connecting their centers. The value or strength of these forces given by formula below.

where m_1m1 ad m_2m2 are masses of these objects, rr is distance between them, and G=6.67*10^-11 ("N m"^2)/"kg"^2G=6.67⋅10−11N m2kg2 is a constant found experimentally by Cavendish (assuming that at least one of the objects is small relative to distance rr).
From the picture we see that forces are equal even if masses are not.
Now we're ready to tackle this problem.
![made by me]()
All masses are equal to mm.
Calculating values of all 3 forces using our magic formula:
F_1=F_2=G(m*m)/a^2=Gm^2/a^2F1=F2=Gm⋅ma2=Gm2a2
The diagonal of this square has length asqrt2a√2, so
F_3=G(m*m)/(asqrt2)^2=Gm^2/(2a^2)F3=Gm⋅m(a√2)2=Gm22a2
Since F_1=F_2F1=F2 and vec(F_1)−→F1 is perpendicular to vec(F_2)−→F2, the vector vec(F_(12))=vec(F_1)+vec(F_2)−→F12=−→F1+−→F2 is parallel to the diagonal and vec(F_3)−→F3 and F_(12)=sqrt2F_1F12=√2F1.
Because of it, the net force is parallel to the diagonal and
F_"net"=sqrt2F_1+F_3=sqrt2Gm^2/a^2+Gm^2/(2a^2)
=(sqrt2+1/2)Gm^2/a^2Fnet=√2F1+F3=√2Gm2a2+Gm22a2=(√2+12)Gm2a2
Plugging values in gives us
F_"net"=(sqrt2+1/2)(6.67*10^-11("N m"^2)/"kg"^2)(800"kg")^2/(0.1"m")^2Fnet=(√2+12)(6.67⋅10−11N m2kg2)(800kg)2(0.1m)2
F_"net"=(1.41+0.5)(6.67*10^-11("N"cancel("m"^2))/cancel("kg"^2))(64*10^4cancel("kg"^2))/(10^-2cancel("m"^2))
F_"net"=(1.91)(6.67*10^-11)6.4*10^5*10^2"N"
F_"net"=1.91*6.67*6.4*10^-4"N"
Grabbing a calculator
F_"net"=81.5*10^-4"N" (truncating, because sqrt2 is already truncated)
F_"net"=8.15*10^-3"N"=8.15"mN" (millinewtons)