Newton's law of gravitation states that any two objects with mass attract each other with equal forces parallel to the line connecting their centers. The value or strength of these forces given by formula below.

where m_1 ad m_2 are masses of these objects, r is distance between them, and G=6.67*10^-11 ("N m"^2)/"kg"^2 is a constant found experimentally by Cavendish (assuming that at least one of the objects is small relative to distance r).
From the picture we see that forces are equal even if masses are not.
Now we're ready to tackle this problem.
![made by me]()
All masses are equal to m.
Calculating values of all 3 forces using our magic formula:
F_1=F_2=G(m*m)/a^2=Gm^2/a^2
The diagonal of this square has length asqrt2, so
F_3=G(m*m)/(asqrt2)^2=Gm^2/(2a^2)
Since F_1=F_2 and vec(F_1) is perpendicular to vec(F_2), the vector vec(F_(12))=vec(F_1)+vec(F_2) is parallel to the diagonal and vec(F_3) and F_(12)=sqrt2F_1.
Because of it, the net force is parallel to the diagonal and
F_"net"=sqrt2F_1+F_3=sqrt2Gm^2/a^2+Gm^2/(2a^2)
=(sqrt2+1/2)Gm^2/a^2
Plugging values in gives us
F_"net"=(sqrt2+1/2)(6.67*10^-11("N m"^2)/"kg"^2)(800"kg")^2/(0.1"m")^2
F_"net"=(1.41+0.5)(6.67*10^-11("N"cancel("m"^2))/cancel("kg"^2))(64*10^4cancel("kg"^2))/(10^-2cancel("m"^2))
F_"net"=(1.91)(6.67*10^-11)6.4*10^5*10^2"N"
F_"net"=1.91*6.67*6.4*10^-4"N"
Grabbing a calculator
F_"net"=81.5*10^-4"N" (truncating, because sqrt2 is already truncated)
F_"net"=8.15*10^-3"N"=8.15"mN" (millinewtons)