For spring extension is xi = x_B - x_A, Newton's 2nd Law applied to blocks is:
Block A: qquad k xi - F = m ddot x_A
Block B: qquad 2F - k xi = 2m ddot x_B
Newton's Law for the system (ignoring spring as it is in the system):
2F - F = (m_A + m_B) ddot x_(cm) implies F = 3m ddot x_(cm) qquad triangle
Centre of mass of system:
x_(cm) = (m_A x_A + m_Bx_B)/(m_A + m_B) = 1/3x_A + 2/3 x_B, so:
....so:
((xi),(x_(cm))) = ((-1,1),(1/3,2/3)) ((x_A),(x_B))
Inverting this:
((x_A),(x_B)) = ((-2/3,1),(1/3,1)) ((xi),(x_(cm)))
and
((ddot x_A),(ddot x_B)) = ((-2/3,1),(1/3,1)) ((ddot xi),(ddot x_(cm)))
Our equations for A& B become:
Block A: qquad k xi - F = m ddot x_(cm) - 2/3 m ddot xi
Block B: qquad 2F - k xi = 2m ddot x_(cm) + 2/3 m ddot xi
We can eliminate x_(cm) using triangle:
Block A: qquad k xi - F = F/3 - 2/3 m ddot xi
implies ddot xi + (3k)/(2m) xi = 2 F/m
This solves as:
xi(t) = alpha sin(omega t) + beta cos(omega t) + (4 F)/(3 k) with omega = sqrt( (3k)/(2m))
xi(0) = 0 implies beta = -(4 F)/(3 k)
xi' = alpha omega cos(omega t) - beta omega sin(omega t )
xi'(0) = 3u implies alpha omega = 3u
implies xi(t) = (3u)/omega sin(omega t) -(4 F)/(3 k) cos(omega t) + (4 F)/(3 k)
=Gamma( ((3u)/omega)/Gamma sin(omega t) -((4 F)/(3 k) )/Gamma cos(omega t)) + (4 F)/(3 k)
where Gamma = sqrt(((3u)/omega)^2 + ((4 F)/(3 k) )^2) = sqrt(16 F^2 ω^2 + 81 k^2 u^2)/(3k omega)
=sqrt(16 F^2 ω^2 + 81 k^2 u^2)/(3k omega) sin(omega t - phi) + (4 F)/(3 k)
EDIT: Now omega^2 = (3k)/(2m) so this simplifies further to:
=sqrt((16 F^2)/(9 k^2) + (6 m u^2)/(k) ) sin(omega t - phi) + (4 F)/(3 k)
Max Extension is therefore:
xi_(max) =sqrt((16 F^2)/(9 k^2) + (6 m u^2)/(k) ) + (4 F)/(3 k)