Soft tissue displacements during minimally invasive surgical procedures may cause target motion and subsequent misplacement of the surgical tool. A technique is presented to predict target displacements using a combination of ultrasound elastography and finite element (FE) modeling. A cubic gelatin/agar phantom with stiff targets was manufactured to obtain pre- and post-loading ultrasound radio frequency (RF) data from a linear array transducer. The RF data were used to compute displacement and strain images, from which the distribution of elasticity was reconstructed using an inverse FE-based approach. The FE model was subsequently used to predict target displacements upon application of different boundary and loading conditions to the phantom. The influence of geometry was investigated by application of the technique to a breast-shaped phantom. The distribution of elasticity in the phantoms as determined from the strain distribution agreed well with results from mechanical testing. Upon application of different boundary and loading conditions to the cubic phantom, the FE model-predicted target motion were consistent with ultrasound measurements. The FE-based approach could also accurately predict the displacement of the target upon compression and indentation of the breast-shaped phantom. This study provides experimental evidence that organ geometry and boundary conditions surrounding the organ are important factors influencing target motion. In future work, the technique presented in this paper could be used for preoperative planning of minimally invasive surgical interventions.