Abstract: A visual surveillance system is presented based on the integration of motion detection and visual tracking in static and dynamic scene image sequence. Four parameters model is established for global motion, and the model parameters estimated by block matching. Then moving objects regions were detected by Horn-Schunck algorithm with global motion vectors modified. The center of mass, width and height of moving objects were tracked by Kalman filter. Experimental results showed that this method is effective for moving objects detection and tracking in static and dynamic scenes.