This paper considers the task of video surveillance for pedestrian crossings. We analyzed peculiarities of the task, proposed the structure of the video surveillance system, and described the developed tracking subsystem. We evaluated accuracy, performance and robustness of the system. The precision of tracking is about 92 %, recall — 75...95% and its speed is close to real time mode.