Sharing and analyzing image data from ubiquitous urban cameras must enable us to understand and predict various contexts of the city. Meanwhile, since such image data always contains privacy data such as people and cars, we cannot easily share and analyze the data through the Internet for the viewpoint of privacy protection. As a result, most of the urban image data are only kept/shared within the camera owners or even discarded to reduce risks of privacy data leakage. To solve the privacy problem and accelerate sharing of urban image data, we propose GANonymizer that automatically detects and removes objects related to privacy from the urban images. GANonymizer combines two neural networks: 1) a network which detects objects related to privacy such as persons and cars in an input image using object detection network, and 2) a network that removes the detected objects naturally as though they do not exist originally. In addition, we propose two padding layers for removing the detected objects more naturally. Through our experiment of applying GANonymizer to urban video images, we confirmed that GANonymizer partially achieved natural removal of objects related to privacy.