[GTA5/GTAV] End-to-end learning for autonomous driving

In the previous post (visit 2017/07/28 - [TORCS] - End-to-end learning for autonomous driving), I showed you a racing car that controls its steering wheel angle according to CNN output. (The input to the CNN is the resized front view camera image and the output is the best steering wheel angle that makes the car keep its lane.)

In this post, I will show you a car that runs on the road in GTA5!!! The CNN for the steering wheel angle control is the same as that used in the previous post. The input to the CNN is also the resized front view camera image obtained from GTA5. The output is the best steering wheel angle that makes the car keep its lane. Since the CNN was trained from the driving images of TORCS, the steering wheel angle output is not suitable for the car in GTA5. So I remapped the output steering angle using the following function and the remapped steering angle was finally used for driving in GTA5. 

<steering wheel angle re-mapping function>

The following video shows the result of the autonomous driving in GTA5. As I mentioned above, the car in the video controls its steering wheel angle according to the CNN output. The speed of the car is controlled by a simple algorithm, which makes the car keep the constant speed.

 

 

The communication between my python codes and GTA5 was done via DeepGTAV. (For more details, visit https://github.com/ai-tor/DeepGTAV.) Every 0.1 second, my python codes receive a front view camera image from GTA5. The image is then resized, cropped, and normalized in order to satisfy CNN input spec. The steering wheel angle value calculated from the CNN is then re-mapped and sent back to GTA5 program via DeepGTAV.

You can see in the video that the CNN sometimes produces unstable steering wheel angle command. This is because the CNN was trained by using the driving images from TORCS. The roads in TORCS have very clear surfaces and lines. On the contrary, the surfaces and lines of the roads in GTA5 are not clear enough so that the CNN misunderstood the grooves on the road as the lines.

 

I would like to note that I was able to access and use GTA5 for my experiments owing to the efforts of Dr. An and student Lee.