Before an autonomous vehicle (AV) is allowed to ply on public roads, the vehicle has to be subjected to rigorous testing to show that it is safe for all road-users under various real-life situations. Testing involves assessment of the AV’s driving behaviour during numerous test trips. The assessment would involve reviewing of logged data and videos throughout the test trips. An automated system is presented in this paper that can help to identify and classify various scenarios the AV had gone through during the test trips. This would in effect increase the productivity of testing, assessment and incident investigation processes. The system was built and tested with an AV at the Singapore one-north AV trial testbed and showed that it can identify, classify and assess the AV’s behaviours accurately and effectively. This paper describes the design of the system and the tests that the system was put through.