Wednesday, 16 April 2014

What is Turing test success?

An unnecessary focus, that does not ‘sex-up’ Turing’s imitation game, rather detracts from Alan Turing’s brilliant ideas on building machines to think, is the notion that machines in practical Turing tests require 30% deception rate for success. This is a misinterpretation of one of Turing’s predictions:

" about fifty year’s time it will be possible to programme computers, with a storage capacity of about 109, to make them play the imitation game so well that an average interrogator will not have more than a 70 per cent. chance of making the right identification after five minutes of questioning” (Computing Machinery and Intelligence, 1950, p.442).

If that statement is to be interpreted as 30% interrogator deception rate for machine success then it has already been surpassed, in the very first Loebner Prize for Artificial Intelligence in 1991:

 “….the highest-ranked computer program was misclassified as a human by five of the ten judges” (by Robert Epstein in his chapter ‘The Quest for the Thinking Computer’ in R. Epstein, G. Roberts & G. Beber Parsing the Turing Test, Springer, 2008, p. 3).

Turing’s success rate required a machine to provide “satisfactory and sustained” answers, not an “easy contrivance” (A.M. Turing, in Computing Machinery and Intelligence, 1950, p. 447), to any questions put to it by a jury panel of interrogators, and that:
 “a considerable proportion of a jury, who should not be experts about machines, must be taken in by the pretence” (A.M. Turing, in ‘Can Automatic Calculating Machines be Said to think’, p. 495 in B. J. Copeland’s The Essential Turing, OUP, 2004). 

To me, a considerable proportion should be more than the rate of chance - this is not ‘moving the goal post’, because half the number of ten judges were deceived in one practical Turing test event (see above, 1991). To do Turing’s ideas justice and to inspire and motivate machine developers, not less than 70% of a jury panel of interrogators should feel the conversational skills of their hidden interlocutor are human-like in any robust Turing test experiment.

The challenge is to attain funding for a full, large–scale Turing test experiment with hundreds of “average interrogators” – that is, all ages of novices and expertise levels - taking part in thousands of tests run over weeks and weeks.

Until that happens practical Turing tests serve many purposes, from measuring machine performance against a human’s in any field, to encouraging/inspiring children to take up computing and robotics. It is also a useful experiment helping to raise awareness and prevention of cybercrime. It is estimated that the cost of cybercrime to the individual is over £3bn* - the full amount is probably not known because people may not always be willing to come forward and report to the police admiting they have had their identity stolen or bank accounts compromised, for fear of appearing foolish at being deceived in cyberspace. There is also the task of making more children prepared to distinguish adults from peers in cyberspace to ensure they do not succumb to grooming on the Internet^.

Reading University (Visiting Professor Kevin Warwick and Dr. Huma Shah) and Coventry University (Deputy Vice Chancellor-Research, Professor Kevin Warwick) are collaborating to host another public Turing test experiment, following on from the June 2012 experiment on the 100th anniversary of Alan Turing’s birth at Bletchley Park.

Details for this year's event:

Practical Turing tests (based on Huma Shah’s PhD: 'Deception-detection and machine intelligence in practical Turing tests' )

Venue location: The Royal Society, 6-9 Carlton House Terrace, London SW1 5AG:

Public day: Saturday 7 June 2014

Timing: 10am - 6.30pm

Follow on Twitter here: @Turing2014

Event animated trailer on YouTube is here:

Previous experiment, Turing100in2012 information here:

^ Child Exploitation and Online Prevention Centre (CEOP):

© Huma Shah, 16 April 2014

No comments:

Post a Comment