‘Eugene Goostman’ fools 33% of interrogators into thinking it is human, in what is seen as a milestone in artificial intelligence.

A “super computer” has duped humans into thinking it was a 13-year-old boy to become the first machine to pass the Turing test, experts have said. Five machines were tested at the Royal Society in central London to see if they could fool people into thinking they were humans during text-based conversations.

The test was devised in 1950 by computer science pioneer and second world war codebreaker Alan Turing, who said that if a machine was indistinguishable from a human, then it was “thinking”.

No computer had ever previously passed the Turing test, which requires 30% of human interrogators to be duped during a series of five-minute keyboard conversations, organisers from the University of Reading said.

But “Eugene Goostman”, a computer programme developed to simulate a 13-year-old boy, managed to convince 33% of the judges that it was human, the university said.

Professor Kevin Warwick, from the University of Reading, said: “In the field of artificial intelligence, there is no more iconic and controversial milestone than the Turing test. It is fitting that such an important landmark has been reached at the Royal Society in London, the home of British science and the scene of many great advances in human understanding over the centuries. This milestone will go down in history as one of the most exciting.”

You will be assimilated

Convincing ⅓ of the judges during 5 minute interrogations is impressive, but I don’t think we can therefore claim that the program is truly intelligent. To properly pass the Turing test, a program has to be able to convince a majority of interlocutors in open-ended interactions. (For practical reasons, such tests usually have a time limit, but 5 minutes is a bit too short, IMHO, and I’m pretty sure that A. M. Turing would agree with me). And even then, passing the Turing test doesn’t prove that a program is really intelligent (whatever that means), although failing the test means that the program (probably) isn’t intelligent.

Turing devised his eponymous test in the very early days of computer science, when computers & their software were still only theoretical. He understood that it is very difficult to adequately define intelligence, consciousness etc, but he wanted to come up with an operational test that would at least give programs a chance of being considered to be intelligent, in some fashion.

Also bear in mind that at that time the philosophy of logical positivism was in vogue, as was its close cousin, the behaviorist school of psychology. According to these schools of thought, external appearance & behaviour is primary, questions about true inner motivation are impossible to answer and hence absurd, if not downright meaningless. Philosophy and psychology today generally consider that sort of attitude to be a bit extreme.

33% is a pass mark now?

But interesting anyway.

I’m frankly unimpressed, but then I’m not impressed by the whole concept of the “Turing Test”. The idea that a well-crafted fake AI is all that’s needed to demonstrate real AI strikes me as being simply nutty.

As I said in chat:

33% is a pretty low bar…

furious said:

As I said in chat:

33% is a pretty low bar…

For a human, sure… For AI it’s pretty good. Today 33% tomorrow – the world!

i think there is a problem with the concept of the turing test

turing has come to the conclusion that humans are the acme of thought

I don’t believe that the turing test proves anything conclusive about machines thinking – they are already thinking at a million miles quicker than we are

the turing test just means that a machines sounds like a human

when will blue fairy make me a real boy?

Bubblecar said:

I’m frankly unimpressed, but then I’m not impressed by the whole concept of the “Turing Test”. The idea that a well-crafted fake AI is all that’s needed to demonstrate real AI strikes me as being simply nutty.

Sure, it’s just a duck test , but IMHO it’s better than no test at all, even for those of us who think that behaviourism and logical positivism suck.

Do you have a superior test in mind?

Eugene Goostman is better than me…

// think that behaviourism … suck.

caricatured as pure; the principles are sound

OCDC said:

Eugene Goostman is better than me…

According to Wikipedia, he’s been around for a while:
Eugene Goostman

Eugene Goostman is an artificial intelligence chatterbot. First developed by a group of three programmers; the Russian-born Vladimir Veselov, Ukranian-born Eugene Demchenko, and Russian-born Sergey Ulasen in Saint Petersburg in 2001, Goostman is portrayed as a 13-year old Ukranian boy in an effort to make his personality and knowledge level believable to users.

Goostman has competed in a number of Turing test contests since its creation, with several second-place finishes in the Loebner Prize. In June 2012, at a competition marking what would have been the 100th birthday of its namesake, Goostman won what was promoted as the largest-ever Turing test contest, successfully convincing 29% of its judges that it was human. On 7 June 2014, at a contest marking the 60th anniversary of Turing’s death, Goostman became, according to Kevin Warwick, the first ever computer to pass the Turing test, by convincing 33% of judges that it was human.

Google brings up a couple of links to Princeton, but I guess the server’s rather busy today.

http://www.princetonai.com/bot
http://www.princetonai.com/bot/bot.jsp

PM 2Ring said:

Bubblecar said:

I’m frankly unimpressed, but then I’m not impressed by the whole concept of the “Turing Test”. The idea that a well-crafted fake AI is all that’s needed to demonstrate real AI strikes me as being simply nutty.

Sure, it’s just a duck test , but IMHO it’s better than no test at all, even for those of us who think that behaviourism and logical positivism suck.

Do you have a superior test in mind?

for a computer to come to a conclusion that we ourselves have found no solution or to point out something that we have never noticed

a true computer will see what we don’t see because it is operating at a level of logic rather than opinion?

And I don’t think that anyone who’s met SCIENCE would think ‘e’s human.

PM 2Ring said:

Do you have a superior test in mind?

wookiemeister said:

for a computer to come to a conclusion that we ourselves have found no solution or to point out something that we have never noticed

a true computer will see what we don’t see because it is operating at a level of logic rather than opinion?

Despite having opinions (some) people can be logical…

There have been mathematical / geometrical theorem proving programs for many years. They don’t have general intelligence and can’t hold a conversation, but they have found new stuff. IIRC, Douglas Hofstadter mentions in Gödel, Escher, Bach an early example of such a program which found a superior proof for one of the theorems in Euclid’s Elements.

Also see Automated theorem proving .

some arguments about AI and turing tests remind me a bit of creationists arguments about the evolution of the eye.

OCDC said:

And I don’t think that anyone who’s met SCIENCE would think ‘e’s human.

:)

On a more serious note, it is a hard problem to devise a test that recognises AI which won’t fail some (allegedly) intelligent humans. OTOH, you don’t have to go far on the Internet to find people whose intelligence is more than a little below par…

>Do you have a superior test in mind?

I’m happy to wait until they’re actually trying to design machines that think, rather than software programs that hope to convincingly mimic small talk. Given that everyone who’s entered the Turing Test has been out to deceive, I think we can write off all such efforts as “fake AI” without even bothering to run the “test”.

JudgeMental said:

some arguments about AI and turing tests remind me a bit of creationists arguments about the evolution of the eye.

I’m more than happy to accept that thinking machines are eminently possible. But they’ll come about as the result of attempts to design thinking machines, not attempts to pass the Turing Test.

Bubblecar said:

JudgeMental said:

some arguments about AI and turing tests remind me a bit of creationists arguments about the evolution of the eye.

I’m more than happy to accept that thinking machines are eminently possible. But they’ll come about as the result of attempts to design thinking machines, not attempts to pass the Turing Test.

Having searched my data banks for statements with similar wording, the associated response with the highest similarity score seems to be:

“I’m inclined to agree”.

Bubblecar said:

I’m frankly unimpressed, but then I’m not impressed by the whole concept of the “Turing Test”. The idea that a well-crafted fake AI is all that’s needed to demonstrate real AI strikes me as being simply nutty.

This.

Show Me Some Real NI

More to the point:

If and when we come up with genuine AI, it will probably be tuned for some specific purpose such as medical diagnosis.

It probably won’t have a bunch of worthless baggage designed for tricking people into thinking it is human.

¿how to genuine?

OCDC said:

Eugene Goostman is better than me…

Only 25% of correspondents thought you to be a real person, and that includes members of your immediate family.

dv said:

OCDC said:

Eugene Goostman is better than me…

Only 25% of correspondents thought you to be a real person, and that includes members of your immediate family.

Exactly.

Still, it is nice to see some great success from this joint Ukrainian-Russian project.

It would be interesting to know how they framed the test.

Goostman has been quasi-famous among AI types for a while. Any AI type, when they sat down and found they were in discourse with a 13 year old Ukrainian boy, would realise it was Goostman and tick “not human”.

Bubblecar said:

I’m frankly unimpressed, but then I’m not impressed by the whole concept of the “Turing Test”. The idea that a well-crafted fake AI is all that’s needed to demonstrate real AI strikes me as being simply nutty.

I’m frankly extremely impressed. I’m not at all sure that I could pass a Turing test, and as for some other members of this forum …

I’m absolutely sure that if I tried, I could fail a Turing test.

wookiemeister said:

i think there is a problem with the concept of the turing test

turing has come to the conclusion that humans are the acme of thought

I don’t think anyone’s claiming that the Turing test is perfect, or even that it’s an adequate test of true intelligence (whatever that is). It’s just a benchmark; an AI with a sufficiently high level of general intelligence ought to be able to pass it. OTOH, as Bubblecar notes, we’re not obliged to consider a program to be intelligent when it was designed with the sole object of passing the Turing test. Still, the Turing test has motivated programmers to improve programs’ ability to interact with humans using natural language, and I reckon that’s a worthwhile attribute even if the program isn’t truly intelligent.

wookiemeister said:

I don’t believe that the turing test proves anything conclusive about machines thinking – they are already thinking at a million miles quicker than we are

the turing test just means that a machines sounds like a human

Yes, machines can do arithmetic and elementary symbol manipulation much faster than humans, but I think that to call such activity thought is stretching the definition beyond breaking point.

Still, the Turing test has motivated programmers to improve programs’ ability to interact with humans using natural language, and I reckon that’s a worthwhile attribute even if the program isn’t truly intelligent.

this is what i was getting at with my previous comment. we have to start somewhere.

Natural language processing is an important technology that has made leaps and bounds lately..

They are having some problems with human like robots as well. No matter how lifelike they make them look they still look NQR and make people uneasy. Plus I suppose it is an expense, robots of the future may be like I, Robot the movie and not attempted to be too human looking.

AwesomeO said:

They are having some problems with human like robots as well. No matter how lifelike they make them look they still look NQR and make people uneasy.

Only to a point.

http://en.wikipedia.org/wiki/Uncanny_valley

Dropbear said:

Natural language processing is an important technology that has made leaps and bounds lately..

Yep.

It’s much harder than they thought it’d be back in the dawn of the computer era. Back in the 1950s, people were claiming that machine translation would be a solved problem within a decade (or so). But that’s because they thought it could be done with simple mindless symbol manipulation.

True, some translation tasks can be done that way, but it seems that to do it properly, a good translator (like a good conversationalist) needs to have some way of internally representing what the text is actually referring to. And getting a computer to internally represent such Real World knowledge adequately has proven to be a very hard problem.

pop

Speaking of translating stuff, this is pretty darn cool.

Got it on my phone, it works fairly well.

>Also bear in mind that at that time the philosophy of logical positivism was in vogue, as was its close cousin, the behaviorist school of psychology. According to these schools of thought, external appearance & behaviour is primary, questions about true inner motivation are impossible to answer and hence absurd, if not downright meaningless. Philosophy and psychology today generally consider that sort of attitude to be a bit extreme”

You either had a good teacher or were a good student to yourself, I think the latter.

Of the various mental states humans have (which are quite varied across any population example, some generalizations can be made however), it strikes me that humans adjust the focus of their mental tools, for different things, at different times, but the whole or even most of the workings probably aren’t ever fully at our (conscious) disposal, which isn’t a bad thing.

Suffering and mortality have been subjects of ‘interest’ with humans going back a long way, of particular interest to the conscious fleshy human. I include ‘avoiding’ suffering and death in that ‘interest’.

I’m going to stray to being a bit metaphysical here for a moment, which may sound like i’m heading for religion, which I wont be but the parallel is there.

What do humans do with knowledge of mortality, if it is really ‘knowledge’, given that it’s an oblivion once the CNS stops functioning. Apart from our contribution to culture, and having children (excuse the order written, it doesn’t suggest priorities), I suppose there’s our friends too, partners, but for many to expire without having blazed a trail is probably quite good enough, just to have been, had a look, and then entered the permanent absence is well-good enough.

But what do conscious humans do with the sobering reality of mortality. It can’t be ignored (well, it sort of can I suppose), but to do so may result in such an example being ‘unsafe company’, because such an example wouldn’t be reliable in important ways. If mortality (some ideas about) wasn’t sobering then doubtful individuals could be or would be so inclined to be considerate of others, and animals too I suppose.

So what do mental states, summoning mental tools and mortality have to do with each other, and how might they relate to detection or recognition of consciousness? I wont answer that, because it’s a work in progress for most conscious creatures, a highly personalized/ing works and part of an individuals identity. There is only one pm2 and there is only one transition, for example, and the qualities of and quantity ’1’ are for more than just simple math. If there’s a God he made ’1’, and if he (or it) didn’t then it probably wishes it did, and if there is no God then in that case it is the very modest ’1’.

Allow me the digression above, because it is of a humorous thing.

How do humans ride the line of bitter/sweet, happy/sad (without being bipolar or schizophrenic – that ‘normalish’ say), how do we ride the line of lightheartedness/seriousness, that sort of thing.

In the above I’d think are hints of where human consciousness resides, how it is gotten. In it also is possibly what a machine can’t do.

transition said:

But what do conscious humans do with the sobering reality of mortality. It can’t be ignored (well, it sort of can I suppose), but to do so may result in such an example being ‘unsafe company’, because such an example wouldn’t be reliable in important ways. If mortality (some ideas about) wasn’t sobering then doubtful individuals could be or would be so inclined to be considerate of others, non-human animals too I suppose.

[…]

In the above I’d think are hints of where human consciousness resides, how it is gotten. In it also is possibly what a machine can’t do.

I agree that awareness of our mortality & how we deal with it are important parts of the human condition and human culture. An argument can be made that religion exists purely to deal with our mortality; its efficacy is debatable…

An AI that can pass all sorts of intelligence tests but which cannot understand human mortality would be alien, and I suspect that we’d always treat such an entity as a “mere” machine, not a person.

OTOH, although a machine AI may be able to exist for a vast period of time, that doesn’t mean it’s immortal. It may be even more bitter-sweet than the human condition to understand that you can exist for billions or even trillions of years but that you are still ultimately doomed.

This is interesting news, of a type I suspect we will hear more and more of in the coming decades.

I tend to think of the Turing Test as being preliminary, rather than definitive. If a computer passes the Turing Test then it has potential to lead to digital cognition, but is not in itself proof of said cognition.

I am a closet Technological Singularitist. I believe that the technological singularity as espoused by folk like Vernor Vinge and Eliezer Yudkowski probably will happen, and it will be a transformative in the same way the Agrarian Revolution was.

Exciting times, and I hope I live long enough to see it all come to fruition.

Go science!

Dropbear said:

Natural language processing is an important technology that has made leaps and bounds lately..

I totally disagree. Natural language processing is so awful that I’ve been beginning to get almost paranoid about it, as if some intelligent entity is deliberately stopping all progress in natural language processing. Consider the following:

1) Spell checkers for the web, Google and Microsoft are total crap.
2) Optical character recognition software is total crap.
3) Natural language translation software is total crap.
4) Speech recognition software is total crap.
5) Chatbots haven’t, until now, significantly surpassed Eliza which was written in 1973.

None of these have advanced significantly in the past ten years. It wouldn’t take a genius to make any of the above much better.

So we can’t scan natural language or input it by voice. If we could input it the software can’t fix errors in it. If by some miracle we had natural language recognition right then we still couldn’t translate it!

mollwollfumble said:

Dropbear said:

Natural language processing is an important technology that has made leaps and bounds lately..

I totally disagree. Natural language processing is so awful that I’ve been beginning to get almost paranoid about it, as if some intelligent entity is deliberately stopping all progress in natural language processing. Consider the following:

1) Spell checkers for the web, Google and Microsoft are total crap.
2) Optical character recognition software is total crap.
3) Natural language translation software is total crap.
4) Speech recognition software is total crap.
5) Chatbots haven’t, until now, significantly surpassed Eliza which was written in 1973.

None of these have advanced significantly in the past ten years. It wouldn’t take a genius to make any of the above much better.

So we can’t scan natural language or input it by voice. If we could input it the software can’t fix errors in it. If by some miracle we had natural language recognition right then we still couldn’t translate it!

More evidence in support of mollwoll:

The face recognition on my Dell laptop is hopeless. It fails to recognise me almost every time I start from a different location
Google image recognition software is hopeless. It will find an exact copy of an image you are searching for well enough, but its “similar” images are almost always of very little similarity at all.
Google searches for anything other than absolutely straightforward text are hopeless.
Context sensitive help is often less helpful than it was 30 years ago.

mollwollfumble said:

1) Spell checkers for the web, Google and Microsoft are total crap.
2) Optical character recognition software is total crap.
3) Natural language translation software is total crap.
4) Speech recognition software is total crap.
5) Chatbots haven’t, until now, significantly surpassed Eliza which was written in 1973.

1) Spell checking is a pretty basic function. Saying they are crap is nonsense. Sure, they are not very good at reading the context of the sentence, but that is not the purpose of a spell checker.

2) This is true, but there is also the intentional sabotage which is so prevalent these days which is holding this back. Things like Captcha, which are designed to improve the data base ( as well as prove the person logging on is not a bot), which get results where every second submission is “nigga” or something completely meaningless, doesn’t help. I can’t help but feel if the average user wasn’t subverting this tech it would be much more advanced now.

3) true.

4) I don’t agree with this at all. Apple’s Siri, for one, is amazing in this regard.

5) Chatbots are lagging, but that is essentially what this thread is about and I suspect we are simply still on the shallow end of an exponential curve.

> Spell checking is a pretty basic function. Saying they are crap is nonsense. Sure, they are not very good at reading the context of the sentence, but that is not the purpose of a spell checker.

The one I’m using now doesn’t recognise the words “english” or “Siri” or “chatbot” or “recognise” or “speciality” for starters. I’d call that a pretty basic fault. Any idiot could write a better word list. If I was writing one I’d have multiple word lists with separate lists for basic English, technical English, archaic English, lower-case acronyms, foreign words, SMS shorthand, manufactured products, rare personal names, etc. I’d set it up so that every document is checked against basic English plus one of the specialty lists, because it would be fairly rare for a single document to use two or more specialty lists. Basic English would also include non-oxford-english-dictionary words commonly appearing on the web. I’d also silently auto-check English vs American spelling to assure consistency throughout a document.

> Apple’s Siri, for one, is amazing in this regard.

Thanks, didn’t know this. I’ll check it out.

mollwollfumble said:

Bubblecar said:

I’m frankly unimpressed

I’m frankly extremely impressed.

I’m in the middle :-) well, about the TT per se, I agree fooling 1/3 is not awesome…

We know from experience that in time-constrained tests, TT challengers will design programs to simulate particular aspects of human speech – slightly obsessive/autistic etc – and I guess this is what BC refers to as ‘fake’ AI.

But if a program could talk seemingly intelligently and creatively in a truly open-ended test, so on just about any subject, I’m not sure how you would distinguish ‘fake’ AI from ‘real’ AI.

IMO Siri is more impressive than a chatbot convincing people it’s a 13 y.o

Dropbear said:

IMO Siri is more impressive than a chatbot convincing people it’s a 13 y.o

But what is your opinion on a genuinely open-ended TT, as opposed to a test where the main design features are there to provide plausible explanations for the inability to meet the expected performance from an open-ended TT?

MartinB said:

Dropbear said:

IMO Siri is more impressive than a chatbot convincing people it’s a 13 y.o

But what is your opinion on a genuinely open-ended TT, as opposed to a test where the main design features are there to provide plausible explanations for the inability to meet the expected performance from an open-ended TT?

Simple to emulate a 13yo boy.
Totally inactive except during masturbation. Communicate in grunts. Never wash.

Tamb said:

MartinB said:

Dropbear said:

IMO Siri is more impressive than a chatbot convincing people it’s a 13 y.o

But what is your opinion on a genuinely open-ended TT, as opposed to a test where the main design features are there to provide plausible explanations for the inability to meet the expected performance from an open-ended TT?

Simple to emulate a 13yo boy.
Totally inactive except during masturbation. Communicate in grunts. Never wash.

Plus ball scratching and nose picking

a computer did not pass turing test

From original press release:

“The development team includes Eugene’s creator Vladimir Veselov, who was born in Russia, and Ukrainian born Eugene Demchenko”

That figures. No-one born in the west working on AI has that amount of intelligence.

“Eugene managed to convince 33% of the human judges. 30 judges took part. Each judge was involved in five parallel tests – so 10 conversations. This event involved the most simultaneous comparison tests than ever before, was independently verified and, crucially, the conversations were unrestricted.”

That’s good enough for me, the result would be meaningless if there were only 3 judges and conversation topics were heavily restricted.