Click Here
home features news forums classifieds faqs links search
6071 members 
Amiga Q&A /  Free for All /  Emulation /  Gaming / (Latest Posts)
Login

Nickname

Password

Lost Password?

Don't have an account yet?
Register now!

Support Amigaworld.net
Your support is needed and is appreciated as Amigaworld.net is primarily dependent upon the support of its users.
Donate

Menu
Main sections
» Home
» Features
» News
» Forums
» Classifieds
» Links
» Downloads
Extras
» OS4 Zone
» IRC Network
» AmigaWorld Radio
» Newsfeed
» Top Members
» Amiga Dealers
Information
» About Us
» FAQs
» Advertise
» Polls
» Terms of Service
» Search

IRC Channel
Server: irc.amigaworld.net
Ports: 1024,5555, 6665-6669
SSL port: 6697
Channel: #Amigaworld
Channel Policy and Guidelines

Who's Online
15 crawler(s) on-line.
 144 guest(s) on-line.
 0 member(s) on-line.



You are an anonymous user.
Register Now!
 kolla:  6 mins ago
 matthey:  7 mins ago
 pixie:  18 mins ago
 amigakit:  30 mins ago
 yoodoo2:  38 mins ago
 vox:  51 mins ago
 Rob:  53 mins ago
 kiFla:  1 hr 3 mins ago
 mordock:  1 hr 30 mins ago
 K-L:  2 hrs 18 mins ago

News   News : NVidia found to be cheating on 3dMark2003 results
   posted by _Steve_ on 25-May-2003 14:57:21 (4282 reads)
Benchmark firm Futuremark has uncovered extensive cheating by NVidia in its 3DMark03 suite.

After an initial report at ExtremeTech, Futuremark revisited the tests and discovered eight instances of cheating, which improved the performance of NVidia's Detonator FX and WHQL drivers by as much as 24.1 per cent.


But NVidia achieved this not through brilliant optimization, but by alternative means which omit graphic details: so the output, while at times similar, does not resemble what it should.

As Futuremark explains:

"The cheating described here is totally different from optimization. Optimizing the driver code to increase efficiency is a technique often used to enhance game performance and carries greater legitimacy, since the rendered image is exactly what the developer intended."

NVidia deployed a variety of cheats: introducing its own pixel shaders and vertex shaders specifically for the benchmark. The shaders improved performance at the expense of image quality.

A PDF Audit file released from Futuremark can be read here. / (750KB).

Source: The Register.
    

STORYID: 521
Related Links
· More about News
· News by _Steve_


Printer Friendly Page  Send this Story to a Friend

Goto page ( 1 | 2 )

PosterThread
Bodie 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 25-May-2003 15:37:14
#1 ]
Super Member
Joined: 9-Jan-2003
Posts: 1439
From: Azjol-Nerub

If Nvidia continue as they are, they will more than likely go the same way as 3dfx.

 Status: Offline
Profile     Report this post  
herewegoagain 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 25-May-2003 16:33:57
#2 ]
Elite Member
Joined: 8-Jan-2003
Posts: 3270
From: Charlotte, NC



So does this mean that all of those sites that posted the new benchmarks showing nVidia passing ATI's 9800 will have to redo their scores?

 Status: Offline
Profile     Report this post  
ajs 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 25-May-2003 16:41:50
#3 ]
Super Member
Joined: 8-Mar-2003
Posts: 1459
From: Surrey

Well they reckon ATI were doing it as well, but not as much


_________________
Everything is proceeding as I have foreseen

Some people are like slinkies. Not really good for anything but they bring a smile to your face when pushed down the stairs.

 Status: Offline
Profile     Report this post  
_Steve_ 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 0:22:41
#4 ]
Team Member
Joined: 17-Oct-2002
Posts: 6814
From: UK

ATI were found to have done it on one test only, and it only affected it by about 8%. NVidia on the other hand did it in more than 8 tests, some of which caused up to a 24% difference.


_________________
Test sig (new)

 Status: Offline
Profile     Report this post  
Anonymous 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 3:28:14
# ]



Lets all boycott NVidia and ATI cards.. go for SiS or something. THAT OUGHTA SHOW THEM !

 
     Report this post  
Hammer 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 4:45:54
#6 ]
Elite Member
Joined: 9-Mar-2003
Posts: 5906
From: Australia

@Bodie, Herewegoagain

Results for UT2003 and Doom3(beta) doesn?t dive with 3DMark2003.

IF Intel can optimize applications for Pentium4?s SSE2; why not a GPU vendor (i.e. treating GPUs like normal CPUs)?

The true gamer?s benchmark is from the actual games themselves e.g. Unreal Tournament 2003, QuakeII, Quake3, Doom3(beta), Aqua Marks 3 (from the next Aqua series) and 'etc'.

The ideal GPU benchmark would be collection of 3D game engines(from real games) within a single benchmark utility.


_________________
Amiga 1200 (rev 1D1, KS 3.2, PiStorm32/RPi CM4/Emu68)
Amiga 500 (rev 6A, ECS, KS 3.2, PiStorm/RPi 4B/Emu68)
Ryzen 9 7900X, DDR5-6000 64 GB RAM, GeForce RTX 4080 16 GB

 Status: Offline
Profile     Report this post  
Hammer 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 5:07:29
#7 ]
Elite Member
Joined: 9-Mar-2003
Posts: 5906
From: Australia

@Hooligan

Note that SIS also compromises the said benchmarks (e.g. reducing texture map's quality upon detection of 3DMarks). But no one cares (or takes an interest) if the cheapo GPU compromises the benchmarks.


_________________
Amiga 1200 (rev 1D1, KS 3.2, PiStorm32/RPi CM4/Emu68)
Amiga 500 (rev 6A, ECS, KS 3.2, PiStorm/RPi 4B/Emu68)
Ryzen 9 7900X, DDR5-6000 64 GB RAM, GeForce RTX 4080 16 GB

 Status: Offline
Profile     Report this post  
Hammer 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 5:29:36
#8 ]
Elite Member
Joined: 9-Mar-2003
Posts: 5906
From: Australia

Hope is not lost i.e. there?s still S3 (with their Direct 9X Delta-chrome offerings). S3 barely survived their legal battle(legal battle of endurance) with nVidia.


_________________
Amiga 1200 (rev 1D1, KS 3.2, PiStorm32/RPi CM4/Emu68)
Amiga 500 (rev 6A, ECS, KS 3.2, PiStorm/RPi 4B/Emu68)
Ryzen 9 7900X, DDR5-6000 64 GB RAM, GeForce RTX 4080 16 GB

 Status: Offline
Profile     Report this post  
Anonymous 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 9:40:30
# ]



Quote:
The ideal GPU benchmark would be collection of 3D game engines(from real games) within a single benchmark utility.


Agreed. 3DMarkXX is too synthetic, it only tests one way to do things that might work well on one card and not so well on others. Also, drivers get optimized for good results with these tests. It doesn't say anything about how well a card will perform with, e.g. Unreal 2.

On a related note, I found it peculiar that nVidia's "Dawn" demo now runs on the 9800pro with an OpenGL wrapper, and that the ATI version seems to be slightly faster than the GeforceFX

 
     Report this post  
VidarL 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 15:04:50
#10 ]
Member
Joined: 16-May-2003
Posts: 75
From: Unknown

Quote:
Agreed. 3DMarkXX is too synthetic, it only tests one way to do things that might work well on one card and not so well on others. Also, drivers get optimized for good results with these tests. It doesn't say anything about how well a card will perform with, e.g. Unreal 2.


I agree that 3dMark only tests one way to do things, but to think that drivers wouldn't be "optimized" for a benchmark that uses several engines is naive IMO.

As long as the demo goes on a predefined path, cheating like Nvidia have done by inserting clipplanes will be easy to do. One solution is to make say 10 random walkthroughs in a level and average the results of those. The problem is a much higher error margin a much longer benchmark run.

While 3DMark doesn't say how fast a card is in a specific engine, I believe it is a valuable indicator of overall performance.

Vidar

 Status: Offline
Profile     Report this post  
_Steve_ 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 15:28:05
#11 ]
Team Member
Joined: 17-Oct-2002
Posts: 6814
From: UK

Quote:
Poster: Hammer Date: 2003/5/26 4:45:54

Results for UT2003 and Doom3(beta) doesn?t dive with 3DMark2003.

IF Intel can optimize applications for Pentium4?s SSE2; why not a GPU vendor (i.e. treating GPUs like normal CPUs)?


As stated in the PDF, optimisation of the drivers is quite a different thing to forcing the display to be shown in a lower quality than you specify in order to achieve a performance boost. What NVidia did was not an optimisation of their driver.


_________________
Test sig (new)

 Status: Offline
Profile     Report this post  
_Steve_ 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 15:31:58
#12 ]
Team Member
Joined: 17-Oct-2002
Posts: 6814
From: UK

Quote:
Poster: VidarL Date: 2003/5/26 15:04:50

While 3DMark doesn't say how fast a card is in a specific engine, I believe it is a valuable indicator of overall performance.


Thats the point though isn't it. The 3DMark series have always been used as an unbiased way of ascertaining the performance levels for gfx cards. It may not be perfect, but it does at least give a reasonable indication of how well the different cards can perform. If manufacturers are going to start detecting the benchmark program and artificially increase their cards performance, there will be a lot of unhappy customers who buy a card expecting it to do well, and find its really not that much better than the one they had before.

In the end, those manufacturers just give themselves a bad reputation, and only hurt their own sales.


_________________
Test sig (new)

 Status: Offline
Profile     Report this post  
Anonymous 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 17:58:49
# ]



Quote:
While 3DMark doesn't say how fast a card is in a specific engine, I believe it is a valuable indicator of overall performance


On one hand, yes, on the other hand, there is obvious room for "misleading" stuff, as the recent cheating shows.

I think a variety of engines would be a better choice, and not only FPS type of engines. I mean, compare "Dungeon Siege", "Freelancer" and "Unreal 2". These have very different requirements, since one is a landscape engine that is guaranteed to fill the screen at least once, one is a space shooter that is heavier on the geonemtry side, and one is an FPS that needs quite some fill rate AND triangle count. If they manage to get good results in those three games, then likely the customer using their cards/drivers would have a benefit from that. As it is now, they can only boast a few more points in 3DMark

 
     Report this post  
VidarL 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 26-May-2003 19:34:45
#14 ]
Member
Joined: 16-May-2003
Posts: 75
From: Unknown

Quote:
If they manage to get good results in those three games, then likely the customer using their cards/drivers would have a benefit from that.


What makes such a benchmark different from 3DMark with regards to cheating? And if it is possible to cheat, the benchmark has little value.

Personally, I believe the best way to benchmark is to use FRAPS, even though it's error margin is a bit higher than normal timedemos. Doing so, should make it virtually impossible to cheat.

The downside is that it is impossible to compare scores between the different sites.

Vidar

 Status: Offline
Profile     Report this post  
Hammer 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 27-May-2003 2:15:02
#15 ]
Elite Member
Joined: 9-Mar-2003
Posts: 5906
From: Australia

Quote:
On a related note, I found it peculiar that nVidia's "Dawn" demo now runs on the 9800pro with an OpenGL wrapper,

It doesn?t matter, since the real world games are the ones that count.

Refer to
http://www.rojakpot.com/default.aspx?location=3&var1=32&var2=0

http://www.hothardware.com/hh_files/S&V/r9800256mb_gffx5900upd(2).shtml

Quote:

and that the ATI version seems to be slightly faster than the GeforceFX

Which Geforce FX (i.e. FX 5900 Ultra or FX5800 Ultra)?


_________________
Amiga 1200 (rev 1D1, KS 3.2, PiStorm32/RPi CM4/Emu68)
Amiga 500 (rev 6A, ECS, KS 3.2, PiStorm/RPi 4B/Emu68)
Ryzen 9 7900X, DDR5-6000 64 GB RAM, GeForce RTX 4080 16 GB

 Status: Offline
Profile     Report this post  
Hammer 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 27-May-2003 2:24:05
#16 ]
Elite Member
Joined: 9-Mar-2003
Posts: 5906
From: Australia

Quote:
While 3DMark doesn't say how fast a card is in a specific engine, I believe it is a valuable indicator of overall performance.

FutureMark's 3D engine is not representative of future Doom3/UT2003 based game engines. In terms of numbers, there are more game creators** using of 3D game engine either from ID ?s and Epic?s 3D engines than FutureMark's 3D engine.

**Titles released based on either Epic's or ID's 3D game engines.

FutureMark is not quite SPECInt/SPECFPU for GPUs.
Optimization is OK (within limits) under SPEC?s regime.


_________________
Amiga 1200 (rev 1D1, KS 3.2, PiStorm32/RPi CM4/Emu68)
Amiga 500 (rev 6A, ECS, KS 3.2, PiStorm/RPi 4B/Emu68)
Ryzen 9 7900X, DDR5-6000 64 GB RAM, GeForce RTX 4080 16 GB

 Status: Offline
Profile     Report this post  
Hammer 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 27-May-2003 2:47:01
#17 ]
Elite Member
Joined: 9-Mar-2003
Posts: 5906
From: Australia

Quote:
What NVidia did was not an optimisation of their driver.

Note that ATI(e.g. quake/quack) was also as guilty as Nvidia btw?The difference was the gain from profiling regime.


_________________
Amiga 1200 (rev 1D1, KS 3.2, PiStorm32/RPi CM4/Emu68)
Amiga 500 (rev 6A, ECS, KS 3.2, PiStorm/RPi 4B/Emu68)
Ryzen 9 7900X, DDR5-6000 64 GB RAM, GeForce RTX 4080 16 GB

 Status: Offline
Profile     Report this post  
_Steve_ 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 27-May-2003 17:19:39
#18 ]
Team Member
Joined: 17-Oct-2002
Posts: 6814
From: UK

Quote:
Poster: Hammer Date: 2003/5/27 2:47:01

Quote:
--------------------------------------------------------------------------------

What NVidia did was not an optimisation of their driver.
--------------------------------------------------------------------------------

Note that ATI(e.g. quake/quack) was also as guilty as Nvidia btw?The difference was the gain from profiling regime.


I never said ATI weren't innocent of it either. However, trusting the cards on games is no better when you consider that all they have to do is apply the same detection techniques to any games you care to run on your card. The result is the same - no matter how detailed you want the display, the card will always render it at a lower quality to give the impression its doing a better job. In the end, its not so much how powerful the card is, but how well the game engines are coded. After all, a crap engine doesn't necessarily perform any better on a higher specced card.


_________________
Test sig (new)

 Status: Offline
Profile     Report this post  
Calken 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 27-May-2003 19:20:34
#19 ]
Member
Joined: 15-Mar-2003
Posts: 41
From: Unknown

Just a link to ATI's story on The Register

 Status: Offline
Profile     Report this post  
Hammer 
Re: NVidia found to be cheating on 3dMark2003 results
Posted on 28-May-2003 9:31:24
#20 ]
Elite Member
Joined: 9-Mar-2003
Posts: 5906
From: Australia

Quote:

I never said ATI weren't innocent of it either. However, trusting the cards on games is no better when you consider that all they have to do is apply the same detection techniques to any games you care to run on your card.

Lowering the texture/picture quality would be unacceptable, but utilization of optimised code would be acceptable (i.e. resulting value must be the same as with the ?older? code).

Quote:

The result is the same - no matter how detailed you want the display, the card will always render it at a lower quality to give the impression its doing a better job.

Of course. I?m not that naive.


_________________
Amiga 1200 (rev 1D1, KS 3.2, PiStorm32/RPi CM4/Emu68)
Amiga 500 (rev 6A, ECS, KS 3.2, PiStorm/RPi 4B/Emu68)
Ryzen 9 7900X, DDR5-6000 64 GB RAM, GeForce RTX 4080 16 GB

 Status: Offline
Profile     Report this post  

Goto page ( 1 | 2 )

[ home ][ about us ][ privacy ] [ forums ][ classifieds ] [ links ][ news archive ] [ link to us ][ user account ]
Copyright (C) 2000 - 2019 Amigaworld.net.
Amigaworld.net was originally founded by David Doyle