Gigabit Ethernet Throughput

Post by **Paradigm Shifter** » 20 Apr 2008, 18:22

Anyone care to hazard a guess why my newly set up GbE network gives me shit-poor transfer rates? I'm getting far lower transfer rates on the supposedly gigabit network, than I was when using 10/100.

All network cards are GbE, the switch is GbE, the NAS drive the same and the cabling is Cat5e.

Out of some perverse amusement, I set the network card to 100mbps max, and I had transfer rates back in the 85% network utilisation again (albeit 100Mb network) rather than <2.5% on a Gb network.

The last thing I expected on a Gb setup was that copying 16GB of files would take 74 minutes. :evil:

Mesh · Post by **Mesh** » 21 Apr 2008, 00:07

I've been having the same headache for hmm, quite some months now.
The best I've gotten was a vista to vista transfer which seemed to work completely fine maxing out at the hdd's capacity (50-60mbytes/s).

So far the only things I've managed to improve are by messing with the TCP receive window and other misc. network settings that used to be for optimizing broadband internet connections.

See here for more details:
http://www.enterprisenetworkingplanet.com/nethub/article.php/3485486

Unlike the article says, I haven't attained 900Mb/s, right now I'm at 10mbytes/s one way and 20mbytes/s the other.

Might try things like swaping cables and switching around things, like if you have a net router or anything else under 1gbit into the segment, remove em and see if what changes, if anything. Could be something as simple as a cable making the gbit traffic going through a 100mbit router.

In any case, good luck.

Cynagen · Post by **Cynagen** » 21 Apr 2008, 01:18

...the cabling is Cat5e...

... The last thing I expected on a Gb setup was that copying 16GB of files would take 74 minutes. :evil:

Switch to Cat6 to reduce the possibility of interference/crosstalk on the cables, for one thing.

Second, there's a lot of software and hardware that comes into play in a gigabit network... you might want to check the switch. Is it a switch or a hub? There is a huge difference. If it's a switch, all gigE connections to the hub should communicate at gigE with one another, if it's a hub (usually cheaper), they relay everything on every port to every active port minus the one received on, so a 100mbit link to say, the router, will cause you undue delay in your transfer as the gigE data is waited upon to be forwarded through the 100mbit to the router. Also, if you're using a router to control/split your connection inside your home, you CAN alter the MTU on your ethernet controllers past 1500, just push them all together, not one at a time, otherwise if you have one trying to talk at 5000 and the other still at 1500, it'll cause some interesting effects.

Frag Maniac · Post by **Frag Maniac** » 21 Apr 2008, 01:37

Check that your NIC drivers are updated. Unless they're older models, NIC card drivers can update fairly often. Then check that the firmware is updated on your modem. I just flashed mine and got a fairly good speed increase. Other than that always check with your ISP if you think it's nothing on your end hardware or software wise. I've had a few times where I called my ISP and they said they were doing repairs in my area.

Post by **Paradigm Shifter** » 21 Apr 2008, 19:43

It's not ISP related, so calling them is pointless. :)

OK. Borrowed some Cat6 from the oh-so-helpful guys and girls at computing services at work, and now have that hooking everything up.

NIC drivers are up-to-date too.

It's a switch, it's not a hub. If I get two things to talk to each other, the others have no input on the 'conversation' (I see no activity on the other devices)...

Tried setting IP addresses manually and removing any 10/100 devices from the switch (the printer and the internet router/firewall).

Nowt makes much difference. The Cat6 upped the PC-PC transfer rates about 1%, but had no impact on the NAS speeds. There are no firmware updates available for the NAS.

...

Again, setting one device to 10/100 made network utilisation go back to the 'normal' regions for 100Mb networks. It just doesn't seem to like GbE for some reason.

...

I'm annoyed by this, but it's not the end of the world when all is said and done. The piss-poor transfer rates to/from the NAS drive is the biggest irritant, believe it or not. I can deal with 30MB/s transfers between computers rather than 10MB/s I was getting on the 10/100 network, but 3.7MB/s to/from the NAS is just... painful.

...

...

Anyone know of an ADSL modem/router that has wireless (G) and 8 wired Ethernet ports in it? I don't care if it's GbE, if it gives me less of a headache, it's worth the performance "deficit".

Mesh · Post by **Mesh** » 21 Apr 2008, 22:57

Paradigm Shifter,

You mentionned a nas, did you compare pc to nas vs pc to another pc see if its the same? I do know my synology nas is worse so far then pc to pc, over the same network equipment.

Also, you could try something like pcattcp to remove the hard drives from the variables, it uses ram instead to test the network.

see:http://www.pcausa.com/Utilities/pcattcp.htm

Cynagen · Post by **Cynagen** » 22 Apr 2008, 10:35

...
I'm annoyed by this, but it's not the end of the world when all is said and done. The piss-poor transfer rates to/from the NAS drive is the biggest irritant, believe it or not. I can deal with 30MB/s transfers between computers rather than 10MB/s I was getting on the 10/100 network, but 3.7MB/s to/from the NAS is just... painful.

...

...

Anyone know of an ADSL modem/router that has wireless (G) and 8 wired Ethernet ports in it? I don't care if it's GbE, if it gives me less of a headache, it's worth the performance "deficit".

First, the NAS, is it a computer running FreeNAS? Or is it actual NAS hardware/software?

Second, the best you'll get is 30MB/s transfers between computers, that's the harddrive max throughput on most SATA drives man, unless you switch EVERYTHING to Ultra320 SCSI, you're not going to EVER be seeing higher than around 35MB/sec (Burst)... I get 30-35 transfer disc to disc in my computer. Be glad at last now, you're not limited by the 100mbit cap on the lines, and your computers can swap at full SATA2 (300mbit) speeds.

Code: Select all

I'm a moron? I dunno...

Third, Do not get gateways (Modem&Router combos), that's a completely retarded thing to do. Most times they're shittier than even the shittiest networking components at Frys.com (and trust me, Fry's sells some pretty shitty networking equipment.)

Post by **Paradigm Shifter** » 22 Apr 2008, 19:50

Mesh, yes, PC to PC is about as fast as it's gonna get, I think. Approaching HDD max transfer rates, anyway. I am interested in why PC to PC the transfer speeds vary so much (up and down like a bloody yoyo).

The NAS is a Western Digital MyBook World Edition 1TB (single drive).

...

OK, what Cyna said makes sense. Wasn't thinking completely, obviously. Doesn't explain the NAS speed, but makes me slightly less GRRR. ;)

And this is why I love this forum. :D

Thanks guys. :D

Cynagen · Post by **Cynagen** » 22 Apr 2008, 21:17

Mesh, yes, PC to PC is about as fast as it's gonna get, I think. Approaching HDD max transfer rates, anyway. I am interested in why PC to PC the transfer speeds vary so much (up and down like a bloody yoyo).

The NAS is a Western Digital MyBook World Edition 1TB (single drive).

...

OK, what Cyna said makes sense. Wasn't thinking completely, obviously. Doesn't explain the NAS speed, but makes me slightly less GRRR. ;)

And this is why I love this forum. :D

Thanks guys. :D

First, yes, PC to PC is as good as it will get.

Second, they vary badly because of fragmentation, HDDs work best when the data is written in a line, not strewn across the HDD, so defrag more regularly.

Third, that NAS only has a 100mbit NIC as far as I know, you'd be better off connecting it via USB2 (480mbit [57.22MB/sec]) to one of your GigE equipped computers for best transfer rates. Oh, and 1TB drives, have horrible Random Access times, and it's likely you're writing a bunch of little files to that thing, so, yeah, you're likely to be experiencing the bad access times, as well as fragmentation layered on top. I never use an external drive for small files, that's what USB thumb drives are for since they use flash memory and are far better for Random Access reading. Use the NAS for large files, I use mine for HD movies, and my USB 250 I use for local HD storage as well.

Fourth, thank you for at least listening to me. Some people would have ignored me, even though I was explaining what the hell was going on.

Post by **Paradigm Shifter** » 22 Apr 2008, 23:02

On your third point, Cyna... that NAS was advertised as a gigabit ethernet connection, and it's ethernet only... it reports 1000Mbps in the speeds and various setup pages, so I have no reason to believe it isn't. If the random access times are that poor... ouch. Oh well. To be honest, once it's full of stuff, I'll only be reading files off it occasionally. Obviously I'm just destined to get annoyed at how long it takes to write to. ;)

Cynagen · Post by **Cynagen** » 22 Apr 2008, 23:51

On your third point, Cyna... that NAS was advertised as a gigabit ethernet connection, and it's ethernet only... it reports 1000Mbps in the speeds and various setup pages, so I have no reason to believe it isn't. If the random access times are that poor... ouch. Oh well. To be honest, once it's full of stuff, I'll only be reading files off it occasionally. Obviously I'm just destined to get annoyed at how long it takes to write to. ;)

Hit up newegg, check out all the TB and 500gb drives there, a 500gb's average read/write latencies are 8-9ms/10-13ms respectively. That's just the base response times, now if a file is fragmented, that increases the time as it has to seek across the drive to get the data. A TB drive's average response times are ... wow, they don't even want to list them, they must be bad then... the best I saw was a seek time of 8.9ms, which is how long on average it takes the read arm to position the heads to read the data you want, now if the data was in a long string, that's one seek, and just adjusting with the line to read the spiral of data. Fragmenting makes that 9ms here, read some, 9ms, read some, 9ms, read some, you see how that adds up, especially on a REALLY fragmented drive. Not sure how that NAS handles fragmentation, but if it just lets the drive do it's thing to reduce the firmware OS size, then it's going to be severely fragmented to hell...

Mesh · Post by **Mesh** » 23 Apr 2008, 01:43

WD's page lists that nas as 10/100/1000. Though listing it as gigabit and performing as such is quite a different things, I've got 5+ gigabit nics, all from different chipsets/manufacturers, so far, few run more then 200mbits/s. There seems to be a general wall whenever we leave 10/100 standards and go into the gigabit hack-standard.

Cynagen, my practical data counters your numbers, I've done 55mbytes/s avg from a single IDE / ATA133 drive ...

And unless you've got a near full drive with countless times of erasing/rewriting large amounts to it, you can forget about fragmentation.

Fragmentation happens only when no space allows for a file to be written in the same spot. On near/half empty drives, it's about pointless.

Case in point, this drive I've been using for years has about a dozen files with even a few fragments, mostly small windows temporary files.

Cynagen · Post by **Cynagen** » 23 Apr 2008, 10:23

WD's page lists that nas as 10/100/1000. Though listing it as gigabit and performing as such is quite a different things, I've got 5+ gigabit nics, all from different chipsets/manufacturers, so far, few run more then 200mbits/s. There seems to be a general wall whenever we leave 10/100 standards and go into the gigabit hack-standard.

Cynagen, my practical data counters your numbers, I've done 55mbytes/s avg from a single IDE / ATA133 drive ...

And unless you've got a near full drive with countless times of erasing/rewriting large amounts to it, you can forget about fragmentation.

Fragmentation happens only when no space allows for a file to be written in the same spot. On near/half empty drives, it's about pointless.

Case in point, this drive I've been using for years has about a dozen files with even a few fragments, mostly small windows temporary files.

Was that 55mb/sec sustained for more than a second or two? And i'm not talking about the window's screen updates, i'm talking actual seconds. Vista likes to post an average, and the first ones are usually high because of the burst of data written/read right off the bat, then it calms down to the average.

Post by **Paradigm Shifter** » 23 Apr 2008, 20:42

Well, I'll do a bit more experimenting when I've got some time... maybe try setting up a RAM disk in a testing OS to remove the HDD speeds from the equation.

Thanks for continued input. :)

Mesh · Post by **Mesh** » 24 Apr 2008, 02:32

Paradigm Shifter:I'm still looking for solutions myself, maybe going back and forth we'll both get our gear working 'as advertised' good luck.

Cynagen: Burst part was over 70, sustained transfer of a 2.3gbytes file at 50-60mbytes/s and since windows typically has the lower number way of displaying data sizes, it's likely a bit over that.

Post by **Paradigm Shifter** » 24 Apr 2008, 19:07

I wonder whether the increased speed in Vista is due to it's better caching system? Or rather, it's propensity for chewing up RAM by caching as much as possible. But for transferring large amounts of data, it would make SFA difference.

Tyinsar · Post by **Tyinsar** » 24 Apr 2008, 21:30

I thought hard drive speeds were in MegaBytes/s.

Code: Select all

              Transfer       Transfer

   IDE              Rate           Rate     Pins

   Drive     PIO   MBytes   DMA   MBytes     in

   Type      Mode  per sec  Mode  per sec   Cable

   ATA       0      3.3     0      4.2      40

   ATA       1      5.2                     40

   ATA       2      8.3

   ATA-2, 3  3     11.1     1     13.3      40

   ATA-2, 3  4     16.6     2     16.6      40

   ATA-4 (ATA-33)           2     33.3      40

   ATA-5                    0     16.6      40

   ATA-5                    1     25.0      40

   ATA-5 (ATA-33)           2     33.3      40

   ATA-5                    3     44.4      80

   ATA-5 (ATA-66)           4     66.6      80

   ATA-6 (ATA-100)          5    100.0      80

   ATA-7 (ATA-133)          5    133.0      80

   Serial ATA (SATA)        5    150.0       4

   Serial ATA II (SATA II)  5    300.0       4

(Source)
As noted above: these are the theoretical maximums of the interface (not the drives).

This is echoed at Wikipedia:

A typical desktop HDD, might store between 120 and 300 GB of data (based on US market data), rotate at 7,200 revolutions per minute (RPM) and have a media transfer rate of 1 Gbit/s or higher. (1 GB = 109 B; 1 Gbit/s = 109 bit/s)
...
The fastest â€œenterpriseâ€

Mesh · Post by **Mesh** » 25 Apr 2008, 03:15

My theory why vista2vista works as it should is because they (finally) redid the (censored) networking systems that have been unstable since at least w2000. It also has more auto-explore/auto-configure settings that are otherwise fixed/set in 2000/xp like the tcp receive window, mtu, etc.

Could be their driver signature required just kicked network maker's rear into making better drivers too, who knows.

Still, it should function the same regardless the os involved, especially under the same hardware.

And Tyinsar, yeah you got it right. Just like usb's theoretical max of 480mbit is 60mbytes.

Cynagen · Post by **Cynagen** » 25 Apr 2008, 11:41

I thought hard drive speeds were in MegaBytes/s.

Code: Select all

              Transfer       Transfer

   IDE              Rate           Rate     Pins

   Drive     PIO   MBytes   DMA   MBytes     in

   Type      Mode  per sec  Mode  per sec   Cable

   ATA       0      3.3     0      4.2      40

   ATA       1      5.2                     40

   ATA       2      8.3

   ATA-2, 3  3     11.1     1     13.3      40

   ATA-2, 3  4     16.6     2     16.6      40

   ATA-4 (ATA-33)           2     33.3      40

   ATA-5                    0     16.6      40

   ATA-5                    1     25.0      40

   ATA-5 (ATA-33)           2     33.3      40

   ATA-5                    3     44.4      80

   ATA-5 (ATA-66)           4     66.6      80

   ATA-6 (ATA-100)          5    100.0      80

   ATA-7 (ATA-133)          5    133.0      80

   Serial ATA (SATA)        5    150.0       4

   Serial ATA II (SATA II)  5    300.0       4

This would mean that SATA2 would be communicating @ 2.4gigabit/s... SATA @ 1.2gbps, ATA-133 @ 1.064gbps, ATA-100 @ 800mbps, etc...

Those numbers just, don't seem realistic. Why would they spend the time and money making these ports as fast as this when there's never going to be a harddrive (short of DDRRAM/SSD drives) that will ever be able to saturate those lines.

Edit: My 400th post, yey?

Tyinsar · Post by **Tyinsar** » 25 Apr 2008, 14:43

This would mean that SATA2 would be communicating @ 2.4gigabit/s... SATA @ 1.2gbps, ATA-133 @ 1.064gbps, ATA-100 @ 800mbps, etc...

As you noted earlier those are just the theoretical max speeds of the interface. Even at that it hasn't been that long since consumer level drives could saturate even the ATA-66 standard (at least from what I've read).

To me it's kind of like graphics cards switching to PCI-e: at the time graphics cards really only used about half of the maximum bandwidth AGP gave them - we didn't need it yet when the switch came.

To me the switches (AGP->PCIe & ATA->SATA) seem to be partly about future potential and partly about marketing (specifically getting us to buy new stuff).

Widescreen Gaming Forum