File server, multiple hdds and cooling advice requested.

Post by **CrackerJackMack** » 29 Oct 2008, 17:00

Hey, I think I lost something, anyone seen the thread topic around? can't seem to find it in this page.

Sorry Mesh.

In my experience, the drives live longer when in an external enclosure. Because when you need to maintenance the server, you aren't spinning the drives up and down. The only problem I've had is with the current brand of enclosures I have....but that is another matter.

The external case for the drives provide excellent cooling, and keeps the overall server temperature down. Below is a printout from my Intel 3065 w/4G of RAM and 3.41 TB of usable storage. Power is another concern that all the attempts before at a cheap storage array have failed me. In typical situations (like when not doing anything) you are fine, but when you start copying files, starting up stuff, performing blah, the voltages are going to fluctuate. Usually it isn't horrible, but it depends on the efficiency of your PSU and at what % load it becomes efficient. Then you have to find one that isn't just 60A +5V because you really need +12V for drives. It is much easier to just have two PSU's @ 500W and a lower efficiency because they are easier to come by and there are mods like this that would allow you to place both PSU's into a single case.

Storage:

Code: Select all

littlenemo ~ # pvs

     PV         VG   Fmt  Attr PSize PFree

     /dev/md2   data lvm2 a-   1.36T   1.26T

     /dev/md3   data lvm2 a-   2.05T 585.88G

Temperature of the server

Code: Select all

w83793-i2c-0-2f

   Adapter: SMBus I801 adapter at 1100

   VCoreA:    +1.05 V  (min =  +0.92 V, max =  +1.38 V)

   VCoreB:    +1.25 V  (min =  +0.00 V, max =  +2.05 V)

   Vtt:       +1.19 V  (min =  +0.99 V, max =  +1.33 V)

   in3:       +0.53 V  (min =  +0.40 V, max =  +0.67 V)

   in4:       +1.82 V  (min =  +1.62 V, max =  +1.98 V)

   +3.3V:     +3.36 V  (min =  +2.96 V, max =  +3.63 V)

   +12V:     +12.00 V  (min = +10.75 V, max = +13.25 V)

   +5V:       +5.12 V  (min =  +4.64 V, max =  +5.65 V)

   5VSB:      +5.12 V  (min =  +4.64 V, max =  +5.65 V)

   VBAT:      +3.22 V  (min =  +2.96 V, max =  +3.63 V)

   fan1:        0 RPM  (min =  712 RPM)                   ALARM

   fan2:        0 RPM  (min =  712 RPM)                   ALARM

   fan3:     1746 RPM  (min =  712 RPM)

   fan4:        0 RPM  (min =  712 RPM)                   ALARM

   fan5:     1755 RPM  (min =  712 RPM)

   fan6:        0 RPM  (min =  712 RPM)                   ALARM

   fan7:     1702 RPM  (min =  712 RPM)

   fan8:        0 RPM  (min =  712 RPM)                   ALARM

   fan9:        0 RPM  (min =  712 RPM)                   ALARM

   fan10:       0 RPM  (min =  712 RPM)                   ALARM

   CPU1 Temp: +42.0 C  (high = +83.0 C, hyst = +78.0 C)

   CPU2 Temp:-104.5 C  (high = +83.0 C, hyst = +78.0 C)

   temp3:    -128.0 C  (high = +83.0 C, hyst = +78.0 C)

   temp4:    -128.0 C  (high = +83.0 C, hyst = +78.0 C)

   temp5:       +38 C  (high =   +50 C, hyst =   +45 C)

   temp6:       +37 C  (high =   +50 C, hyst =   +45 C)

   

   coretemp-isa-0000

   Adapter: ISA adapter

   Core 0:      +36 C  (high =  +100 C)

   

   coretemp-isa-0001

   Adapter: ISA adapter

   Core 1:      +35 C  (high =  +100 C)

Mesh · Post by **Mesh** » 29 Oct 2008, 19:14

I've seen those 'use 2 psu as one' deals before, even those 5 1/4 bay gpu psus. I've never trusted either because my electrical instincts tell me one way or another 1 of the 2 will at some point try to 'charge' the other and cause major damage.

External enclosures are nice, when they work. None I've had so far have, performance is (censored), heavily dependant on the chipset, especially usb versions. And esata is about as stable as a mammoth sanding on 1 leg on the edge of the grand canyon.

Now, I'm past the buying stage, I'm currently in the built stage.

Case will have 1 120mm per 4 drives, the mobo is an atom 330 board with ~50w max.

I'm trying to see if a 350W phantom would work, gonna temp it with my 620w corsair while I wait for the power adapters show up and see startup and max loads on 5v and 12v rails.

Both psus are about highest efficiency there is atm so I'm pretty sure the 350w would suffice given that there's no high end gpu to drain everything.

Total is about 200w shared on the 5+12v lines which fit the total watt and total amps on the phantoms rails. Mobo+boot drive shouldn't be enough to overload it. No optical which is a plus in power concerns.

Oh and external cases with sata multiplier lanes and etc. were one of the option I looked at before. I gave up however given that the total cost would have been more than I could spend to build a 10tb+ rig using 1.5tb drives. And using fast spinning loud fans to boot.

Post by **CrackerJackMack** » 29 Oct 2008, 20:35

Ah, I should have mentioned that I'm using an infiniband/esata setup. So it's line speed with direct onboard sata. I get 60-99MB/s on a raid-resync.

Cynagen · Post by **Cynagen** » 29 Oct 2008, 23:32

I hate to blow y'all outta the water, but... YOUR DOING IT WRONG.

I have an 8 drive enclosure (I'll borrow the neighbor's camera and take photos) with a 500w supply (these are actually dirt cheap, $40), loaded to the brim.

RAID JBOD configs on both controllers:
250+250+400+500
250+250+400+500

Each resulting in a 1.27TB formatted usable space, on FW800 controllers.

If you're going to build... do it right. Buy an enclosure power supply, there's no cabling for motherboards on it, just drive molex connections. Then Go to IOI Technology, and get some of their FW800-to-Whatever controllers, and a $40 FW800 controller card for your PC. Done. I'm using a pair of IOI's FW800-to-Dual-ATA133 (FW-IDE02) adapters chained together, then linked to my computer, with 8 removable drive cages. The only issue is I'm using Visduh, and it shoots external transfer speeds in the head. So yeah... I might be looking to sell this should the event come that none of the Vista patches (minus SP1 which breaks my computer even more) fix my FW transfer speeds, then I'll be looking to sell the enclosure and controller card.

Mesh · Post by **Mesh** » 13 Nov 2008, 06:49

Figured I'd bump my thread to ask for help.
I built the server using:
Centurion 590 case + 2 4in3 modules
8x WD 500gb hard drives
1x WD 80gb boot drive
Intel Atom 330 motherboard
Highpoint rocketraid 454 pci controller.
Vista x64

80gb boot is off the motherboard controller, 8 500gb on the 454.

I've thoroughly tested drives with WD diagnostic with full test (~2h per drive). Same with ram with memtest. Both from a boot cd I had with a bunch of tools.

The problem I'm having is that as soon as I try to use the drive: make a partition etc. Windows is unable to shutdown, it just shuts off the monitor and sticks there, fans running and everything.

If I force a power off, it then blocks off at the boot logo anim, doesn't freeze, just never ends (even after 8h+ left it to boot and went to work).

Now, as long as I install and don't format/partition the drives, it can shutdown just fine. I'm at a loss why making a partition, even on just 2 of the 8 drives would halt it's shutdown and boot capacity though.

I've already reinstalled the OS 3 times, always block at the same point, I've updated bios and drivers to no avail.

I don't think it's a defective hardware part since problems start only after partitionning but my return window is closing shortly, if they let me return anything at all, seeing as hardware seems fine.

In case the pci controller did something odd when making the partitions I even tried putting 2 on the onboard controller (boot moved to SATA with an adapter I have), making the partitions there (raid 0 with 2x500gb dynamic disks). Shutdown, plug em into the 454 instead. Boot fine, but not shutdown.

Any insight someone might have on what the problem is would be appreciated.

Post by **CrackerJackMack** » 13 Nov 2008, 07:25

Interesting

http://www.newegg.com/Product/Product.aspx?Item=N82E16816115016 gave it 3/5
what is the powersupply in the computer? It could be a voltage problem, those drives pull a lot of amps

Mesh · Post by **Mesh** » 13 Nov 2008, 09:34

I've tried both a corsair HX620 and once I knew it took about 220w peak on boot, I put it on the 350w antec phantom I wanted to use for the build.

I've calculated the max amps on 5v and 12v and as far as I can tell either of these power supplys is more than enough.

I don't think it's a power issue since i should have problems at any time during the use of the rig, not just at shutdown and especially not only if I make partitions on the drives.

It shutdowns fine if everything is plugged in but no partitions are made to drives on the 454.

Post by **CrackerJackMack** » 13 Nov 2008, 14:44

Good point, I must not have been thinking well...

Mesh · Post by **Mesh** » 15 Nov 2008, 07:10

Don't worry about it, I'm just just at the end of my wits with this, I gave up for a few days since I simply don't see any solution to this. I've tried all I could think of short of trying the card in another rig, but I don't want to screw up another rig I'll need to reinstall either.

Post by **CrackerJackMack** » 15 Nov 2008, 19:04

Just curious, but does it exhibit the same behavior with linux or BSD? If not, it's not the controller or the drives, but some type of systems management or driver issue.

At least this would send you (or us) investigating in the right direction. A live CD would suffice.

Mesh · Post by **Mesh** » 15 Nov 2008, 21:30

I would, except I haven't been able to boot off the cd unless no OS is installed. It boots to hdd no matter that I set cd first in boot order and put 'usb boot first' active either.

Had to frankenstein a drive on the floor next to it to reinstall it last time, guess I'll have to do that again.

Post by **CrackerJackMack** » 15 Nov 2008, 22:19

You have another machine you could PXE from ? Do your motherboard support PXE booting ?

Mesh · Post by **Mesh** » 15 Nov 2008, 22:44

PXE booting? network boot? yes it does, keeps feeding that part since it keeps skipping over the cdrom.

I've attempted right now to boot to windows cd via said cd on the floor method, now even the cd halts on logo ...

I'm about to find an old pci card to see if the issue shows up with something other than the controller. I'm beginning to think these intel atom boards are for show and not use.

[email protected] · Post by **[email protected]** » 15 Nov 2008, 23:57

That would not surprise me, I have been thinking of an atom based file server, let me know if it ever works out.

Mesh · Post by **Mesh** » 16 Nov 2008, 00:39

So far seems theres something that just won't work between this rocketraid controller and the D945GCLF2 motherboard.

I bought 2 D945GCLF2, one for the file server and one for an almost silent pc.

I've just tried the controller in the 2nd D945GCLF2, it's been fine since I installed it, just putting in the controller, without drives. It just halts, I'm not even able to get into bios, I just get a blinking cursor in top left.

Remove the controller and it boots back just fine.

Mesh · Post by **Mesh** » 16 Nov 2008, 02:45

I've just tried a couple old pci cards I had.

results are:

freeze on animated boot logo:
rocketraid 454
dlink 530CT NIC

boots but crash on shutdown:
maxi gamer phoenix (voodoo banshee)

no problem:
xg sound card

So, seems there's an issue with the pci slot itself, since the same happens in the other motherboard I'm doubting a defective slot, the odds of both being defective aren't that great, I think intel has better quality testing than that.

Now please, anyone have a clue what's going on?

Post by **CrackerJackMack** » 16 Nov 2008, 08:49

Don't take the questions personally as even the most basic questions can sometimes be the root cause. I know I've done some extensive troubleshooting when it turned out I wasn't seating an AGP card fully...just an example.

I put the reason why I'm asking in (...)
[list]
During these tests, are all the 500G drives powered? (PSU)
Were you using the same rated PSU in both motherboards? (PSU)
Did the processor used change between tests? (test grouping)
Were the card known good before testing ? (Quality control)
You always made sure to touch the case before trying to insert the cards? (Quality control)
[/list:u]

(risking sounding like a broken record...)
If you look at the cards you testing with, they are mostly interrupt intensive. But the sound card is not, leading to only a crash at shutdown. If the drives are powered, but not plugged in, I have to say power. If the drives aren't plugged in, I'd say BIOS compatibility issue. Since you can't boot, I next say it's a heat issue, though I would doubt it. If you can get booted up let's test the North and south bridge temperatures while we are at it.

Mesh · Post by **Mesh** » 16 Nov 2008, 20:23

During these tests, are all the 500G drives powered? (PSU)
As far as I can tell yes. Problem happens without them plugged into controller so I don't think they have a say in the crash.

Were you using the same rated PSU in both motherboards? (PSU)
Tried both psu in both motherboards

Did the processor used change between tests? (test grouping)
Can't change it, it's soldered to the motherboard.

Were the card known good before testing ? (Quality control)
Bought at same time so no clue.

You always made sure to touch the case before trying to insert the cards? (Quality control)
Always do.

If you look at the cards you testing with, they are mostly interrupt intensive. But the sound card is not, leading to only a crash at shutdown.
The sound card is fine completely, it's the vga card that boots but crashes on shutdown.

If the drives are powered, but not plugged in, I have to say power.
Thought so too, till problem showed up while only boot drive was powered.

If the drives aren't plugged in, I'd say BIOS compatibility issue.
Wouldn't doubt it, seems the mobo and bios lack serious fixing, even if it's like their 3rd or 4th version already.

Since you can't boot, I next say it's a heat issue, though I would doubt it. If you can get booted up let's test the North and south bridge temperatures while we are at it.
Each 4 drives has a 120mm fan for it, they seem quite cool.

North / South bridges, heatsink doesn't look that warm, I'm wondering how much the chip transfers though, heatink seems held there by the clip without much else.

edit - I've just tried reinstalling vista, 32bit this time, to see if maybe it's a 64bit os glitch. For whatever reason on the last step setup actually bluescreened ...

And before that, it wouldn't boot off the dvd from usb drive, said bootmgr not found (I checked the file, it's there, can even copy it)

I put back the floor drive and tried it, went fine till it bluescreened on last step, no clue why since it restarted immediatly.

The last time I had such random glitches it turned out to be bad ram, I guess if the reinstall doesn't work again I'm gonna swap the 1 stick with the other one.

[email protected] · Post by **[email protected]** » 16 Nov 2008, 21:05

And make sure it is seated really well, improperly seated ram has given me no end of trouble.

Mesh · Post by **Mesh** » 16 Nov 2008, 21:26

I've removed and reseated it, doesn't seem to be poorly fit.

Yet I'm unabled to install vista 32bit. It expands files and everything, reboots once, gets to completing install. and bluescreens. Too short for me to see much except some icg or something dll failure. On restart it gives no clue as to what fails and just says to restart install.

Memtest+ didn't show any issues, know any reliable memory testers?

Widescreen Gaming Forum