Server design for busy site

fnord · 7th May 2007, 08:43 AM

I run a busy community site and due to this my current server can no longer take it. I need to get a new one. I want to get one that can sustain in future growth, but lack the expertise to know the optimal solution.
I'll soon have the funds available to buy a extremely good box.

It's a heavily modified vbulletin forum with lots of queries, file hosting, multimedia(video, music) running, chat rooms, wiki and various functionality. Currently running on apache / mysql / php 5.2.1

Here's the requirements:
The box needs to cope with over a 1.000.000 unique visitors and a terrabyte of bandwidth a month. 10.000 visitors in 3 hours.
Due to file hosting I need a good deal of HD space. 500 GB storage + 500 as mirror sounds good.
The server will be hosted in a data center with a fiberglass network, so a goon network card would be good.
I want the server to create DB backups several times a day and full back ups at least weekly. I'd want to download backups trough ftp frequently. This without affecting visitor experience.

I assume that there will be many things I have left out that you need to know, so please ask.

I'd like to ask for advise on what hardware or server I should buy. I have a budget of €5.000.00

peaz · 7th May 2007, 08:55 AM

CPU speed won't really be an issue becuase the amount of processing isn't really that high. What you will need is fast I/O

I'd suggest getting more hard disk and RAIDing them with RAID 5.

perhaps. 4x 250GB in RAID 5 configuration? that way you'll have 3x250 usable space.

RAM will also be important to handle the large number of concurrent sessions. but again, i doubt that the demand of a php app can be that high. but it never hurts to have more RAM. at least 2GB? best if 4GB.

One thing to check is also your hosting package, make sure there is no network bandwidth limit on your network card out to the host-side routers. standard packages (even dedicated hosting and co-lo) usually have a 10mbps cap. for file sharing sites, this is usually not sufficient especially if you have a lot of concurrent downloads.

One more thing, perhaps you can help us understand what the current server can't cope with? too high CPU utilization? insufficient memory?

fnord · 7th May 2007, 09:21 AM

I just updated my post above.

I am currently hosting the site on a old Intel Celeron CPU 2.00GHz. I'll ask my hosting company for more details about what the problem is. Main problem is that the server gets to many requests, gets slower and slower leading to a crash. Whether this is a CPU or RAM problem I don't know. I guess a combination of both.

Please define "I/O".

peaz · 7th May 2007, 01:11 PM

I/O = Input Output. I this case, it;s your hard disk and network requirements.

to be honest, it can really be anything, it can simply be just MySQL not configured properly

try asking your host what's the problem. it could also just be them limiting the network bandwidth to the router as well.

Adrian Wong · 12th May 2007, 05:53 AM

Yeah, they have been known to do that - limit the network bandwidth AND/OR limiting the number of connections.

fnord · 12th May 2007, 07:31 AM

100% of both cpu and RAM is used the majority of the time and my server is kept up by several scripts which kill processes that threaten my server from crashing.

My host advises these servers:

Server A:
- 2u server housing
- 2x quad-core intel xeon 1.6 Ghz (replaceable)
- 8GB DDR2 ECC memory (upgradeable to 64GB)
- 2x 146 GB SCSI 15.000rpm
- SCSI RAID controller
- redundant power supply
- 2x 1Gbit network
- 3 year on site 24x7x4 hardware replacement (or 5 years + 1000 Euro)

5100 euro.
-----

Server B:
- 2u server housing
- 2x dual-core intel xeon 2.0 Ghz (replaceable by quadcore at later time)
- 8GB DDR2 ECC memory(upgradeable to 64GB)
- 2x 146 GB SCSI 15.000rpm
- SCSI RAID controller
- redundant power supply
- 2x 1Gbit network
- 3 years on site 24x7x4 hardware replacement(or 5 years + 1000 euro)

4650 euro
-----

Server C:
- 2u server housing
- 2x dual-core AMD opteron 2.0 Ghz (replaceable by quadcore at later time)
- 8GB DDR2 ECC memory(upgradeable tot 64GB)
- 2x 146 GB SCSI 15.000rpm
- SCSI RAID controller
- redundant power supply
- 2x 1Gbit network
- 3 years on site 24x7x4 hardware replacement(5 years: price not known yet)

4270 euro

All prices include 3 years on site hardware replacement. (Within 4 hours) Without this the price goes down with €800.

What do you think about this?

peaz · 12th May 2007, 10:02 AM

Hmmm. What processes are those that threatens to crash the server? constant 100% cpu utilization? Wow. that's odd. even when techarp's server gets slashdotted, we don't see anything near 50%.

fnord · 12th May 2007, 09:03 PM

I don't know the exact processes, but it mostly happens when several things are happening at the same time(cron, backup, many users in chat, many users up/downloading files, various attacks together with heavy traffic)
But overall my site stays up while it has 600 - 1000 users online most of the time.

peaz · 13th May 2007, 11:51 AM

Hmmm all in all, could be high disk activity from the database updating on and on and on. I guess you can try upgrading the server. Even server C is overkill already. haha

seriously.

fnord · 13th May 2007, 09:43 PM

Yes, I am aware of that. But there are several factors that makes me decide for a top notch sever. I do not only want to cope with current server user, but also future use. I manage my site in order to maintain steep increase of use and traffic.

I have a 300% increase in visitors per year and a steep increase of bandwidth use.
Current server limitations have thus far limited site expansion. New expansions will surely intensify server use in many ways.

7th May 2007, 08:43 AM	#1 (permalink)
fnord Newbie Join Date: 22 Apr 2007 Posts: 12 Reputation: 0 Rep Power: 0	Server design for busy site I run a busy community site and due to this my current server can no longer take it. I need to get a new one. I want to get one that can sustain in future growth, but lack the expertise to know the optimal solution. I'll soon have the funds available to buy a extremely good box. It's a heavily modified vbulletin forum with lots of queries, file hosting, multimedia(video, music) running, chat rooms, wiki and various functionality. Currently running on apache / mysql / php 5.2.1 Here's the requirements: The box needs to cope with over a 1.000.000 unique visitors and a terrabyte of bandwidth a month. 10.000 visitors in 3 hours. Due to file hosting I need a good deal of HD space. 500 GB storage + 500 as mirror sounds good. The server will be hosted in a data center with a fiberglass network, so a goon network card would be good. I want the server to create DB backups several times a day and full back ups at least weekly. I'd want to download backups trough ftp frequently. This without affecting visitor experience. I assume that there will be many things I have left out that you need to know, so please ask. I'd like to ask for advise on what hardware or server I should buy. I have a budget of €5.000.00 Last edited by fnord : 7th May 2007 at 09:17 AM. Reason: forgot backup functionality

7th May 2007, 08:55 AM	#2 (permalink)
peaz ARP Webmaster Join Date: 13 Oct 2002 Location: http://atpeaz.com/ Posts: 8,595 Reputation: 1673 Rep Power: 32	CPU speed won't really be an issue becuase the amount of processing isn't really that high. What you will need is fast I/O I'd suggest getting more hard disk and RAIDing them with RAID 5. perhaps. 4x 250GB in RAID 5 configuration? that way you'll have 3x250 usable space. RAM will also be important to handle the large number of concurrent sessions. but again, i doubt that the demand of a php app can be that high. but it never hurts to have more RAM. at least 2GB? best if 4GB. One thing to check is also your hosting package, make sure there is no network bandwidth limit on your network card out to the host-side routers. standard packages (even dedicated hosting and co-lo) usually have a 10mbps cap. for file sharing sites, this is usually not sufficient especially if you have a lot of concurrent downloads. One more thing, perhaps you can help us understand what the current server can't cope with? too high CPU utilization? insufficient memory? __________________ Ken Ng Tech ARP Follow me at Twitter Blogs @ http://www.atpeaz.com/

7th May 2007, 01:11 PM	#4 (permalink)
peaz ARP Webmaster Join Date: 13 Oct 2002 Location: http://atpeaz.com/ Posts: 8,595 Reputation: 1673 Rep Power: 32	I/O = Input Output. I this case, it;s your hard disk and network requirements. to be honest, it can really be anything, it can simply be just MySQL not configured properly try asking your host what's the problem. it could also just be them limiting the network bandwidth to the router as well. __________________ Ken Ng Tech ARP Follow me at Twitter Blogs @ http://www.atpeaz.com/

12th May 2007, 05:53 AM	#5 (permalink)
Adrian Wong Da Boss Join Date: 10 Oct 2002 Location: In front of my ASUS F8V notebook! Posts: 33,118 Reputation: 3730 Rep Power: 78	Yeah, they have been known to do that - limit the network bandwidth AND/OR limiting the number of connections. __________________ Dr. Adrian Wong Tech ARP \| Blog @ Tech ARP \| The Free Trade Zone DYKT : The only offshore account I have is at the sand bank? We need PROGRAMMERS and TECHNICAL WRITERS! Contact us if you are a hot shot programmer or technical writer! My items for sale : 50x SD Card \| Memory Stick PRO \| Cyclone Energy Saver \| Seiko SS watch \| Tiger/Carlsberg beer jugs \| Travel Speakers \| Motorola V600 \| Nokia N90 SOLD! \| New Lowepro Mini Trekker AW Other items for sale @ the FTZ : Zalman CNPS9500 LED @ $20 \| Zalman CNPS7700 Cu @ $20 \| Zalman CNPS7000 Cu @ $20 \| Swarovski bracelet watches \| Dell 17" LCD \| Hi-Fi speakers \| English DIVX movies \| HP LaserJet toners! \| Office chairs

12th May 2007, 10:02 AM	#7 (permalink)
peaz ARP Webmaster Join Date: 13 Oct 2002 Location: http://atpeaz.com/ Posts: 8,595 Reputation: 1673 Rep Power: 32	Hmmm. What processes are those that threatens to crash the server? constant 100% cpu utilization? Wow. that's odd. even when techarp's server gets slashdotted, we don't see anything near 50%. __________________ Ken Ng Tech ARP Follow me at Twitter Blogs @ http://www.atpeaz.com/

7th May 2007, 09:21 AM	#3 (permalink)
fnord Newbie Join Date: 22 Apr 2007 Posts: 12 Reputation: 0 Rep Power: 0	I just updated my post above. I am currently hosting the site on a old Intel Celeron CPU 2.00GHz. I'll ask my hosting company for more details about what the problem is. Main problem is that the server gets to many requests, gets slower and slower leading to a crash. Whether this is a CPU or RAM problem I don't know. I guess a combination of both. Please define "I/O".

12th May 2007, 07:31 AM	#6 (permalink)
fnord Newbie Join Date: 22 Apr 2007 Posts: 12 Reputation: 0 Rep Power: 0	100% of both cpu and RAM is used the majority of the time and my server is kept up by several scripts which kill processes that threaten my server from crashing. My host advises these servers: Server A: - 2u server housing - 2x quad-core intel xeon 1.6 Ghz (replaceable) - 8GB DDR2 ECC memory (upgradeable to 64GB) - 2x 146 GB SCSI 15.000rpm - SCSI RAID controller - redundant power supply - 2x 1Gbit network - 3 year on site 24x7x4 hardware replacement (or 5 years + 1000 Euro) 5100 euro. ----- Server B: - 2u server housing - 2x dual-core intel xeon 2.0 Ghz (replaceable by quadcore at later time) - 8GB DDR2 ECC memory(upgradeable to 64GB) - 2x 146 GB SCSI 15.000rpm - SCSI RAID controller - redundant power supply - 2x 1Gbit network - 3 years on site 24x7x4 hardware replacement(or 5 years + 1000 euro) 4650 euro ----- Server C: - 2u server housing - 2x dual-core AMD opteron 2.0 Ghz (replaceable by quadcore at later time) - 8GB DDR2 ECC memory(upgradeable tot 64GB) - 2x 146 GB SCSI 15.000rpm - SCSI RAID controller - redundant power supply - 2x 1Gbit network - 3 years on site 24x7x4 hardware replacement(5 years: price not known yet) 4270 euro All prices include 3 years on site hardware replacement. (Within 4 hours) Without this the price goes down with €800. What do you think about this?

12th May 2007, 09:03 PM	#8 (permalink)
fnord Newbie Join Date: 22 Apr 2007 Posts: 12 Reputation: 0 Rep Power: 0	I don't know the exact processes, but it mostly happens when several things are happening at the same time(cron, backup, many users in chat, many users up/downloading files, various attacks together with heavy traffic) But overall my site stays up while it has 600 - 1000 users online most of the time.

13th May 2007, 11:51 AM	#9 (permalink)
peaz ARP Webmaster Join Date: 13 Oct 2002 Location: http://atpeaz.com/ Posts: 8,595 Reputation: 1673 Rep Power: 32	Hmmm all in all, could be high disk activity from the database updating on and on and on. I guess you can try upgrading the server. Even server C is overkill already. haha seriously. __________________ Ken Ng Tech ARP Follow me at Twitter Blogs @ http://www.atpeaz.com/

13th May 2007, 09:43 PM	#10 (permalink)
fnord Newbie Join Date: 22 Apr 2007 Posts: 12 Reputation: 0 Rep Power: 0	Yes, I am aware of that. But there are several factors that makes me decide for a top notch sever. I do not only want to cope with current server user, but also future use. I manage my site in order to maintain steep increase of use and traffic. I have a 300% increase in visitors per year and a steep increase of bandwidth use. Current server limitations have thus far limited site expansion. New expansions will surely intensify server use in many ways.