How does Vanilla scale?

goyangi

Hello vanilla community,
we're currently evaluating vanilla for a roughly 7k members community (somewhat tired of vbulletin). Questions regarding performance/scaling came up.

-How big is the biggest community vanillaforums.com hosts?
-Is that a special version of vanilla forums or can it be done with the stock downloadable version too?
-Would you recommend vanilla for a 7k-8k members community? Roughly causing 70-100GB/month traffic.

Thanks for any help

Rgds

Tim

Hello vanilla community,
we're currently evaluating vanilla for a roughly 7k members community (somewhat tired of vbulletin). Questions regarding performance/scaling came up.

-How big is the biggest community vanillaforums.com hosts?
-Is that a special version of vanilla forums or can it be done with the stock downloadable version too?
-Would you recommend vanilla for a 7k-8k members community? Roughly causing 70-100GB/month traffic.

Thanks for any help

Rgds

Hi there.

The short answer is that Vanilla scales very well.

The longer answer is far more complicated.

Vanilla's main advantage in the scaling arena is that it is designed to take full advantage of a multi-server environment. We currently host a forum that receives on the order of 12 million hits per month, transferring about 1.6 Tb of data in the same time. The forum itself loads in under 1 second on every page and sustains between 1,000 and 2,000 *concurrent* connections at any given time (fluctuating by time of day, obviously). There are approximately 100,000 members and 20 million comments.

Obviously such a site is not running off one server. In actual fact the setup is quite advanced:

8 servers in total, all running linux:

2 loadbalancers (1 active, 1 waiting to hot failover in case of a problem) using IPIP and pulse
4 webservers running nginx, each with 1Gb of ram
1 database server running Percona MySQL, with 8 Gb of ram (also running Sphinx search indexer)
1 content/cache server running nginx and memcached, with 1Gb of ram

We are able to scale such a cluster in 3 primary ways:

1) Additional webservers - This is done if the volume of traffic increases to the point where nginx is unable to satisfy requests in under a second and is having to delay responding until PHP gateways free up.

2) Additional cache servers - This is rare, but would happen if we increased the amount of cached data stored per user, per thread, etc and needed more ram (or to distribute cache hits to different systems due to network bottlenecks)

3) Database splitting or replication - This is a last resort and more complex than the previous two options, but still possible.

As you can see, Vanilla itself is able to handle a high volume of requests, but the technology stack required can be daunting. Considering that, we offer just such a stack as a hosted, Infrastructure As A Service offering through VanillaForums.com. We take care of hosting, maintenance, software updates, security, and backups.

I hope I've answered your questions to a satisfactory degree.

If you need any more information about how best to scale Vanilla, or about our vanillaforums.com hosting plans, don't hesitate to contact me!

Tim

422

Wufoo uses vanilla

goyangi

Thanks.

I know this is a somewhat generic question thats probably hard to answer. I'm speaking just ballparks. So any help is greatly appreciated.

x00

while you are still waiting for you answer here is my 2p

Generally the query's aren't bad, although I would not have a global activities, it is mostly mindless gibberish anyway, and it can slow things down. The user notifications and activities you might want to keep, depending.

I understand that some run on nginx with micro-caching

Vanilla main plus is it adaptability, but it might be more rough around the edges than you are used to. It suits developers, first and admins second.

goyangi

Tim,
thanks for your reply. This was exactly the kind of information I was looking for.

markd

Thanks for the informative post, Tim. I'm also evaluating Vanilla, but my community is more like the size of the client you are describing (100k members, 500k logins/month, 200k posts/month).

I'm curious about why it makes sense to run so many servers, instead of giving more resources to the ones you already have?

Todd

Using multiple servers is almost always the way you want to scale. We absolutely could combine some things into less, larger servers, but since we are on the cloud there is not much upside in using less servers. Let's look at some of the pieces.

Load Balancers
Since all traffic comes in through the load balancers it's really nice to have more than one just in case one goes down. That being said, I wouldn't lose too much sleep if I had only one load balancer. The load balancers are actually tiny servers and are never at load.

Webfronts/Webservers
We have four of these because if one web front goes down then it will only be a 25% reduction in our service. If we scale up a little more though we will probably go with three webfronts that are a little larger each.

Database Server
We do only have one database server and would prefer a master/slave scenario, but replication in MySQL is really a pain. Having multiple database servers used to be the absolute standard in scaling. Nowadays caching and nosql alternatives to mysql are a much better solution to scaling.

Cache Server
We only use one server for caching because memcached is very efficient and if the cache server goes down it's no big deal since the cache can always rebuild itself.

Combining Webservers

We've actually tried other combinations of servers. We ran with the cache on the database server for a while, but both those services need a lot of io so they were bottlenecking there. In the end we find it nice to be able to separate the services so that we can monitor what is and isn't causing bottlenecks. Sometimes it's difficult to pin point the culprit of a slow down if everything is mushed together.

What would I use for the least number of servers?

If I wanted to use less servers, but could still use multiple servers then I'd have the following setup:
* 1 loadbalancers.
* 2 webfronts running nginx and memcached.
* 1 database server.

What can I do if I only have one server?

If you only have one server then here is what you should do:

1. Don't use apache. It doesn't scale at all. Use nginx.
2. Use memcached.

These two things will be your two biggest wins.

jspautsch

Bookmarked! Some very useful information here, thanks guys.

x00

Some good info here. There is, I'm sure unintentional, the implication that that nosql isn't about using replication. Of course nosql is all about replication. I'm sure you were referring specifically to relational db clusters.

Tim

We were referring to MySQL replication being a pain, and NoSQL being less needy resource wise therefore prolonging the time during which replication is not required.

Todd

Yeah, most nosql solutions are great with replication thank goodness. It's great to see these solutions out there and I'd love to work with them some day. NoSQL requires a pretty big mindset change in a lot of cases though.

x00

Sure it does. I remember mark considering ORM, but in the the went with the code igniter solution which was more light weight. Nowadays, with noSQL they try to make migration minimal or non issue.

Honestly the mindset, except in transactional, is as it probably more how it should be in sql solutions but isn't, in other words much of data integrity is should be handled at the script/framework level, moving way from the old normalised way of doing it. A good example is how you zip the categories to the discussions.

x00

Also many noSQL solutions don't have integrity single server, they are dependant on replication.

arsipen

@todd: have you tried mysql-proxy yet? it is useful when there are plenty same query send to mysql

Todd

I haven't tried mysql-proxy yet no. We actually don't have scalability problems on the database side right now though. I think that caching is almost always superior to replication.

We do want to get more into replication for availability though. Do you recommend mysel-proxy?

Man, why is mysql replication such a pain?