|
Home > Archive > Perlbal > May 2007 > Anybody interested in testing patch to allow weighted nodes?
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Anybody interested in testing patch to allow weighted nodes?
|
|
| Brett G. Durrett 2007-05-16, 7:12 pm |
|
I have attached a patch to 1.57 that will allow weighting of nodes
specified in a nodefile. This might be handy if your servers have a
wide range of performance and you want to direct less traffic to less
powerful servers.
I am putting this out in case anybody has use for it and is willing to
test it 
Some notes:
- It only works using nodefiles, not the config file node definitions or
the configuration interface "add" or "remove" commands.
- I am running about 5% of our traffic through my production test
server... it appears to be working well but I would not feel comfortable
asking others to try it in production. I have done very little
testing... I am not sure about the behavior in failure cases, especially
in regards to failing-out bad nodes.
- To weight a node, add whitespace and weight:<int> to the nodefile...
for example:
10.10.10.78 weight:1
10.10.10.79 weight:1
10.10.10.124 weight:5
10.10.10.125 weight:5
- If you do not specify a weight for any node the weighting code is
ignored.
- If you specify a weight for some (but not all) nodes, it will assume a
weight of "1" for nodes that did not have a weight.
- If weights are assigned, the "show pool <pool>" command will add the
weights of all nodes to the output... if you did not use weighting, the
standard output will be displayed. For example:
show pool virtual_web
10.10.10.78:80 765 weight:1
10.10.10.79:80 770 weight:1
10.10.10.124:80 4007 weight:5
10.10.10.125:80 3921 weight:5
Again, this was a quick hack and I am providing this for testing... if
your Perlbal is controlling a nuclear reactor, please don't try this.
Thanks in advance for any testing / feedback you can provide,
B-
| |
| Brad Fitzpatrick 2007-05-16, 7:12 pm |
| I don't think this would do what you think it would.
This would just be for allocating new connections (which is kinda
boring, and happens relatively rarely, and still only one
per-node-in-flight), not for how requests are actually assigned to
backends.
On Wed, 16 May 2007, Brett G. Durrett wrote:
>
> I have attached a patch to 1.57 that will allow weighting of nodes
> specified in a nodefile. This might be handy if your servers have a
> wide range of performance and you want to direct less traffic to less
> powerful servers.
>
> I am putting this out in case anybody has use for it and is willing to
> test it
>
> Some notes:
>
> - It only works using nodefiles, not the config file node definitions or
> the configuration interface "add" or "remove" commands.
> - I am running about 5% of our traffic through my production test
> server... it appears to be working well but I would not feel comfortable
> asking others to try it in production. I have done very little
> testing... I am not sure about the behavior in failure cases, especially
> in regards to failing-out bad nodes.
>
> - To weight a node, add whitespace and weight:<int> to the nodefile...
> for example:
>
> 10.10.10.78 weight:1
> 10.10.10.79 weight:1
> 10.10.10.124 weight:5
> 10.10.10.125 weight:5
>
> - If you do not specify a weight for any node the weighting code is
> ignored.
> - If you specify a weight for some (but not all) nodes, it will assume a
> weight of "1" for nodes that did not have a weight.
> - If weights are assigned, the "show pool <pool>" command will add the
> weights of all nodes to the output... if you did not use weighting, the
> standard output will be displayed. For example:
>
> show pool virtual_web
> 10.10.10.78:80 765 weight:1
> 10.10.10.79:80 770 weight:1
> 10.10.10.124:80 4007 weight:5
> 10.10.10.125:80 3921 weight:5
>
> Again, this was a quick hack and I am providing this for testing... if
> your Perlbal is controlling a nuclear reactor, please don't try this.
>
> Thanks in advance for any testing / feedback you can provide,
>
> B-
>
>
>
>
>
>
| |
| Brett G. Durrett 2007-05-16, 7:12 pm |
|
Actually, that is what I thought it would do
My preference would have been to have it be on each request but we are
running enough perlbals and have enough traffic that the reduced new
connections should have an impact (we have about 20+ perlbals for this
service and they create new connections with reasonable frequency, even
with back-end keepalives on).
I am not sure this would be valuable for light traffic... in my initial
testing connections favored (and stuck) to super high weights and would
sometime stick with a lower weight, once picked.
B-
Brad Fitzpatrick wrote:[vbcol=seagreen]
> I don't think this would do what you think it would.
>
> This would just be for allocating new connections (which is kinda
> boring, and happens relatively rarely, and still only one
> per-node-in-flight), not for how requests are actually assigned to
> backends.
>
>
> On Wed, 16 May 2007, Brett G. Durrett wrote:
>
>
| |
| Brad Fitzpatrick 2007-05-17, 1:11 am |
| Okay, so it won't be what other people expect.
On Wed, 16 May 2007, Brett G. Durrett wrote:
>
> Actually, that is what I thought it would do
>
> My preference would have been to have it be on each request but we are
> running enough perlbals and have enough traffic that the reduced new
> connections should have an impact (we have about 20+ perlbals for this
> service and they create new connections with reasonable frequency, even
> with back-end keepalives on).
>
> I am not sure this would be valuable for light traffic... in my initial
> testing connections favored (and stuck) to super high weights and would
> sometime stick with a lower weight, once picked.
>
> B-
>
> Brad Fitzpatrick wrote:
>
>
| |
| Ask Bjørn Hansen 2007-05-17, 1:11 am |
|
On May 16, 2007, at 4:05 PM, Brett G. Durrett wrote:
> I have attached a patch to 1.57 that will allow weighting of nodes
> specified in a nodefile. This might be handy if your servers have
> a wide range of performance and you want to direct less traffic to
> less powerful servers.
Why not just set the less powerful servers to only allow however many
connections they can deal with?
- ask
--
http://develooper.com/ - http://askask.com/
| |
| Brad Fitzpatrick 2007-05-17, 1:11 am |
| On Wed, 16 May 2007, Ask Bj=F8rn Hansen wrote:
>
> On May 16, 2007, at 4:05 PM, Brett G. Durrett wrote:
>
>
> Why not just set the less powerful servers to only allow however many
> connections they can deal with?
Exactly.
| |
| Justin Huff 2007-05-17, 1:12 am |
| > Why not just set the less powerful servers to only allow however many
> connections they can deal with?
We've considered using a weighted system so that we could send some
small amount of traffic to a particular backend to test out code changes.
Of course, in our case, using a cookie to make sure that chosen users
hit the same server consistently would probably be better.
--Justin
| |
| Brett G. Durrett 2007-05-17, 1:12 am |
|
Yeah, you're right... I think people will probably expect what you
expected, a per-request weight. I should have been more clear about the
"weighting" mechanism. Like I said, my preference would be on a per
request weight but this was a quick hack that looks like it meets our
specific needs.
To get more feedback and possibly influence a revised patch... I assume
people are achieving weighting by limiting the connections at the web
server level... does this just work for everybody or is there a
preference for controlling the weighting via perlbal? Controlling
weighting at the load balancer level seems like a common expectation
(granted, with a per-request level of control) but maybe I am thinking
about this the wrong way.
So my question to this mailing list is... does the perlbal community
see any value in having a weighting mechanism? If the answer is no, I
would love to hear how you are achieving the same effect and why your
solution is your preferred mechanism.
Thanks in advance for any constructive feedback,
B-
Brad Fitzpatrick wrote:[vbcol=seagreen]
> Okay, so it won't be what other people expect.
>
>
> On Wed, 16 May 2007, Brett G. Durrett wrote:
>
>
| |
| Brad Fitzpatrick 2007-05-17, 1:12 am |
| On Wed, 16 May 2007, Brett G. Durrett wrote:
> Yeah, you're right... I think people will probably expect what you
> expected, a per-request weight. I should have been more clear about the
> "weighting" mechanism. Like I said, my preference would be on a per
> request weight but this was a quick hack that looks like it meets our
> specific needs.
>
> To get more feedback and possibly influence a revised patch... I assume
> people are achieving weighting by limiting the connections at the web
> server level... does this just work for everybody or is there a
> preference for controlling the weighting via perlbal? Controlling
> weighting at the load balancer level seems like a common expectation
> (granted, with a per-request level of control) but maybe I am thinking
> about this the wrong way.
>
> So my question to this mailing list is... does the perlbal community
> see any value in having a weighting mechanism? If the answer is no, I
> would love to hear how you are achieving the same effect and why your
> solution is your preferred mechanism.
In my experience, people think they want a weighting mechanism, but they
really, really don't. The load balancer can be much smarter and quicker
to adapt than any human adjusting weights.
Especially modern web apps with widely-varying response times. Perhaps in
the past with all backend nodes serving the same 3 static files, then
statically-defined weighted load balancing might make sense... but still
not really.
I'd need a great reason to include such a feature.
That said, an upcoming change to Perlbal will make weighted requests to
backends a lot easier to implement properly, but I might still fight it,
or at least make it really hard to enable on accident.
- Brad
| |
| Ask Bjørn Hansen 2007-05-17, 7:11 am |
|
On May 16, 2007, at 10:44 PM, Brett G. Durrett wrote:
> So my question to this mailing list is... does the perlbal
> community see any value in having a weighting mechanism? If the
> answer is no, I would love to hear how you are achieving the same
> effect and why your solution is your preferred mechanism.
"Load balancing" is short hand for "give each server an appropriate
load".
If you have two servers, one "twice as powerful" as the other, the
"appropriate load" isn't to send twice as much traffic to the "more
powerful" one.
The appropriate load is to send as much traffic to each server as it
can efficiently handle.
For most applications the primary resource constraint is either
memory or CPU. In either case you figure out how many simultaneous
requests the server can/should handle and set the "maximum concurrent
requests" to that number. Viola, perfect distribution of the traffic.
- ask
--
http://develooper.com/ - http://askask.com/
| |
| dormando 2007-05-17, 7:11 am |
| > So my question to this mailing list is... does the perlbal community
> see any value in having a weighting mechanism? If the answer is no, I
> would love to hear how you are achieving the same effect and why your
> solution is your preferred mechanism.
>
> Thanks in advance for any constructive feedback,
Twiddling values on the arrowpoint/bigip/netscaler/alteon/whatever is
something I don't miss. It's embarassing how bad an expensive netscaler
does its job compared to perlbal.
I used to wish perlbal would allow me to specify the max connections per
node in the nodefile. Firstly to make it easier to twiddle the settings
without having to push config updates to dozens of servers. Secondly so
perlbal knows not to make an extra connect attempt if you're at the
limit already.
But uh. We now have scripts that detect how fast a machine is, reduces
the number if it's running other services (mogstored, memcached), and
handles that magically now.
Are you having an issue with connecting more often than necessary? With
almost 200 backend servers with 4-12 apache processes each, we're only
initiating new connections a couple times per second. Once you're
connected up the load just "flows" I've only ever had issues with
perlbal's backend connection mechanisms when something's breaking keepalive.
-Dormando
| |
| Brett G. Durrett 2007-05-17, 1:11 pm |
|
Okay, it sounds like most people are finding the optimal connections for
a specific server and then limiting the connections on a server by
server basis. That makes sense.
Here is the problem where I thought weighing adds value. Again, I may
be thinking about this the wrong way so feedback from others is
appreciated...
If I balance loads by limiting the number of connections to each machine
to what is optimal for that machine, this should work great under
"normal" conditions, especially since the pool of servers has enough
available connections to handle the total number of backend connections
and connections to spare. However, sometimes our loads (and required
back end connections) will dramatically change... maybe a code change
unexpectedly requires more resources and bogs down servers or perhaps we
just get a big surprise surge in usage and need to deploy more hardware
to address the load. In these cases where the total load (and required
back end connections) is simply too great for each server to run
optimally I have wanted the non-optimal loads to be shared among the
pool relative to their ability to handle the load. By limiting
connections at the server level I would expect that in a load surge my
servers would be at optimal loads but more requests would simply fail
(as opposed to requests taking longer). Of course we could setup
scripts to look at the overall connections required and adjust each web
server configuration dynamically if we exceed peak, but the simplicity
of a Perlbal weighting mechanism seemed to work well in this particular
situation.
To be clear, I am not arguing that weighting is the best solution or
even the right solution, I am trying to explain where I think it can add
value and find out from others how they address similar situations. I
would be quite happy to change my ways to something that works a lot better.
Thanks,
B-
dormando wrote:
>
> Twiddling values on the arrowpoint/bigip/netscaler/alteon/whatever is
> something I don't miss. It's embarassing how bad an expensive
> netscaler does its job compared to perlbal.
>
> I used to wish perlbal would allow me to specify the max connections
> per node in the nodefile. Firstly to make it easier to twiddle the
> settings without having to push config updates to dozens of servers.
> Secondly so perlbal knows not to make an extra connect attempt if
> you're at the limit already.
>
> But uh. We now have scripts that detect how fast a machine is, reduces
> the number if it's running other services (mogstored, memcached), and
> handles that magically now.
>
> Are you having an issue with connecting more often than necessary?
> With almost 200 backend servers with 4-12 apache processes each, we're
> only initiating new connections a couple times per second. Once you're
> connected up the load just "flows" I've only ever had issues with
> perlbal's backend connection mechanisms when something's breaking
> keepalive.
>
> -Dormando
|
|
|
|
|