Nano-Howto to use more than one independent Internet connection.

by Christoph Simon v1.0, Dec 2, 2001

The solution presented here is based on the patches by Julian Anastasov. Like several other users on the net struggling with this, I believe they fill in yet another gap for using Linux as a very advanced router. It was hard for me not only to get this running for the first time, but also to understand most of the principles. Also in this Julian has proofed to have unlimited patience. Thanks.

He also agreed to expose this file on his site:

http://www.linuxvirtualserver.org/~julian/nano.txt

I've been looking for a solution to use more than one connection to Internet at the same time for a long time. I've found out, that I'm not alone; there seem to be many having this very problem. The reasons are generally just to cheaply increase total throughput, or to provide for a fail-over in case of cheap lines. This is not the situation of a cluster of web-servers, where main traffic is input, but for networks which are mainly dedicated to any kind of download.

The setup of all this is not a question of 5 minutes, so I'll tell at the beginning where are the limits.

First, this will not work for a single computer with two modem connections. Load balancing does work reasonably well, but only with a high number of simultaneous connections to different remote sites.

The tests I'm performing are on a network with 17 client computers with an daily download (12h) of some 600MB of HTTP traffic in several hundreds of thousands of connections.

When one connection fails while being used, this connection will be lost and can't be continued on another line. The user will have to establish this connection anew. If it was a big download and the server doesn't support continuation, he'll be out of luck.

The lines I'm using have external modems which react as any other device with Ethernet. There is no PPP, PPPoE, etc. I can't tell if this works or would need modification for instance to use two telephone lines. Maybe someone else can fill in this.

What does work is:

• there is no restriction to 2 lines, though I didn't test more.
• using 2.4 kernels
• interaction with ip(8) and iptables(8)
• usage of both lines at the same time
• if any line fails, new connections will use any other which working.
• if a line fails, no reconfiguration of the network is required.

Right now, I can't tell how long it takes in case of a fail over, but there where several cases I found in the logs and no user realized that there was a problem.

Stock kernels are not able to provide this functionality. Julian Anastasov wrote a patch for 2.2 and 2.4 Linux kernels. I only tried with several 2.4 kernels, now running 2.4.15pre5. The patches have not been written for this version, but apply cleanly. The patches are available on Julian's web-page:

http://www.linuxvirtualserver.org/~julian/#routes

Choose the patches for your kernel, download them, and apply all of them.

The patches will not offer any additional configuration options. The kernel needs to have “equal cost multi path” enabled. This is done in Network Options. After choosing “IP: advanced router” there will be an option for it. Only an indirect requirement is, that we will need NAT: Any host on the local network needs to be able to appear on the Internet with any of the external IPs; this is the main purpose of NAT. To get NAT, of course we have to enable connection tracking. But this would be done anyway. No further options are required, but any networking option may be used. Particularly, I've enabled almost all options for netfilter and QoS and it's working fine.

Nothing special is required to run this kernel (no boot options,etc.).

As I stated at the beginning, to make use of the patches and the configuration described here, we need a relatively high number of connections and remote servers. I've tried bitterly, but wasn't able to achieve this effect alone. I started several simultaneous downloads, ran Mozilla on several computers at the same time, opening different big sites, but no avail. When the regular users came, things suddenly showed to work correctly. So this is certainly not a setup for a single box, and then, we'll need to do some address translation. Beside this, a firewall will almost always be unavoidable, but it's setup is out of scope here. A minimal setup would enable forwarding in the kernel, establish a default policy of ALLOW, and create at least two SNAT rules:

iptables -t nat -A POSTROUTING -o IFE1 -s NWI/NMI -j SNAT --to IPE1
iptables -t nat -A POSTROUTING -o IFE2 -s NWI/NMI -j SNAT --to IPE2

If the ISP didn't give us a fixed IP, for instance using PPPoE, but it seems that it must be an ARP type device, we would say:

iptables -t nat -A POSTROUTING -o IFE1 -s NWI/NMI -j MASQUERADE
iptables -t nat -A POSTROUTING -o IFE2 -s NWI/NMI -j MASQUERADE

I didn't try this, but it will work if the analog thing works with just one line. This is not really new, as we would have to do something similar anyway, even with only one line to the Internet.

At least for netfilter (not sure for ipfwadm/ipchains), the firewall must be stateful. This can be done by:

iptables -t filter -N keep_state
iptables -t filter -A keep_state -m state --state RELATED,ESTABLISHED \
	-j ACCEPT
iptables -t filter -A keep_state -j RETURN

iptables -t nat -N keep_state
iptables -t nat -A keep_state -m state --state RELATED,ESTABLISHED \
	-j ACCEPT
iptables -t nat -A keep_state -j RETURN

and calling this at the beginning of the script:

iptables -t nat -A PREROUTING -j keep_state
iptables -t nat -A POSTROUTING -j keep_state
iptables -t nat -A OUTPUT -j keep_state
iptables -t filter -A INPUT -j keep_state
iptables -t filter -A FORWARD -j keep_state
iptables -t filter -A OUTPUT -j keep_state

Here I'll assume that the computer has three network cards: one to the local network, and two for each of the external links (DSL modems in my case). This needs not to be so, as there could be more DSL modems, and, at least in theory, it's possible to use one card for more than one modems, using a hub and two or more IPs on the same device. This is not a recommended setup because it may introduce security problems, but might be the only one if we had more lines than PCI slots for additional network cards.

Abbreviations:

The local loopback must be up and running. There is nothing unusual about this.

ip link set lo up
ip addr add 127.0.0.1/8 brd + dev lo

At this point, we should be able to ping localhost.

We also set up the internal interface IFI with the internal address IPI and the netmask for the local network NMI. Assigning an address will cause the kernel to automatically insert an appropriate route into table main. This is just fine. Later, there will also be a route for each external interface, but none of the routes in table main will have a gateway.

We'll want to give table main a priority of 50 to make sure it is looked at first.

But also we want to make sure that there is no default route in table main. The commands so far didn't insert any, but if this is a reconfiguration, there might remain some from the previous setup. If there is none, the last command will fail, but that's OK: we just want to be sure.

ip link set IFI up
ip addr add IPI/NMI brd + dev IFI
ip rule add prio 50 table main
ip route del default table main

At this point, we should be able to ping any host on the local network.

The configuration of the external interfaces isn't really different from that of the internal, but for now, we don't specify a gateway nor do we insert a default route.

ip link set IFE1 up
ip addr flush dev IFE1
ip addr add IPE1/NME1 brd BRD1 dev IFE1

ip link set IFE2 up
ip addr flush dev IFE2
ip addr add IPE2/NME2 brd BRD2 dev IFE2

As ip(8) allows to assign more than one address to an interface, it's important that there doesn't remain any configuration from a previous setup; failing to do so might cause the same address to be specified twice. That's what the “addr flush” commands are for. When we actually specify the address (the last line of each block), the kernel, again, will insert automatically a route into table main, very much like in the case of the internal interface. Now, we should be able to ping the external IPs (IPE1 and IPE2), as well as the gateways or any other host or device which belongs to those external network addresses. In my case there is none beside the gateways. For this reason it would be correct to consider such an interface like a point-to-point device, but technically it is not.

This is the complicated part of the setup. During this discussion, I'll mention the setup commands step by step. In the end, I'll repeat them all together.

Before really starting, let's make a small experiment: Using any kind of packet tracer (logs from iptables, tcpdump, etc), we will observe that if we ping 127.0.0.1, the source address of that packet will be 127.0.0.1; if we ping any address of the local network, the source address will be that of the internal interface. And if we ping any host or device outside, the source address will be that of the device where the packet is going out. Who is being that smart? It might be the application (e.g., ping) itself, using a bind(2) call, but this is not normally done for a client. It's the routing system. In other words, until a packet actually reaches the routing system, it doesn't have a source address.

Well, in practice, some programs try to be smarter than the routing system, use the bind(2) call, but actually choose a less than optimal route. This would easily be the case under our setup, as a program can't know which of the two external interfaces it should use. Also, there are some versions of ping which allow the user to specify the source address. While being an interesting tool also in our setup, the `being smart' is then really up to the user.

As I stated in the requirements, NAT is one of them and so is connection tracking. This consists of a set of information for each known connection, including specific NAT details.

We see, even if the packet originates on the same computer, in our case, a packet can have one of four addresses: 127.0.0.1, IPI, IPE1, or IPE2. This is called the local source address, which plays a major role when doing NAT. Here, this was one of the key concepts I had to understand to be able to understand the rest.

I'll try to `deduce' the routing commands by following a packet from a host on the local network. While it might also originate from the computer we are configuring, this will be the most typical path. I'll call that host w1 (the first workstation).

The setup of w1 is as usual: It has a loopback interface, a network card with an IP address compatible with the local network address, and a default route which uses as a gateway the computer we are configuring here (IPI). As soon as the user on w1 does anything which requires access to Internet, her program will set up a socket and call connect(2). Then, an IP packet will be formed. The routing system on w1 will set the source address to w1's local IP, the destination address to that of the remote server on the Internet. It will use the default route to send this packet to us.

The packet will come in by the local interface IFI, being welcome, if present, by the prerouting chain of the mangle table, but certainly by the prerouting chain of the nat table.

The nat table will search the already existing connection structures and find out, that this is a new connection. It will create an uninitialized info-block, and pass the packet to the routing system.

At this time, we have the following information: The packet came in by the local interface IFI, has the source address of w1, but we don't know yet the local source address.

Being a new connection, there will be nothing matching in the cache. Also, our configuration so far doesn't have any default route yet. So we need one which tells what to do with such a packet:

ip rule add prio 222 table 222
ip route add default table 222 proto static \
	nexthop via GWE1 dev IFE1 \
	nexthop via GWE2 dev IFE2

This is a multipath default route, causing the kernel to extract each time another alternative; there could be more than these two.

Now, we get two additional informations: the output device (either IFE1 or IFE2) and the IP address of the gateway (either GWE1 or GWE2). Actually we also may get a third information, which is our local IP address, which is not contained in this route, but the system will deduce it by interface and gateway if it is not a dynamic IP. That's it, routing is doing for us right now, beside sending the packet to the forward chain of the filter table.

Of course, forwarding must be enabled in the kernel, and there must not be any netfilter rule disallowing this packet. After that, the packet is passed along to the postrouting chain of the nat table.

Using the information about the packet we gathered so far, the NAT system will look for a rule, and find one of the two we gave in the previous section. If our IP is fixed, now the local source address will became known, but if it is dynamic, i.e., we used the MASQUERADE target, we will need to go back to routing, as there is no other way to find out the right local source address. The answer will come from table 222, the multipath route, but this time, also interface and gateway need to match. The NAT system will insert this into the, now initialized, NAT information in the connection structure, replace the source address in the packet by this one (which was still the IP of w1), and get this packet on it's way to the gateway, going out by the proper output device.

Note here, that when asking the multipath route, table 222, with a known output device and gateway, the kernel still has to deduce the IP address. It will do so, looking at the first address attached to the interface which, by netmask, is compatible with the gateway address. If there were more than one possible answers, only the first will be used. This might be the case if we have both lines on the same network card, which will work only if we use the MASQUERADE target in the iptables rules. If we didn't use the gateway (stock kernels don't) to make this match, we surely would be in troubles, because the output device is the same for both. This means: To be able to use more than one Internet connection on one and the same network card, the gateways must be different, the IP addresses must be public IPs and must not belong to the same subnet. Additionally, we should avoid SNAT in the iptables rules and take special measures to protect against IP spoofing.

The gateway, and potentially lots of routers in the Internet, are supposed to do the right thing to make this packet reach it's destination. There, the answer packet will use the source address (which is now either IPE1 or IPE2) to come back into our computer by the same interface where it left. Essentially, the procedure is reversed; our routing table will find the way to w1 already in table main.

Most user actions on the Internet aren't happy with just one packet of data; let's have a look what happens if a second packet is sent from w1 to the same destination:

Again, the packet will arrive at the local interface of this computer and pass through the prerouting chain of the mangle table if that exists. The prerouting chain of the nat table will start looking for an existing, matching connection, and this time, it will find one. As there is no DNAT rule in the netfilter setup for this packet, NAT will just hand over the packet to the routing system.

The routing system will look at the connection, and the patches will find that there is NAT information present, and that after routing some mangling will be done. Now, the routing question will give the input device (IFI), the source address (the IP of w1), the destination address (that of the server on the Internet), and the local source address (either IPE1 or IPE2). In this situation, we do not want to match the multipath route, because the multipath route doesn't look at local source address, we couldn't make sure that we get this time the same line as before. So we add a table for each interface:

ip rule add prio 201 from NWE1/NME1 table 201
ip route add default via GWE1 dev IFE1 src IPE1 proto static table 201
ip route append prohibit default table 201 metric 1 proto static

ip rule add prio 202 from NWE2/NME2 table 202
ip route add default via GWE2 dev IFE2 src IPE2 proto static table 202
ip route append prohibit default table 202 metric 1 proto static

This can't be matched when table 222 should be matched, as we now require the local source address to be known. We choose a priority that makes sure these routes are found after the main table and before table 222, which is more general than these.

These are still default route, because they must match any Internet address.

The third line of each block is similar to a REJECT target in iptables in case the corresponding interface is not working: If the client on the local network sends a packet on an established connection, but in the meanwhile the interface stopped operating, we will send this client an ICMP controll message `PKT_FILTERED', hoping to cause it to stop sending packets, and the user might wish to open a new connection, which will succeed if at least one other line is still working.

Stepping through all routing tables, either table 201 or table 202 will match, according to the output device selected when we matched the multipath route for the first packet. The tricky part is, what the patches are doing here: They look at the local source address. If it is zero, the old behavior would just match the multipath route in table 222, but as we do know already the local source address (i.e., it is not zero) from what we found in the connection structure, particularly what concerns NAT, we will try to match the rules against the local source address rather than the packet's source address, which is still the IP of w1. This is why either table 201 or table 202 will provide the match, telling the gateway address and the output device.

Having done it's job, the routing system will send the packet to the forward chain of the filter table, to continue in the postrouting chain of the nat table. That has an easy game now, because, no matter if the found target is MASQUERADE or SNAT, all necessary information is already known. The packet's source address will be replaced by our local source address and the packet can continue it's way going out by the chosen output device, and reach it's destination through the Internet. The same procedure is repeated for all further packets, because the connection information is kept alive during a certain time, even after closing the connection (FIN or RST).

Now let's have a look at a packet which originates on this computer with a destination on the Internet. As we found out in our experiment, the source address is not known before routing, and we won't need any NAT. Here we will assume that the program which generates this packet does not use the bind() call. If it would, either table 201 or table 202, according to the IP this program binds to, would be used and no (further) load balancing can take place. So, the packet will start crossing the output chain of the mangle table, if that is present, and go to the output chain of nat. As there is no rule for a destination nat, it will continue being evaluated in the routing system. No local source address is known yet, so the multipath route in table 222 will match. This will provide the output device, the gateway and also the local source IP. Now the connection information can be completed. With that, the packet will cross the output chain of the filter table and reach the postrouting chain of the nat table. From there the packet can be sent to the gateway. From now on, the connection is known and all further packets will use this path.

Now the full sequence of commands, all together. For convenience, I'll repeat the meaning of the abbreviations:

Abbreviations:

ip link set IFI up
ip addr add IPI/NMI brd + dev IFI
ip rule add prio 50 table main
ip route del default table main

ip link set IFE1 up
ip addr flush dev IFE1
ip addr add IPE1/NME1 brd BRD1 dev IFE1

ip link set IFE2 up
ip addr flush dev IFE2
ip addr add IPE2/NME2 brd BRD2 dev IFE2

ip rule add prio 201 from NWE1/NME1 table 201
ip route add default via GWE1 dev IFE1 src IPE1 proto static table 201
ip route append prohibit default table 201 metric 1 proto static

ip rule add prio 202 from NWE2/NME2 table 202
ip route add default via GWE2 dev IFE2 src IPE2 proto static table 202
ip route append prohibit default table 202 metric 1 proto static

ip rule add prio 222 table 222
ip route add default table 222 proto static \
	nexthop via GWE1 dev IFE1 \
	nexthop via GWE2 dev IFE2

Let's check it out:

ip address

This should print on the terminal one entry for the local loopback, IFI, IFE1 and IFE2, and maybe some other things, if we have it configured (like my GRE tunnels).

ip rule

This should look like this:

0:      from all lookup local
50:     from all lookup main
201:    from NWE1/NME1 lookup 201
202:    from NWE2/NME2 lookup 202
222:    from all lookup 222
32766:  from all lookup main
32767:  from all lookup default

Table local is used for the local loopback, table main has the network routes to the internal network and for the external interfaces, which only give access to our gateways. Tables 201 and 202 (which also might have the same priority), will provide a default route if the local source address is known (because they have to match NWE1 or NWE2). And table 222 will provide the multipath route. The tables with priority 32766 and 32767 will not be used.

  ip route list table main

Giving:

NWI/NMI    dev IFI   proto kernel  scope link  src IPI
NWE1/NME1  dev IFE1  proto kernel  scope link  src IPE1
NWE2/NME2  dev IFE2  proto kernel  scope link  src IPE2

These are only routes to the corresponding networks without using a gateway.

ip route list table 201

Giving:

default via GWE1 dev IFE1  proto static  src IPE1
prohibit default  proto static  metric 1

And:

ip route list table 202

Giving:

default via GWE2 dev IFE2  proto static  src IPE2
prohibit default  proto static  metric 1

These are the default routes requiring the local source address to be known. And finally:

ip route list table 222

Giving:

default  proto static
        nexthop via GWE1  dev IFE1 weight 1
        nexthop via GWE2  dev IFE2 weight 1

which of course is our multipath route. The weights where inserted by the kernel and could be changed. Actually, they need to be changed specially if the two lines have a very different capacity. With this configuration, both lines are going to be used the same amount of times (at least in the long run), even in case the first line has 4 times the capacity of the second. To compensate for that, we would add “weight 4” after IFE1, like in:

ip route add default table 222 proto static \
	nexthop via GWE1 dev IFE1 weight 4\
	nexthop via GWE2 dev IFE2 weight 1

The numbers should be as small as possible, because in this case, the kernel will create 5 alternative routes, four of them being the same. For this reason, it wouldn't really be an equivalent to say “weight 400” and “weight 100”, nor would it be a good idea.

Now, let's see how the kernel chooses certain routes:

Asking, how to reach the gateway:

ip route get GW1

We get the route from table main:

GW1 dev IFE1  src IPE1
    cache  mtu 1500

Now, let's ask for the route to a host on the internet. Note, that the ip(8) command does no DNS lookups; we need to provide the IP. This one goes to www.kernel.org:

ip route get 204.152.189.113

My system answered:

204.152.189.113 via GWE1 dev IFE1  src IPE1
    cache  mtu 1500

i.e., having chosen the first interface. As it got this from the cache, it won't be easy to get the answer for the second interface until the cache expires, or we flush it explicitly. If not found in the cache, this answer must come from table 222. To test tables 201 and 202, we can use:

ip route get from IPE1 to 204.152.189.113
ip route get from IPE2 to 204.152.189.113

and would get respectively:

204.152.189.113 from IPE1 via GWE1 dev IFE1
    cache  mtu 1500
204.152.189.113 from IPE2 via GWE2 dev IFE2
    cache  mtu 1500

Up to here, we know how all external lines are being used while they are working. It's also easy to guess what happens if all external lines fail. But let's see what happens if only one external line fails:

At a first glance, nothing special happens. If one of the routes in the multipath statement is unavailable, it just stops being eligible and we get always the other one (one of the others, if more than two). But it is less evident, what happen during the transition.

When we try to send a packet to another device, and that fails, like in real live, we'll blame the other side. But if we can't reach one neighbor, it doesn't mean necessarily that we can't reach any neighbor. This depends on the reason for the failure. If the problem is in our network card or in the cable attached to it, we won't reach any other, but it's also possible that only this remote device is down. Well, in our particular case, this is the same, because, from the functional point of view, this is a point to point connection, and beside the gateway and ourselves, there is no other device. But the lower level transport system can't know that and hence will mark this link just as suspected to fail. Neither the routing cache nor the connection information have yet knowledge of the problem, and the user's connection will block.

After a certain time, Ethernet will consider the gateway dead and mark it as unreachable. If this is because the link was brought down, the routing cache will be flushed immediately, forcing the next packet to choose a new route, which will get only a working one. Otherwise, the packet could use a route in the cache which is actually unreachable, causing it to fail: All connections which are in the cache and which use this route will just go nowhere. After a certain interval of time, the entries in the routing cache expire, and than the route have to be looked for again, offering only working routes, and things will work again.

From the user's point of view: An established connection using a line which stops working will be lost. She'll need to establish a new connection. Until the cache expires, a new connection will work only if the route isn't yet in the cache, or if the cached route uses a working line. But if the route is cached and uses the failing line, the connection will not work.

The time until the cache expires is controlled by the kernel variable /proc/sys/net/ipv4/route/gc_interval and defaults to 60 seconds. The bigger this number, the longer it will take until failing routes are not being used again, but the shorter this time, the smaller the chances to find a connection in the cache, loosing time to look up the connection in the routing tables.

The inverse is analog: If the line comes back up, cached routes won't use it. But routes found by matching the multipath route in table 222 will consider this route as eligible again, and possibly use it. But more on this in the next section.

We're almost done. There is a subtle issue about the kernel detecting the state of an interface. We can not find out whether an interface is working without just trying to use it. As long as the kernel doesn't use the interface, it can't know if it is working or not. When we try to use it, the kernel will mark the interface according to the result of that operation. If this means, the interface is down, and we continue trying to access Internet, the kernel will simply use the other interface. How then can the kernel find out, that the failing interface is working again? Well, it can't, because it wouldn't even try. This is good, because users we wouldn't like to have to wait for a timeout with each packet. But then, there needs to be a way to notice when the interface comes back up. The only way is to use the interface explicitly with non critical packets. Before using these patches I had written a daemon which would set up explicit routes to each gateway and send pings in regular intervals (once per minute). Then the default route was set either to the preferred interface or to the only one working. Now, this daemon doesn't need to change routes anymore, but it still has to send the pings.

But there isn't really a daemon needed for this. A simple script:

while : ; do
  ping -c 1 GWE1 > /dev/null 2>&1
  ping -c 1 GWE2 > /dev/null 2>&1
  sleep 60
done

can do this.

At the beginning, I was worried because I've a link where the provider doesn't reply pings. But this is really not necessary. The trick is, that ARP requests must be answered for anything to work. One consequence of this is, that for now I'm only sure that this works for devices which are based on Ethernet, i.e., which use the ARP. Other devices, like dialup links, can only work if the underlying network protocol provides a correct status change. But I'm not aware if or which Linux protocols are implemented this way.

What I've learned from this is, that it's not enough to just set a device up or down, or to add or remove a route. Somehow, the kernel must also be told, that this device or route is working or stopped doing so.

Running the pings once every minute all the time isn't what seems to be an elegant solution, but it works. Hopefully, someday we'll get this done behind the scene.

During development of this setup, I've considered using arping rather than ICMP echo replies. Both can be problematic. The gateway can drop echo messages, so ping wouldn't work. On the other hand, I've noticed in two situations that the provider apparently had installed some kind of load balancing making the MAC address of the gateway toggling from one to another. This renders arping useless. So I stayed with ping. This will work anyway, because my daemon doesn't need to know the result: Even if I don't get an ICMP echo reply, the kernel will fill in the right MAC address for this gateway and know that it is reachable or not.

It seems that newer versions of arping use a broadcast mode, which would solve the problem with a provider that changes the MAC address of the gateway every few minutes. I didn't try it yet, but I will.

Some of us are curious and wish to know what the Julian's patches actually do. Here is a summary:

• With the patches enabled, we can use “proto static” for all routes. Doing so, the route for a link will survive even if that device stops working. This is essential, specially if the reason for using two lines is redundancy: If a line fails, all attributes of the device remain, beside the fact that it can't be used until it comes up again. We can continue pinging the gateway for to detect the moment the link comes up again. Equal cost multipath exists also without these patches, but wouldn't work in such a situation.

• If the device continues existing even if it doesn't work, the kernel needs a way to decide whether this is a route might be chosen, specially if several are available. This is called the ``dead gateway detection; it works for any kind of unicast routes (multipath/unipath, input/output) and provided by these patches. * A multipath route consists of several routes with the same source and destination, but a different path, which is specified by the device and the gateway. It is because of the patches that the kernel knows how to distinguish the route not only by source and destination, but also by device and gateway. Also, at least when each line has a different device, the kernel's reverse path checking (rp_filter)'' will work correctly.

• The NAT system has no idea how to deal with alternative routes, this is something the routing system has to do. And one of the additions of these patches is make the routing system prepare the route of an individual packet for the nat system, such that it looks as if it where the only possible route.

It's very little time that I use this, so obviously I can't give any long term results. But after the first few days of working, I can say that load-balancing is working reasonably well. I could imagine a more fine grained behavior which would benefit single users with two DSL lines, but in my particular situation I've seen a difference on total load of only 15%.

Fail-over happened already several times and went by without being noticed by the users. In the time I'm using this, I didn't experience any problem. It just works .

ciccio@kiosknet.com.br