I will probably be laughed at, but I'll ask just in case.
We are having particularly bad luck trying to run VPN tunnels over
Comcast cable in the Chicago area. The symptoms are basically complete
loss of connectivity (lasting minutes to sometimes hours), or sometimes
flapping for a period of time. More often than not, a reboot of the
cable modem is required. The most interesting ones involve the
following: a PIX or ASA configured as an EZvpn client, connecting to a
3000 concentrator, authentication over RADIUS. When I go to look at the
RADIUS logs, I see connections from the same box with small intervals.
Timeout is 8 hours, so theoretically I should see 3 connections in a
24-hr period. In some cases, I see dozens, in the most egregious cases,
thousands over a 24-hour period. I am taking that as an indicator of a
really unstable Comcast circuit. We have not had this problem with any
other ISP, anywhere in the country.
I am pretty much down to telling customers to find another provider...
Any thoughts or ideas on the matter will be appreciated.
PS. To be fair (?) to Comcast, this is not a ubiquitous problem. It
affects about 25% of the installations I get to see.
Sincerely,
Michael Malitsky
We experienced the same thing, and switching from UDP tunnels to TCP tunnels fixed it. There are two things at play here.
1) The SMC modem/router that they insist you use for their "Small Business" cable internet service seems to have trouble with very high rates of non-TCP traffic going through its NAT.
2) Comcast rate limits non-TCP traffic somewhere on their network.
Tunneling TCP inside TCP is a bad idea, but actually made the VPNs useful for us. Using IPSEC or UDP tunnels left us with tunnels that were rate limited to about 1mbps each way, until either the modem crashed or their network throttled us down to near useless speeds. I don't know if they're trying to stop customers from DoS'ing people or... exactly what the goal of it is, and couldn't ever get them to explain anything.
I will probably be laughed at, but I'll ask just in case.
We are having particularly bad luck trying to run VPN tunnels over
Comcast cable in the Chicago area. The symptoms are basically complete
loss of connectivity (lasting minutes to sometimes hours), or sometimes
flapping for a period of time. More often than not, a reboot of the
cable modem is required. The most interesting ones involve the
following: a PIX or ASA configured as an EZvpn client, connecting to a
3000 concentrator, authentication over RADIUS. When I go to look at the
RADIUS logs, I see connections from the same box with small intervals.
Timeout is 8 hours, so theoretically I should see 3 connections in a
24-hr period. In some cases, I see dozens, in the most egregious cases,
thousands over a 24-hour period. I am taking that as an indicator of a
really unstable Comcast circuit. We have not had this problem with any
other ISP, anywhere in the country.
I am pretty much down to telling customers to find another provider...
Any thoughts or ideas on the matter will be appreciated.
PS. To be fair (?) to Comcast, this is not a ubiquitous problem. It
affects about 25% of the installations I get to see.
Sincerely,
Michael Malitsky
We experienced the same thing, and switching from UDP tunnels to TCP tunnels fixed it. There are two things at play here.
1) The SMC modem/router that they insist you use for their "Small Business" cable internet service seems to have trouble with very high rates of non-TCP traffic going through its NAT.
If you have business class service, insist that they put the cablemodem in BRIDGE-ONLY mode. This will resolve this issue and eliminate the unnecessary NAT.
2) Comcast rate limits non-TCP traffic somewhere on their network.
Comcast rate limits traffic in general. TCP is not less rate limited than anything else in my
experience.
Owen
I ran into issues in various Comcast serviced regions with SSL VPN over tcp-443. From testing we started getting drops or severe rate limits on the flow after 7-10 minutes. Best guess was it was anti-p2p systems throttling encrypted/unknown protocol traffic after a set timer. Disconnecting and reconnecting pushed performance back up to normal until the timer kicked in again. We ended up setting the SSL tunnel to re-key via new sessions every 5 minutes to keep the flow shorter then the observed timer intervals. Other then running into a Cisco AnyConnect client bug (the app would steal focus at the re-keys) worked around the issue on Comcast and even some FiOS end users.
http://ckdake.com/content/2008/disable-gateway-smart-packet-detection.html
showed a feature of "Gateway Smart Packet Detection" in some SMC
cable modem.
The current solution is to identify the affected Comcast modem, and ask
Comcast engineer to turn that "IDS" feature off remotely.
I spend several days to talk with comcast about our blackboard will
not work sometimes in some shared business class residential building.
Finally got hold of a Regional Engineer to confess this with my
tcpdump proof. Local comcast engineer may not be aware of this
feature.
Schilling
You can get into the SMC device yourself by going to the http://10.1.10.1/login.asp link on the SMC. The username/password are well known as cusadmin/highspeed. I also recommend against using the integrated services in the device if at all possible. It's also mildly annoying that it does not respond to traceroute either when it's your gateway with a pool of static ips.
I did have one case where it reverted to a mode where it ran dhcp/nat but that was shortlived and has not happened again.
- Jared
In June of last year, when Comcast did firmware updates on the business gateways in the MSP area, we lost all (3) of our sites with Netgear gateways, but not the sites SMC gateways (the management interface is almost identical, but the brand is marked on the modem). Business support was apparently aware of a Cisco VPN problem through the Netgear, and simply replaced the Netgear units with SMC, and we haven't had issues since. This is IOS to ASA site-to-site VPN.
Mark Mayfield
City of Roseville
Network Systems Engineer
2660 Civic Center Drive
Roseville, MN 55113