As an internet service provider with many small business and residential customers, our most common tech support calls are speed related. Customers complaining on slow speeds, slowdowns, etc.
We have a SNMP and ping monitoring platform today, but that mainly tells us up-time and if data is flowing across the interface. We can of course see the link speed, but customer call in saying the are not getting that speed.
We are looking for a way to remotely test customers internet connections besides telling the customer to go to speedtest.net, or worse sending a tech out with a laptop to do the same thing.
What opensource and commercial options are out there?
We ask our customers use iperf3 to test speed.
Good luck with that if their only devices are tablets, phones, and Rokus?
You could optionally run tools such as iperf[123] on the CPE device
directly. If you can install it on the CPE, that is.
Alternative is the classic of doing a L2 loop with dedicated VLANs on
the CPE. E.g. Server1 -> VlanX -> CPE -> VlanY -> Server2 in your own
network.
-Christoffer
Might be worth while to get some graphing on customers max transmission speeds over the period of three days, a week, two weeks, month to better predict what they may be seeing so you can better predict the area’s that could be effected due to whatever causes.
A lot of times I find this comes down to name resolution as where to the customer it looks slow but is more likely being drowned out by other traffic or slow responses from the name servers them self that traverse <yournet>. But those are just common causes. Prioritizing traffic will greatly depend on the information you are seeing and a root cause will greatly evade you just doing speed tests.
MRTG, rrdtool and some others can accomplish this for you.
Good luck
Last time I setup Iperf3 it was semi difficult, and would be impossible trying to coach a soccer mom on how to setup over the phone.
I am leaning towards a CPE that has speed test built in, or a low cost, sub $100 device we could ship to the customer to install. Anyone know of something like that?
Would a raspberry pi work for this?
Could 3D print a nice case with your logo for it.
I don’t think a raspberry pi will reliably fill a full Gig and keep it full (maybe that’s not required in this scenario), but I’ve installed a Linux based OS with the PerfSONAR tools (including iperf) on a couple of different mini PCs in the “few hundred dollars” price range.
The last one was the Liva X from ECS. It was more than capable of filling 1G circuits with traffic and keeping them full without loss or wonky results due to things like CPU overrun or other processes causing bus contention. I’m pretty sure the Liva X is retired now, but their current gen should suffice as should a number of comparable competitors.
Maybe try setting up an Ookla on-site speedtest server? I believe the product is called Speedtest Custom. Setup is pretty simple, and is relatively inexpensive.
That gives you the ease-of-use of speedtest.net, with the accuracy similar to having a local iperf server.
Zach Puls
Network Engineer
MEF-CECP
![Title: KsFiberNet - Description: Macintosh HD:Users:chunt:Documents:Logos_Brand Guide_KFN:WEB_logos:KFN_logo-WEB.png]()
Direct: +1 (316) 221-2094
Email: zpuls@ksfiber.net
Technical Support: 855-KFN-HELP (536-4357)
This e-mail, including any attachments, is intended only for the person or entity to which it is addressed and may contain confidential, proprietary and/or privileged material. Any review, retransmission, dissemination or other use of this information, directly or indirectly, by persons or entities other than the intended recipient is prohibited and may subject you to legal liability. If you received this in error, please contact the sender and delete the material from all computers in which it resides. This email, including any attachments, is not intended to constitute the formation of a contract binding KsFiberNet. KsFiberNet will be contractually bound only upon execution, by an authorized representative, of a definitive agreement containing agreed upon terms and conditions.
A Raspberry Pi uses USB 2 for Ethernet interconnection to the CPU so it most definitely will not keep even half a gig full. It’ll do a bit over 300 Mbps.
Ryan Wilkins
In the US this type of testing may be an actual requirement for some ISP’s if they get funding from the government.
Depending on the Bandwidth needed, yes, but the Pi is limited at the NIC level because it is on a shared USB 2.0 Bus.
I investigated building a product that could reliably speedtest up to a gig and found the same thing. A raspberry Pi 3B or 3B+ can reliably test up to ~100Mbps. The 3B only has a 10/100 NIC; The 3B+, while having a gigabit NIC, tops out at ~300Mbps internally. Both models of the Pi are available as a kit that retails under $100. For testing up to 1Gbps, an x86 mini PC like those sold for firewall appliances () are available and retail for $200-$300 at the low end. My conclusion was that doing testing within the CPE was the most cost effective way to go. One should keep in mind that even gigabit CPEs may not be able to reliably test > 100Mbps due to CPU or other software limitations. Public speedtest servers may also not reliably test > 100Mbps, so for reliable gigabit testing you’ll need to run your own.
The Pi has a bandwidth limit at 300mbits/sec due to a USB port being used.
I wonder if something like the RIPE Atlas probes could be flashed with suitable
code. They're smaller than a Pi, and easy to set up - connect a USB power cord
and an RJ45 on some cat-5 and away you go. Mine showed up with the two cords
needed and everything.
https://www-static.ripe.net/static/rnd-ui/atlas/static/docs/probe-images/v1.jpg
So out of curiosity - does anybody have info on what percentage of residential
internet connections are on CPE that's been suitably de-bufferbloated?
I have not read anything suggesting that de-bufferbloating has happened. It would be nice to see something.
I participated in bufferbloat testing on Comcast Business Internet some years ago and have never seen any results published.
James R. Cutler
James.cutler@consultant.com
GPG keys: hkps://hkps.pool.sks-keyservers.net
Hi Colton,
In the past I have used CPEs which support remote loopback. When the
customer complains we enable remote loopback, send the traffic to that
customers connection (rather than requiring a CPE that can generate
the traffic or having an on site device) and measuring what comes
back.
Cheers,
James.
All, thanks for the recommendations both on and off list.
It has been brought to my attention that a Mikrotik has a bandwidth speed test tool built into their operating system. Someone recommended a https://mikrotik.com/product/hap_ac2 for MSRP of $69. The release notes of the newest version say:
!) speedtest - added “/tool speed-test” for ping latency, jitter, loss and TCP and UDP download, upload speed measurements (CLI only);
*) btest - added multithreading support for both UDP and TCP tests;
Do you think this device can push a full 1Gbps connection? It does have a quad core qualcom processor.
Besides mikrotik, I haven’t found anything that doesn’t require me to build a solution. Like OpenWRT with ipef3, or something like that.
Seems like a commercial solution would exist for this. I though CAF providers have to test bandwidth for the FCC randomly to get funding?
Just download the btest.exe
It run on windows PC.
Most routerboards not fast enough for TCP test as TCP packet assembly is intensive.
There’s a Montreal startup called Obkio who are doing network probes (VM and hardware). I’ve tested the product in its early phase (i.e. it was lacking features that are now implemented or are going to be implemented soon).
They recently launched the speed test feature: https://medium.com/obkio/app-new-release-v1-6-0-public-agents-support-chat-and-speed-tests-c84651f7008a and launched a beefier probe called X5001 which can supposedly do 940Mbps: https://medium.com/obkio/new-hardware-agent-x5001-the-10x-agent-b278e435c458
I think it’s worth a look.
Disclaimer: the CEO is an acquaintance of mine
Eric