Glass and Copper (and aluminum) infrastructure is a natural monopoly, similar to water service.
It was purely by chance IMHO that we ended up with Cable Co and Tel Co internet competing with each other in many locations in the US.
That was aided by the following:
Technology for TV over telephone wire really did not exist at the time
Telcos were not very interested in PayTV at the time
Technology for Telephone over Coax really did not exist at the time
Cable Co’s were not very interested in Telephone service at the time
Basically they were viewed as two very different businesses, with very different physical plant needs. Now both of them are primarily fiber based, with Coax or Telephone Wire (in many cases you can not even call it twisted pair) as the very last interconnect segment.
We can all agree with hind sight (and a lot of us at the time) that the Tel Co’s made some very stupid decisions. Perfect example being installing remote DLC/SLC units when the demand for analog dial tone skyrocketed, along with more copper in the ground/on poles in neighborhoods. At first this blocked ADSL deployment until remote DSLAMs were installed, then it turns out most were NOT close enough to enable VDSL2 or g.FAST for the majority of customers serviced by them. They were both “in the way” and “too far away” at the same time. If instead of the DLC/SLC units the Tel Cos had instead favored (with the correct tariffs) moving any residential customer who requested a second POTS line to ISDN BRI, they would have saved all of the physical plant work, which has turned out to be a horrible investment.
We learned a long time ago that water lines, sewer lines, and electric lines were natural monopolies, and should either have a municipal granted license, or should be run by the municipality.
The next generation last mile will almost have to be a similar structure for Layer 1 and a form of Layer 2, with Layer 3 and above services being sold by anyone who wants to provide the service. This will collapse Cable Co, Tel Co, and independent ISPs onto the same physical infrastructure. This will work well for dense locations of course.
Wireless ISPs, and LEO based ISPs will still of course have a major role to play for at least several decades if not more.
I also agree entirely that most consumers will “pay the ISP too much” for service they “don’t need”. I have worked with several people who were paying for Gigabit Cable Service, with 30Mbit upload, or in Spectrum territory, they had 400Mbit service with 20Mbit upload, and the “downgrade” was 200Mbit service with 10Mbit upload. Being as that was a single individual with very low upload needs beyond video meetings, I recommended he downgrade to the 200/10 service. In all cases, a proper WiFi network and wireless offloading has made far more difference vs upping the cable co speeds. My personal sweet spot right now is 100/20 business cable or 100/100 small business fiber (for the few spots that have GPON service in Tucson). The next tier of business cable is 200/20, and I find the extra 100Mbit download really does not change much. If it was 200/30 or 200/40, I would probably consider it.
None of the realities of current “needs” and “wants” really are going to change the financial need to consolidate physical networks. Unfortunately instead of it being a Layer1/2 provider and L3+ competition, most Internet networks in new developments around here are being deployed as physical layer and service monopolies. The home builder will make an alliance with Cox, Comcast, or CenturyLink, and then the others will not build out physical plant in the community.