Novarum has released a limited set of its first-half 2007 findings while testing metro-scale Wi-Fi and cell data networks: The rankings are fine, but I'm more interested in what they discovered while performing their tests. They discovered that high-powered Wi-Fi adapters really do make an appreciable difference in providing an improvement in coverage and performance--something that's not always been clear--but were surprised to find that regular-power 802.11n adapters have about 2/3rds of the high-powered radios' advantage in reception and throughput at much lower cost.
Novarum's other key findings are that even more than 40 Wi-Fi nodes per square mile are needed for something that approaches 100-percent "service availability," a term they define as providing the ability to access the network and perform tasks, as opposed to just presence of a signal; and that 3G cell data networks continue to improve on performance - measures of bandwidth - even as they excel in service availability because of client hardware's ability to drop down to 2G service where 3G is unavailable, keeping a seamless connection.
Phil Belanger and Ken Biba, Novarum's founders, date back to the early days of Wi-Fi and wireless networking. They've been through the ringer in several firms, and started testing Wi-Fi networks in 2006 with the idea of selling reports to cities and companies on a spec basis--conducting the studies and then finding buyers for the data--and being hired to perform independent audits or competitive analysis. They decided to start making reports available for a la carte purchase (beginning today), along with continuing to sell subscriptions for their full feed and custom work.
Novarum's methodology involves taking a standard suite of clients that they can test from a set of 20 locations around a city, performing the same measures each time they re-test a city. They drive around, stop at specific locations chosen for particular purposes, and use a standard laptop 802.11g card, a newer but not fancy 802.11n adapter, a 300 mW laptop card with an external 5 dBi antenna mounted on the car from which they test, a Ruckus router, and various cell data modems, among other devices. They test Wi-Fi and cell data to provide apples-to-apples comparisons. Novarum uses four measures, weighting service availability and performance (itself a throughput number weighted towards downstream speeds) more heavily than ease of use and value. (They tested whether using more locations provided more accurate results, and decided it did not change the averages or standard deviation enough to matter.)
In an interview, Belanger said that their testing this year--some of which involved revisiting networks--showed that cellular carriers and metro-scale Wi-Fi networks were both improving, in many cases rather dramatically. (You can view a summary of their findings on their Web site.)
Belanger said that, notably, Philadelphia's EarthLink Feather network had improved dramatically since their last tests when it was just out of its pilot stage. "EarthLink is doing what the new CEO said; they're going to stay in the current footprint and make that work," Belanger said. Novarum measured a 50-percent increase in node density over last year's tests, and saw "much better service availability" in the covered area. Speeds of nearly 1 Mbps downstream aren't unusual. Novarum dubbed the Philadelphia network "most improved."
Mountain View, Calif.'s network, run by Google on a free basis, has also improved tremendously since their last visit in 2006. Belanger said that Google and its hardware provider Tropos were treating the town like a lab, constantly tweaking the network. A strange bit of interference in the 2.4 GHz band in Mountain View has required both companies to adapt. Node density is now at about 43 per square mile, but a combination of other tweaks and firmware upgrades boosted throughput: "The dramatic improvement was in performance," Belanger noted.
I asked about Portland, Ore., the only major city in MetroFi's deployment plans at the moment, and where it was recently revealed by the mayor's office and confirmed by the firm that without a change in the city's commitment or an infusion of venture capital, the network will grow only slighty beyond its current mostly-downtown footprint. Belanger said their testing found poor availability and lots of inconsistency in Portland's Wi-Fi service.
In their first test, they found service availability in Portland with a regular Wi-Fi client in just 30 percent of the locations they tested. (MetroFi publishes a full-disclosure coverage map, so there shouldn't be any ambiguity about live areas of the network.) "We were so disappointed with the results, that we've gone back and tested it again" in the third quarter of 2007, wondering if they had just hit a bad day on the network. But the results were consistent.
With a node density of roughly 30 per square in the areas tested in Portland, Novarum found service "very poor" with a regular client. They also found that the session-based login would time out even when they were actively using the network, requiring a new login. (Belanger noted that Novarum had bid as a service auditor for the city of Portland's evaluation of the network, which another firm won.)
Their single biggest conclusion from testing dozens of networks is that service availability remains metro-scale Wi-Fi's biggest weakness compared with cellular data networks. In the cities they've tested to date, cellular service availability was 100-percent across 15 municipalities, while only the St. Cloud, Flor., network among Wi-Fi services reached that 100-percent mark (requiring a high-powered adapter). On average, cell networks had 87-percent service availability; metro-scale Wi-Fi, 71 percent.
Sprint Nextel and Verizon's EVDO Rev. A upgrade also showed an appreciable improvement in speed in areas they tested before, running 20 to 30 percent faster, and making the difference between metro-scale Wi-Fi speeds, which tend to run below or well below 1 Mbps downstream and 3G cell data networks even less pronounced. The gap in price (Wi-Fi being from free to $20 per month; cell data, $60 to $80 per month) and the lower service availability of Wi-Fi in supposedly covered areas reamining the biggest differentiator. This should move coverage into a more important part of the service matrix as providers decide where to spend money.
In their last report, several months ago, Novarum said "40 was the new 20," meaning that 40 nodes per square mile appeared to be the density at which metro-scale Wi-Fi networks operated well and had wide availability, rather than the 20 nodes per square mile that equipment makers like Tropos once said would work. (Tropos no longer gives that number out.)
Belanger said that through continued testing, they're finding that slightly north of 40 is even better at achieving service availability, and that if a network has good uptake, additional nodes would have the added advantage of keeping throughput high. Belanger said that with the service availability in most cities they tested - remember that 71 percent was the average - "It's just we're not there at that threshold where it appears to be a solid service." He noted, "At about 85% is about the area where it's going to appear like a robust service, you'd actually pay for it."
Even at over 40 nodes per square mile, Belanger said a Wi-Fi network should weigh in at $100,000 to $150,000 per square mile, which he puts as much cheaper than equivalent technologies deployed in cities, especially if spectrum licenses are considered for alternatives, like mobile WiMax. (Costs may vary for both licenses and infrastructure when looking outside cities; one reader said that WiMax-like technology has very competitive costs in more rural environments were licenses are relatively cheap.)
This time around, 9 of the top 10 slots for best overall networks, combining their various measures, were occupied by metro-scale Wi-Fi networks, although most required the use of a high-powered 802.11g adapter to achieve those marks. Those adapters are built into Wi-Fi bridges provided by Ruckus and Peplink, and which service providers recommend.
In comparing availability and performance between a regular adapter and the high-powered one they tested, they found a 36-percent improvement in performance and 38-percent improvement in speed. But their big surprise was in testing a generic, inexpensive, regular-powered 802.11n adapter against a regular 802.11g Wi-Fi radio: performance jumped 20 percent and availability, 26 percent, with 802.11n.
Belanger noted that with performance that high from 802.11n and on an asymmetrical basis--the Wi-Fi networks are still pumping out old 802.11g--that it's a good sign for the future as people move to 802.11n. He said that equipment makers told him the improvement in speed is a bit of a mystery.
Even with "better radio chips, a better antenna, multiple antennas more cleverly used," that there's "probably also other things like the way it does associating and roaming from one AP to the other" that help. In any case, an 802.11n USB adapter for Mac OS X and Windows weighs in at about $60 to $70, while Peplink and Ruckus bridges run $100 to $400.
As they plan for the future, Belanger said that both mobile devices--specifically the iPhone--and mobile WiMax loom large. The company is now testing with an iPhone because the device has built-in seamless handoff between 2G EDGE and Wi-Fi networks. This lets the device have a much better overall service availability figure even when a Wi-Fi network is spotty.
Likewise, pre-WiMax technology like Clearwire and mobile WiMax technology appearing next year will figure into their testing. They've already scanned Chico, Calif., and Eugene, Ore., with Clearwire's current gear.
While there are other firms that audit and test wireless networks, Novarum appears to be the only company revealing as much information publicly; they may be the most extensive testers of Wi-Fi networks, but they're certainly the most frank.
While we aren't a company per se, Caleb Phillips and I did a pretty thorough job of testing outdoor coverage of Portland's MetroFi network. I haven't seen a more reliable methodology than the one we used. Our testing covers only one city, so city-to-city comparisons aren't possible, but as for "more frank", I don't see how anyone can beat us. We published (or are willing to) everything we collected and how we collected it. I have yet to hear any serious technical criticism of how we did our testing.
[Editor's note: The fact that it's one city makes it most interesting to people within Portland, I suspect, rather than anything being wrong or uninteresting with the work you did. Novarum, by testing many cities, can put Portland's service into context, allowing a broader apples-to-apples (iPhones to iPhones?) comparison.
Novarum has the possible bias of not being selected to be paid to perform an audit; Personal Telco has the potential bias of being opposed to Portland's plan that brought MetroFi. Neither potential for bias appears to me to have affected outcomes. -gf]
As a member of the Board of Directors and Secretary of the Personal Telco Project, I can say that PTP as an organization does not oppose Portland's plan that brought MetroFi. Various members have publicly expressed various opinions about it, but the vast majority of the criticism has been on their execution.
I participated in the UnwirePdx Watch investigation *not* because I was opposed to MetroFi, but because I wanted to see how they were doing. Furthermore, we selected a methodology (simple random sampling) that did not allow bias, so that whatever our unconscious feelings might have been, they could not have affected our results. Also, we should be clear that when Caleb and I did our study, we were *not* doing it as a Personal Telco study, but on our own. Personal Telco as an organization did not support or direct us in or even know about our investigation.
I agree that multi-city surveys are interesting. Also, I appreciate that Novarum did multi-client testing, because of the obvious dependency on client signal for success in achieving connection. We weren't able to, instead choosing a client based on the language of the RFP. It was primarily the "most frank" part I was reacting to, because I think we've been pretty darn frank.
[Editor's note: I see. The "most frank" remark was in the context of other firms that do testing, but frank is an antonym of "not forthcoming." Other testing firms aren't releasing information. There are two other firms that do some Wi-Fi network testing (Uptown and another), but there are plenty of other companies--Lockheed, for one--that are doing cell and Wi-Fi testing and it's all private.
So "most frank" here means that they're a) a firm devoted to this that's b) releasing lots of information publicly about their process and results --gf]
What is the value of frankness if no one is holding testing firms to account for their methodology? Uptown quite frankly shared that they used a hardware array that was the equivalent of a Ruckus mounted outside a van when they did their Portland testing. While I don't doubt that their data is accurate, that test rig is neither representative of the type of hardware a mobile user would have nor the placement that a Ruckus for an indoor user, making the data and their analysis meaningless from the standpoint of an end-user.
"Personal Telco has the potential bias of being opposed to Portland's plan that brought MetroFi."
Glenn, this is not a true statement. I don't believe Personal Telco was ever biased. I do believe that there was a general opinion "we'll believe it when we see it". And guess what. I don't think we see it yet. Also, our participation in the planning was only community based, showing up at public planning meetings and asking questions. We had no intention of making the plan or changing the plan. We just wanted to know what the plan was going to be and hopefully share some of our knowledge to the community.
I believe you have a "biased" view of The Personal Telco Project. Do you feel like we're trying to compete with a) metrofi or b) the city's plan?
[Editor's Note: I have had so much communication with folks at Personal Telco, and have read so many statements over the many months since Portland released its RFP and then chose MetroFi about the concerns over the way the plan was assembled, its details, and the vendor, that it's hard for me to countenance the notion that Personal Telco wasn't per se opposed, even if that wasn't an official stance. Skepticism seemed very shaded into opposition.
But fair enough: You guys own your position and your words, and I'll be sure in the future to not characterize Personal Telco's position as once opposed to the network, but rather Missouri (show me). I'll leave my original remark in place since we have this record in the comment that provides more light on the subject. Thanks! -gf
Glenn,
Since many of your exchanges have been with me, I'd like to clarify that most of those exchanges involved debunking the oft-repeated claim that Personal Telco was passed over for Unwire Portland, that Personal Telco was involved in the testing and other claims that Personal Telco was in some way snubbed in the planning and execution of Unwire Portland. That's an issue of fact or fiction. It doesn't represent or prove any bias or opposition, it was just clearing up a frequent error.
In the course of addressing the erroneous claim that PTP was biased and unreliable, I pointed you to many points in the public record that supported my own skepticism. This does not constitute unfounded opposition (which you seem to imply).
Let's clear up a point about bias:
I am personally, and I expect others are as well, biased against concepts that my experience suggests won't work. So, yes, I am biased against some components of the MetroFi business plan and the original Unwire Portland RFP. That should have been clear since those statements were obviously opinions and opinions will necessarily reflect a personal rather than universal experience. The question is whether those opinions were informed or off-base. It would seem they were fairly accurate, since:
a) no parking meters are served by the MetroFi deployment that won the RFP.
b) the testing you like best (as well as the testing I like the best) have found the MetroFi deployment to be far short of the 95% goal.
c) the ad-supported model does not seem to be financially viable.
d) no-one has presented any evidence to discount our criticisms, they have instead thrown their hands up and screamed "Bias!" or resorted to attacks on our credibility.
Lest we be characterized as nay-sayers who would do or say anything to bring down the Unwire Portland project, I want to point out that Personal Telco is actively working with the City to find ways that we can help address the public benefit aspects of the Unwire Portland project. We believe that a lot of good ideas informed Unwire Portland and the success or failure of MetroFi should not dictate which of those ideas see the light of day.
[Editor's Note: I do think there's a misunderstanding here when I talk about bias, but I appreciate the rest of the clarification. When you write, "This does not constitute unfounded opposition," I hope I have never categorized the opposition as unfounded. -gf]