Novarum has released a limited set of its first-half 2007 findings while testing metro-scale Wi-Fi and cell data networks: The rankings are fine, but I'm more interested in what they discovered while performing their tests. They discovered that high-powered Wi-Fi adapters really do make an appreciable difference in providing an improvement in coverage and performance--something that's not always been clear--but were surprised to find that regular-power 802.11n adapters have about 2/3rds of the high-powered radios' advantage in reception and throughput at much lower cost.
Novarum's other key findings are that even more than 40 Wi-Fi nodes per square mile are needed for something that approaches 100-percent "service availability," a term they define as providing the ability to access the network and perform tasks, as opposed to just presence of a signal; and that 3G cell data networks continue to improve on performance - measures of bandwidth - even as they excel in service availability because of client hardware's ability to drop down to 2G service where 3G is unavailable, keeping a seamless connection.
Phil Belanger and Ken Biba, Novarum's founders, date back to the early days of Wi-Fi and wireless networking. They've been through the ringer in several firms, and started testing Wi-Fi networks in 2006 with the idea of selling reports to cities and companies on a spec basis--conducting the studies and then finding buyers for the data--and being hired to perform independent audits or competitive analysis. They decided to start making reports available for a la carte purchase (beginning today), along with continuing to sell subscriptions for their full feed and custom work.
Novarum's methodology involves taking a standard suite of clients that they can test from a set of 20 locations around a city, performing the same measures each time they re-test a city. They drive around, stop at specific locations chosen for particular purposes, and use a standard laptop 802.11g card, a newer but not fancy 802.11n adapter, a 300 mW laptop card with an external 5 dBi antenna mounted on the car from which they test, a Ruckus router, and various cell data modems, among other devices. They test Wi-Fi and cell data to provide apples-to-apples comparisons. Novarum uses four measures, weighting service availability and performance (itself a throughput number weighted towards downstream speeds) more heavily than ease of use and value. (They tested whether using more locations provided more accurate results, and decided it did not change the averages or standard deviation enough to matter.)
In an interview, Belanger said that their testing this year--some of which involved revisiting networks--showed that cellular carriers and metro-scale Wi-Fi networks were both improving, in many cases rather dramatically. (You can view a summary of their findings on their Web site.)
Belanger said that, notably, Philadelphia's EarthLink Feather network had improved dramatically since their last tests when it was just out of its pilot stage. "EarthLink is doing what the new CEO said; they're going to stay in the current footprint and make that work," Belanger said. Novarum measured a 50-percent increase in node density over last year's tests, and saw "much better service availability" in the covered area. Speeds of nearly 1 Mbps downstream aren't unusual. Novarum dubbed the Philadelphia network "most improved."
Mountain View, Calif.'s network, run by Google on a free basis, has also improved tremendously since their last visit in 2006. Belanger said that Google and its hardware provider Tropos were treating the town like a lab, constantly tweaking the network. A strange bit of interference in the 2.4 GHz band in Mountain View has required both companies to adapt. Node density is now at about 43 per square mile, but a combination of other tweaks and firmware upgrades boosted throughput: "The dramatic improvement was in performance," Belanger noted.
I asked about Portland, Ore., the only major city in MetroFi's deployment plans at the moment, and where it was recently revealed by the mayor's office and confirmed by the firm that without a change in the city's commitment or an infusion of venture capital, the network will grow only slighty beyond its current mostly-downtown footprint. Belanger said their testing found poor availability and lots of inconsistency in Portland's Wi-Fi service.
In their first test, they found service availability in Portland with a regular Wi-Fi client in just 30 percent of the locations they tested. (MetroFi publishes a full-disclosure coverage map, so there shouldn't be any ambiguity about live areas of the network.) "We were so disappointed with the results, that we've gone back and tested it again" in the third quarter of 2007, wondering if they had just hit a bad day on the network. But the results were consistent.
With a node density of roughly 30 per square in the areas tested in Portland, Novarum found service "very poor" with a regular client. They also found that the session-based login would time out even when they were actively using the network, requiring a new login. (Belanger noted that Novarum had bid as a service auditor for the city of Portland's evaluation of the network, which another firm won.)
Their single biggest conclusion from testing dozens of networks is that service availability remains metro-scale Wi-Fi's biggest weakness compared with cellular data networks. In the cities they've tested to date, cellular service availability was 100-percent across 15 municipalities, while only the St. Cloud, Flor., network among Wi-Fi services reached that 100-percent mark (requiring a high-powered adapter). On average, cell networks had 87-percent service availability; metro-scale Wi-Fi, 71 percent.
Sprint Nextel and Verizon's EVDO Rev. A upgrade also showed an appreciable improvement in speed in areas they tested before, running 20 to 30 percent faster, and making the difference between metro-scale Wi-Fi speeds, which tend to run below or well below 1 Mbps downstream and 3G cell data networks even less pronounced. The gap in price (Wi-Fi being from free to $20 per month; cell data, $60 to $80 per month) and the lower service availability of Wi-Fi in supposedly covered areas reamining the biggest differentiator. This should move coverage into a more important part of the service matrix as providers decide where to spend money.
In their last report, several months ago, Novarum said "40 was the new 20," meaning that 40 nodes per square mile appeared to be the density at which metro-scale Wi-Fi networks operated well and had wide availability, rather than the 20 nodes per square mile that equipment makers like Tropos once said would work. (Tropos no longer gives that number out.)
Belanger said that through continued testing, they're finding that slightly north of 40 is even better at achieving service availability, and that if a network has good uptake, additional nodes would have the added advantage of keeping throughput high. Belanger said that with the service availability in most cities they tested - remember that 71 percent was the average - "It's just we're not there at that threshold where it appears to be a solid service." He noted, "At about 85% is about the area where it's going to appear like a robust service, you'd actually pay for it."
Even at over 40 nodes per square mile, Belanger said a Wi-Fi network should weigh in at $100,000 to $150,000 per square mile, which he puts as much cheaper than equivalent technologies deployed in cities, especially if spectrum licenses are considered for alternatives, like mobile WiMax. (Costs may vary for both licenses and infrastructure when looking outside cities; one reader said that WiMax-like technology has very competitive costs in more rural environments were licenses are relatively cheap.)
This time around, 9 of the top 10 slots for best overall networks, combining their various measures, were occupied by metro-scale Wi-Fi networks, although most required the use of a high-powered 802.11g adapter to achieve those marks. Those adapters are built into Wi-Fi bridges provided by Ruckus and Peplink, and which service providers recommend.
In comparing availability and performance between a regular adapter and the high-powered one they tested, they found a 36-percent improvement in performance and 38-percent improvement in speed. But their big surprise was in testing a generic, inexpensive, regular-powered 802.11n adapter against a regular 802.11g Wi-Fi radio: performance jumped 20 percent and availability, 26 percent, with 802.11n.
Belanger noted that with performance that high from 802.11n and on an asymmetrical basis--the Wi-Fi networks are still pumping out old 802.11g--that it's a good sign for the future as people move to 802.11n. He said that equipment makers told him the improvement in speed is a bit of a mystery.
Even with "better radio chips, a better antenna, multiple antennas more cleverly used," that there's "probably also other things like the way it does associating and roaming from one AP to the other" that help. In any case, an 802.11n USB adapter for Mac OS X and Windows weighs in at about $60 to $70, while Peplink and Ruckus bridges run $100 to $400.
As they plan for the future, Belanger said that both mobile devices--specifically the iPhone--and mobile WiMax loom large. The company is now testing with an iPhone because the device has built-in seamless handoff between 2G EDGE and Wi-Fi networks. This lets the device have a much better overall service availability figure even when a Wi-Fi network is spotty.
Likewise, pre-WiMax technology like Clearwire and mobile WiMax technology appearing next year will figure into their testing. They've already scanned Chico, Calif., and Eugene, Ore., with Clearwire's current gear.
While there are other firms that audit and test wireless networks, Novarum appears to be the only company revealing as much information publicly; they may be the most extensive testers of Wi-Fi networks, but they're certainly the most frank.