New measurement firm joins two others that aim to audit performance of metropolitan-scale Wi-Fi networks: Wi-Fi veterans Phil Belanger and Ken Biba today launch Novarum, a firm that will produce 10 reports per quarter on how Wi-Fi networks that span cities and counties measure up. Novarum joins a field that's not yet crowded, but has at least two competitors I spoke with that make measurement part of larger businesses. Each of the three firms has a distinct approach to taking stock of these new networks.
For some time, I've been banging the drum of network performance audits for muni-Fi, because it makes little sense for a city or civic group to ask the same group that they paid or allowed to design, build, and operate the network to also provide guidelines for evaluating that network's performance. A disinterested third party with no financial stake in the outcome of a deployment should look at tests, pilot projects, and production rollouts to determine whether coverage and performance meets the contracted specifications.
I spoke with Belanger of Novarum, and the heads of Unplugged Cities and Uptown Services to learn about their methodology in measuring Wi-Fi networks of a scale that only came into being in the last year.
(Follow the link below for the rest of the article.)
Belanger helped bring Wi-Fi into being. His name is on the 802.11b specification, and he was an early head of the Wi-Fi Alliance, partly responsible for the Wi-Fi name. Belanger has worked for several notable firms, including Wayport, Vivato, and, most recently, BelAir. His partner in this enterprise, Ken Biba, was an early engineer of basic Internet protocols in the 1970s, and guided a number of pioneering wireless firms, most recently as the head of Vivato.
(While Vivato got a black eye from its ultimate inability to deliver a product that conformed to its early promise, both Belanger and Biba left the firm years before its collapse, at the point at which, according to private and public sources, the firm picked the wrong future course.)
Belanger said he and Biba set out to determine whether Wi-Fi networks could deliver on the metro-scale promise. His high-level marketing work at BelAir, a maker of equipment for that market, gave him insight into the overall product picture, but with many vendors and many service providers building networks, Belanger wanted to see what real networks looked like.
"If you compare it to broadband that you have in your house, if you compare to broadband in public places, how does it stack up?" he said, given that many networks are being advertised as broadband alternatives, typically as dial-up replacements, but sometimes as wireline DSL or cable replacements or backups.
Novarum's methodology involves significant wardriving across a city to determine the heat map of Wi-Fi. Belanger said they chose to measure both third-generation (3G) cellular performance and Wi-Fi performance, looking at 3G within the defined area that Wi-Fi was supposed to operate in cities they tested.
What Novarum discovered is that despite Wi-Fi's much higher potential capacity--a raw data rate of 54 Mbps and a net rate of perhaps mid-20s--3G networks and Wi-Fi networks delivered about the same throughput of several hundred kilobits per second downstream, with some large variability in coverage areas.
Novarum tested four cities initially: Anaheim, Galt, Palo Alto, and Santa Clara, Calif. Belanger said he and Biba, the hands-on operators in these early tests, chose 20 locations in a city after a wardrive to do more extensive measurement using IxChariot, a well-regarded tool for delivering bandwidth over a network and measuring results. One end point for the test was in the gear they traveled with. The other, outside of the provider's network, elsewhere on the Internet. This allowed them to measure as-delivered Internet throughput, but not to test, say, video quality across the local network or other local services.
The firm tested networks using average client equipment, including laptops and smartphones. They'll increase the suite of devices they test with, and add authentication testing in the future, too. Belanger noted that smartphones might perform better over a 3G network with less bandwidth than a laptop over a higher-speed Wi-Fi networks because the phones' applications are tuned for less throughput and less interaction.
The results surprised Belanger, he said. First, the consistent availability of true 3G cellular was highly variable across the areas they tested, with frequent dropdowns to 2G and 2.5G speeds. In three cities, only one provider in each location achieved 100-percent 3G coverage in the Wi-Fi area; in Galt, Verizon hit 100 percent, but none of it was 3G coverage. In a more urban city, they found just 34% of tested locations had 3G coverage from the best provider. "We thought that 3G was more ubiquitous; it's not."
And while Cingular had the least coverage in the cities they tested, their HSDPA technology "seems to be the one that really works the best. It's just not very available. But when it is available, it works very well," Belanger said.
On the Wi-Fi side, Belanger said that the numbers of nodes they're seeing per square mile is typically 30 or higher, with the exception of Galt, which had an outdoor coverage mandate--"paint the outside of the building"--and has about 12 nodes per square mile. He said that about two years ago when Tropos, as the dominant vendor, was suggesting as few as 20 nodes per square mile, their competitors also suggested that number. However, the industry now recognizes and recommends 30 or more; EarthLink now typically talks about 35 nodes per square mile.
To Belanger's surprise, he found that regardless of vendor or service provider, Wi-Fi networks of this scale tended to work reasonably to quite well, even at this early stage in rollouts. Belanger used to be skeptical about eqiupment from vendors other than his former employer, BelAir. No longer.
"One of the things that was validated by this first phase of the market is that -- yes, Wi-Fi can be used as a technology to do this," he said. "I am fairly enthusiastic and positive about" Wi-Fi's use for large-scale networks. "It's not about its technical superiority for" that purpose, he said. "The fact that we're using [Wi-Fi] is driven by the economics" of cost for clients and infrastructure. That is, faster networks could be offered if it made financial sense to do so.
"The networks that we've tested, they're not running at the limits of the Wi-Fi performance; they're running at the limits of the economic justification for doing them," Belanger said. Belanger said there was no way to tell the difference in testing between networks that used throttling and those that performed at lower levels.
One interesting "network" that Novarum tested was Palo Alto, which Belanger said has no official citywide network. Rather, he and Biba tested the lily-pad notion, in which enough public and private open access points could provide relatively contiguous access, although not with handoffs. (Not all metro-scale networks offer seamless handoffs, either, they found.) The service availability measure Novarum uses--not just seeing a signal, but being able to accomplish a task over the Internet--in Palo Alto was only slightly below the next lowest performing network they looked at.
Unintentionally open access points are becoming a thing of the past, Belanger said, something that's been confirmed by a number of other sources that scan large numbers of APs in metropolitan areas. Belanger said that they found that 60 to 65 percent of APs they saw were secured, with about half of the remaining networks clearly designed for public access.
Belanger said that service availability was clearly something that should concern service providers. MetroFi's Santa Clara network had just 55 percent service availability in Novarum's testing, while EarthLink scored the highest with 72 percent in Anaheim tests. Santa Clara's network is one of the largest older networks, and was established without municipal involvement. Anaheim is one of the newest networks, and was bid out by the city to EarthLink; they tested the network after its first phase was deployed and tested only the stated working area.
Novarum plans to offer paid subscriptions for its blind studies to analysts and others that want the 10 reports that they produce each quarter, but they will also be offering marketing, troubleshooting, and other consulting to municipalities and service providers, as well as custom audits.
Belanger said the firm would move into testing VoIP as service providers start promoting voice over Wi-Fi as an offering. He said that as they are contracted directly by cities or service providers, they would almost certainly test from Wi-Fi adapters through to end points within the network to better identify bottlenecks.
Based in the Twin Cities region of Minnesota, Unplugged Cities' most recent press has to do with their design of a network for St. Louis Park, an area suburb; the city just gave its go-ahead for the network, which will be municipally funded and owned. The design and operation of Wi-Fi networks is part of this firm's purview, although they have developed a measurement practice because of the obvious need for that sort of service. (In St. Louis Park, they are the local contractor for ARINC, a multinational firm that had the winning bid.)
Chad Peterson, the head of the firm, said that Unplugged Cities works closely with the network's owner, which has included cities for which they have built networks. They ask, "What do you want to measure and what are you looking to get out of the network?" said Henry Camacho, director of technology for the company. This can include tasks like delivering streaming video from crime scenes to a headquarters or pushing voice applications over Wi-Fi, especially in cities that combine public safety and public access on the same networks.
Camacho said that the company develops a list of the most challenging locations on a network to be tested, as well as some places that the service provider believes it has its best performance. They place hardware at the egress point, where traffic from a mesh cluster or node leaves the public access spectrum to provide testing across the local link. "I always do end up going to some access points that the vendor doesn't want me to" in order to get the best picture of the network's performance.
The company doesn't release results, Peterson said, except to their customer and the service provider as they don't want to single out vendors or providers; they also prefer to avoid publicizing faults of metro-scale equipment makers. "That's one reason why we try not to divulge the access point providers that we've tested on or the cities or networks we've tested on," Peterson said. We're not trying to give a head's up to one company or not. We're trying to test the network as a whole. We're trying to provide objective results."
Unplugged Cities doesn't do the kind of blind testing that Novarum expects to release in great quantities. "When you test that, going in blind, it's not really effective," Camacho said. "I prefer to create the relationship with the vendor, get the testing end points into the right spot in the network--then have a collaborative process to deploy the testing."
The company hasn't found any great surprises in the several cities' networks they tested so far, but note that they believe the number of nodes necessary to achieve complete coverage is far higher than typically cited. In St. Louis Park, they believe they may require 50 nodes per square mile due to the dense canopy of trees in that city.
But Peterson notes, "We shy away from just saying, or generalizing the number of access points per square mile. There really is no average deployment. Every city is different, and inside of every city, they have all their own individual areas that react differently."
Neil Shaw started working with municipalities on broadband back in 1998, when fiber optic and coaxial were the only reasonable choices. His company has consulted on several fiber to the home (FTTH) projects in smaller cities and towns, where the scale and nature of FTTH makes sense for moving a city from mid-20th to early 21st century networking. In the last three years, they've helped four FTTP networks in Oklahoma and Tennessee come into being.
Shaw found that the rise of municipal wireless carried with it the need by cities he'd worked with for an independent evaluation of pilot projects and deployed performance. Shaw's firm doesn't design or build wireless networks. This year, he worked with a developer to create a measurement package that would allow him to validate networks. The service is called, cleverly, WiVeriFi.
For sampling, Shaw drove every street in three cities in California: Lompoc, Mountain View, and Sunnyvale. Shaw said that his firm would provide every-100-foot metrics of throughput, latency, and coverage and turn that into a geographical color-coded heatmap for visualization.
His testing methodology involves tracking metrics by retrieving the gateway page or authentication page on the local network. Shaw might test beyond that point in the future, but he thought this was the only reliable measure of the local link without access to the network. "I'm guessing the bottlenecks are going to be on the wireless side, not on the Internet side," he said. While his firm might carry out blind testing, he expects the majority of business to involve cooperation with the service provider.
As both other firms interviewed for this article noted, measuring indoor performance will be a bear, because it involves gaining access to typical locations, often in residential neighborhoods to perform testing. Shaw was involved in third-party validation of Metricom's Ricochet network where they had to test 500 indoor points, so he's well familiar with the drawbacks and complications. "We are going to be developing an indoor testing scheme, both for cooperative and for blind," he said, because of the utility of that information.
While Shaw remained close to the vest about particular results, he did allow that Lompoc's network performed well. Across three cities, he found performance ranging from 273 Kbps to 1.1 Mbps and coverage from 69 percent to 90 percent. In the 90-percent covered city, coverage was consistent in areas, but in the lowest-covered town, consistency of coverage was "all over the place."
"The results of our testing of the three systems that we tested validated the need for this testing," Shaw said. Meaning, I have to conclude, that he felt the level of service provided wasn't up to what he would recommend as acceptable were he evaluating those networks for his clients.