Reflections on the Early Days of Los Nettos
by Walter W. Prue III
T1's
We got quotes from PacBell and GTE for the T1 lines. We developed minimal cost topologies, star topologies, and finally settled on a ring topology, which meant that everyone paid for one T1 to interconnect and we bought robustness for a relatively small monthly cost. When Trusted Information Systems (TIS) decided to play too, that messed up our ring topology. When we ordered the T1's we were told that T1 lead time was 35-45 days for the installation. When the T1s at UCLA were due we had a schedule slip because someone parked on the manhole cover on campus.
In contrast to the 35-45 days install time then, we got a T1 installed in 1999 in one day through PacBell which normally gives us good service.
Our first T1's averaged about $1,000/month. T1's now are about half that price.
Router Selection
We looked at Cisco, NSC, and Proteon routers. As part of our evaluation of routers, Danny Cohen and I flew up in a rented private plane through a thunder storm to meet with Cisco. We met the president, Sandy Lerner, Len Bosac and others.
Proteon had their own proprietary LAN technology that ran 80 Mb/s in addition to their standard Ethernet offering. Proteon was ahead of the rest of the industry, but their solution was never implemented by anyone else. There was one chief engineer for the company that could explain the interworkings of their router and how it reacted to IP stimuli. When we asked what their router did when congestion occured, whether it sent source quenches, for example, we had to give our questions to the sales droids who then got answers from the one engineer who knew. We couldn't talk to him directly. Cisco, in contrast was wide open to us. We could talk to anyone we wanted to. Those were the good old days.
Cisco's success started there and as their fortunes went up Proteon's future went down. It ended up that used Proteon routers were valuable. You could pull out the processor and keep the Ethernet line cards and populate it with Cisco processors and line cards and run it as a Cisco. They both used the same chassis and multi-bus. They both bought Ethernet boards and early serial interface cards from the same supplier. Cisco later designed their own line cards that could go full T1 data rates.
CSU/DSU Selection
We bought Datatel 3552 CSU/DSU's. I saw them at a trade show and convinced their engineers to put in a data invert function in their CSU/DSUs so we could run data at 1.535 8 bit clean, taking advantage of the fact that the HDLC zero insertion algorithm satisfied the telco ones density rules just fine. Other CSU/DSUs threw 1/8th of a T1 away assuring 1's density by inserting a 1 every 8th data bit, stripping it out at the other end.
V.35 Cables
We got V.35 cables from IDS. They were suitable for use in a nuclear war. The cables took a minute to plug in screwing in both sides as you went and 1/16th inch thick hoods protecting the pins and cable end. However, when we went to plug these war-ready cables they wouldn't plug in far enough to make electrical contact because the female V.35 recepticals on the Cisco routers were recessed too far. So we had to ship the cables back to IDS to get the hoods removed.
Putting it all together
We then scheduled the first Los Nettos technical committee meeting at ISI. The idea was to plug the routers in, demonstrate the capabilities of the router and how to log in and configure the Ciscos. We plugged them in and couldn't get any of the links to come up. We used T1 crossover cables to simulate the T1 lines between the CSU/DSUs. The CSU/DSUs were happy but the routers weren't seeing each other. We had intended to give everyone their router and a pair of CSU/DSU's at the meeting and send them home to plug them in. When nothing worked we sent UCLA home with their router so we could test against their site because their T1 came in first as I recall.
Bringing up the first T1
Of course the T1 didn't work because GTE rolled the transmit and receive pairs. Fortunately, we could fix this with a screw driver at the CSU/DSU. Then the CSU/DSUs were happy. The routers were still not talking. At this point the CSU vendor, Datatel, and Cisco were blaming each other. Then I pulled out a scope to look at wave forms on the V.35 interface. It turned out that Cisco accepted a transmit data clock from the CSU/DSU and hung onto the clock cycle for about 250 nanoseconds before sending the data and clock back to the CSU/DSU. This phase shifted the clock by about 130 degrees. Adding in the phase shift of the clock due to the 6 foot length of the cable the CSU was sampling the data bit state at about 160 degrees out of phase of where it should have. There were two problems here. The router shouldn't have delayed the clock and the Datatel should have used the retransmitted clock but didn't. They did in later versions of this model. At those data rates the data leads square wave is anything but square and it takes many nanoseconds to settle in. We solved this problem by optioning the Cisco serial interface for inverted clock on some of the ports!! Curiously, only some of the Cisco ports would only work with inverted clock. Some required normal clock sense.
So we were up and running on the first link and we delivered the rest of the routers and CSU/DSUs. We brought up the rest of the links soon after.
We brought up the whole network so that the routers could talk to each other and ping themselves. Then site by site they turned off their gateways to the ARPANET and changed their default route to route to the Los Nettos backbone. All went relatively smoothly until we did TIS. TIS got reports that they were no longer accessible even though we were advertising their route via EGP. Checking their DNS name lookup revealed that their mail host was still shown at their net 10 address instead of their /24 address.
Initial data rates were 10-20 pkts/sec. I got worried that the network would melt when I first saw greater that 100 pkts/sec. We are now seeing 10's of thousands of packets per second over our Verio link. That has been a big jump.
T1 Billing Woes
Our next challenge was to get the phone companies to bill the T1's at the proper rate. The TIS to UCLA line bill was sent to UCLA for the first six months instead of ISI as our order indicated. When GTE finally did give us a bill the bill was for about $5000 more than it should have been. GTE billed us for 19 miles instead of 1 mile until we called them on it. It took us over 8 months from the time we got our first bill until the time we had the TIS-UCLA circuit properly billed and over payments returned.
We learned early on that you had to examine every first T1 bill very closely. The phone companies would get the mileage wrong, or the incorrect tariff was used or taxes for FCC tariff bills were used on intra-LATA bills. The most common problem was the start date and when a T1 was terminated. Since you paid in advance, unless you canceled the T1 on the last day of the billing period they owed you a credit which they were happy to hold onto until you demanded it back.
Some routing issues
We ran into routing problems to some ARPANET sites a bit later. There was an LSI-11 gateway node with an 1822 interface to the ARPANET IMP, which was ISI's gateway to the ARPANET. It turned out that the number of routes in EGP routing updates exceeded the maximum Ethernet packet size and the overflow packet was malformed. We did an Ethernet capture of the EGP routing updates and after analyzing them sent them to BBN. BBN saw the problem and fixed their EGP implementation. They never designed into the routing protocol implemenation the possibility of overflowing a maximum size packet.
With an Ethernet interconnect between CERFnet at UCLA and Caltech we also sent bits to the NSFnet. That became much more appealing when the NSFnet moved from 56K backbone links and fuzzballs to T1 links and IBM's collection of token ring-connected PC's for routers. The IBM PC collection did work after a fashion, though the first PC's couldn't fill up a T1.
Soon we were getting full routes from CERFnet and distributing over 1000 routes through Los Nettos using IGP. We needed to upgrade our routers to AGS3+'s. We ordered these replacement chassis to arrive in December. When they arrived they were double boxed for protection. On opening the outer box we were surprised to find a big red bow on the inner box, just in time for Christmas.
More Connections to Los Nettos
Soon after bringing up the first ring in Los Nettos with USC, Caltech, UCLA and ISI with TIS hanging off UCLA we needed to add JPL Rand, IBM and UNISYS. To that end, we bought about 10 new Datatels. When they arrived we tested them and half didn't work out of the box. That is when we learned that the Datatels were sensitive to how they were shipped. We returned the bad Datatels and had the replacements and future orders shipped by shippers other than UPS. I believe Datatel later figured out how to make their CSU's not so sensitive to shipping abuses.
As I recall our first connected associates were NRTC and Network Research, which had a previous connection to ISI's network to permit their access to the ARPANET. Aerospace was connected at ISI too but they were connected to the MILNET so they didn't really touch down on ISI's network. After a while ISI decided that they needed to normalize their networking relationship by having their networks connected to Los Nettos not ISI's internal LAN network. At first they were not charged. Then we were being contacted by new organizations who wanted to be connected to the Internet not just the ARPANET. We decided to charge folks to be connected so we converted our first associates to paying customers too. When the MILNET closed up shop we got Aerospace as a customer too. By then we had a number of associates and the numbers were growing.
An interesting new member connection was our connection to NOSC. I believe that it was DARPA that decided that NOSC should be connected to both CERFnet and Los Nettos. To that end they were willing to pay for a T1 from San Diego where NOSC was to ISI. Ron Broersma and I spent some interesting time trying to bring this T1 up. The main problem was that the phone company dropped the T1 off a long ways away from where the equipment was. NOSC is on a very big campus. So Ron had to negotiate with the local networking folks to extend the T1 through campus wiring to where the router was. This proved to be a bit more than the Datatel 3552's could handle so Ron adquired a CSU which we connected between the inhouse wiring and the T1 demarc in the MPOE room of PacBell. That did the trick. NOSC already had an Ethernet connection to CERFnets Ethernet interconnect LAN for connecting routers to the NSFnet. This back door path to the NSFnet created some routing oportunities and challenges for us and was a learning experience for me.
Through it all we bought state of the art networking hardware and someone paid the bills to let us play with all of it while we built Los Nettos together.