Earlier this week I ran into a pesky issue when using slrn on a remote machine, inside Screen, over an SSH session, via rxvt, running under Cygwin on a WinXP box. The problem looked like this, and seems to be some sort of either character-encoding or display problem with non-ASCII characters used in slrn’s text-mode interface.

It’s an obnoxious problem because it rather seriously interferes with slrn’s thread-tree display, and because there are so many different layers involved, any of which could be to blame. Starting from the user and working backwards towards the source, there’s rxvt, SSH, Cygwin, Windows, Screen, slrn, and Linux, any of which could be causing it. (Although some are a lot more likely than others.)

The simplest and least elegant way to solve the problem is just to force slrn into pure-ASCII mode, by putting

set simulate_graphic_chars 1

into .slrnrc. However, that just seems wrong. VT100 box-drawing characters, which is all slrn seems to be using, aren’t exactly high-tech stuff — we’re talking about the very best of 1978, here. This isn’t Unicode or anything sexy; it’s just an alternate 7-bit character set triggered by “shifting in” and “shifting out” using escape sequences.

I think the problem is probably a termcap/terminfo issue, but I haven’t been able to get any results via any combination of terminfo settings that I’ve tried.

A few days ago I mentioned I’d picked up Jared Diamond’s book Collapse as my travel reading. Due to a few long airport delays (thanks, Delta), I’m now more than halfway through.

Rather than holding my thoughts until the end, I’ll be blunt: so far, I’m tremendously impressed. It’s a much more engaging read than Guns, Germs, and Steel, perhaps because it’s focused more narrowly, and the style is a little less academic. It is at times a downright chilling book.

Diamond’s discussion of the fate of the Easter Islanders is often mentioned in summaries and descriptions of the book, maybe because it’s in the first chapter. However, he spends far more time talking about the fate of Norse Greenland, and from the perspective of a modern American, it’s an easier story to relate to.

There’s a certain horror-movie quality to reading about the downfall of societies, especially ones who arguably doomed themselves or contributed to their own demise. Except instead of “don’t go in the basement!” it’s “don’t cut down those trees!” or “don’t try to graze sheep there!” But we know, of course, what’s going to happen.

You would have to be particularly thick to read Collapse and not draw substantial parallels to the fragility of our current society — not least of all because Diamond sometimes goes out of his way to explicitly make the point. Recently, the New York Times Sunday Book Review asked a number of prominent authors for books that they’d like to see the current crop of Presidential candidates reading. Personally I’d be happy if any of them picked up Collapse.

I’ve been pretty pleased with results of my experimental entry into the world of VoIP, because it had been working without a hitch. Up until tonight, anyway.

I noticed the problem when I went to call the new home VoIP number from my cellphone, and got a “Not available” message from Callcentric. I know immediately something was not right, because that shouldn’t ever happen (unless the power was out or Internet service was interrupted). When I got home I logged into the router’s configuration page, and discovered that the line was no longer registered with Callcentric’s servers.

I started off by fixing the obvious things, including network connections and a power cycle. I made sure I could ping Callcentric, so no problems there. The configuration on the ATA matched their website (plus, it had been working fine for a week), so hopefully no problems there. To rule out NAT issues, I put the ATA temporarily in the LAN DMZ. Still no dice.

Getting a little more desperate, I turned on the SPA-2102’s syslog feature, turned the debug verbosity up, and started tailing the output on my PC. The result was mildly enlightening:

 Jun 12 00:33:25 192.168.1.150 system request reboot
 Jun 12 00:33:25 192.168.1.150 fu:0:45af, 0038 043c 0445 0001
 Jun 12 00:33:25 192.168.1.150 fu:0:4605, 03e4 05b0 0001
 Jun 12 00:33:30 192.168.1.150 System started: ip@192.168.1.150, reboot reason:C4
 Jun 12 00:33:30 192.168.1.150 System started: ip@192.168.1.150, reboot reason:C4
 Jun 12 00:33:30 192.168.1.150   subnet mask:    255.255.255.0
 Jun 12 00:33:30 192.168.1.150   gateway ip:     192.168.1.1
 Jun 12 00:33:30 192.168.1.150   dns servers(2): 
 Jun 12 00:33:30 192.168.1.150 192.168.1.1 
 Jun 12 00:33:30 192.168.1.150 71.170.11.156 
 Jun 12 00:33:30 192.168.1.150 
 Jun 12 00:33:30 192.168.1.150 fu:0:4648, 03f6 0001
 Jun 12 00:33:30 192.168.1.150 RSE_DEBUG: reference domain:_sip._udp.callcentric.com
 Jun 12 00:33:30 192.168.1.150 [0]Reg Addr Change(0) 0:0->cc0bc017:5080
 Jun 12 00:33:30 192.168.1.150 [0]Reg Addr Change(0) 0:0->cc0bc017:5080
 Jun 12 00:33:38 192.168.1.150 IDBG: st-0
 Jun 12 00:33:38 192.168.1.150 fs:10648:10720:65536
 Jun 12 00:33:38 192.168.1.150 fls:af:1:0:0
 Jun 12 00:33:38 192.168.1.150 fbr:0:3000:3000:04605:0002:0001:3.3.6
 Jun 12 00:33:38 192.168.1.150 fhs:01:0:0001:upg:app:0:3.3.6
 Jun 12 00:33:38 192.168.1.150 fhs:02:0:0002:upg:app:1:3.3.6
 Jun 12 00:33:38 192.168.1.150 fhs:03:0:0003:upg:app:2:3.3.6
 Jun 12 00:33:39 192.168.1.150 fu:0:465a, 0003 0001
 Jun 12 00:34:02 192.168.1.150 RSE_DEBUG: getting alternate from domain:_sip._udp.callcentric.com
 Jun 12 00:34:02 192.168.1.150 [0]Reg Addr Change(0) cc0bc017:5080->cc0bc022:5080
 Jun 12 00:34:02 192.168.1.150 [0]Reg Addr Change(0) cc0bc017:5080->cc0bc022:5080
 Jun 12 00:34:34 192.168.1.150 RSE_DEBUG: getting alternate from domain:_sip._udp.callcentric.com
 Jun 12 00:34:34 192.168.1.150 [0]RegFail. Retry in 30

After that, there are just a lot of “unref domain” errors, repeated over and over every 30 seconds, as the 2102 tries to register and can’t. (Can we hear it for the guy at Linksys who got them to keep the remote logging feature?)

From this we can tell a few things. It looks like the 2102 is booting up, and then it’s looking for Callcentric’s SIP server, by querying the DNS SRV record. This is as it should be. However, for some reason it’s apparently not getting back the right server to use.

Just as a first shot to eliminate DNS issues, I changed out the DNS server values in the 2102 configuration (normally, I use my gateway/router, which lives at 192.168.1.1) with my ISP’s DNS servers. No improvement. Then, I decided to try pulling the SRV records manually, to see if there was an obvious misconfiguration on Callcentric’s part, or if they weren’t returning DNS SRVs.

Without getting into a whole sidetrack on how DNS SRV records work, the way to pull them is via dig. To get the server and port for SIP traffic carried on UDP for the Callcentric.com domain, you would run

 $ dig _sip._udp.callcentric.com SRV

 ; <<>> DiG 9.3.2 <<>> _sip._udp.callcentric.com SRV
 ;; global options:  printcmd
 ;; Got answer:
 ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11397
 ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 3, ADDITIONAL: 5

 ;; QUESTION SECTION:
 ;_sip._udp.callcentric.com.     IN      SRV

 ;; ANSWER SECTION:
 _sip._udp.callcentric.com. 1800 IN      SRV     5 5 5080 alpha4.callcentric.com.
 _sip._udp.callcentric.com. 1800 IN      SRV     5 5 5080 alpha2.callcentric.com.

This tells us that UDP SIP traffic should be directed to either alpha2.callcentric.com or alpha4.callcentric.com, both on port 5080. The servers have equal priority so either one can be used. Running a quick host alpha2.callcentric.com gives the A record for that server, which turns out to be 204.11.192.23.

What we’ve accomplished at this point is what the SPA-2102 is supposed to do every time it tries to register with Callcentric. Query the domain-level SRV record to get the particular server for SIP traffic, and then query that server’s record for its IP address, and then connect to it. We just did that, and now have an IP and port.

To see if that server worked, I put it into the SPA’s incoming and outgoing proxy fields, and turned “Use DNS SRV” off. Lo and behold, after I rebooted it, I was back online.

For the moment, anyway, things are working again. However, they’re not working the way they’re supposed to. If Callcentric decides to change its server’s IP address, I’ll no longer be able to connect. Ditto if that particular server gets overloaded. All the benefits of DNS are lost when you go this route. Therefore, it’s not really a satisfactory long-term solution.

I’ve opened a trouble ticket with Callcentric and will see what they say. Googling terms like “RSE_DEBUG” and “unref domain” produce some results — I’m apparently not the only person to have experienced this problem! — but no good solutions. It’s obviously a DNS problem, but who’s exactly to blame isn’t clear. I suspect Callcentric is going to blame either the ATA configuration or my LAN setup, and in their defense, their DNS records seem to be correct. However I can’t see how the problem can be misconfiguration when it worked well for more than a week. I suspect I’ll probably end up on the phone with Linksys eventually.

If I do figure out some sort of solution, or even a satisfactory explanation, I’ll be sure to post it. In the meantime, if anyone happens to come across this page because they’re experiencing the same problem, the only workaround I’ve found is to manually query the SIP server IP and put that into the 3102’s configuration. (And pray your VoIP provider’s IP address assignments are relatively stable.)

Any thoughts or suggestions are, as always, appreciated.

FOLLOWUP: I got a form response back from Callcentric noting that my device was registered again, and blaming the problem on my Internet connection. (Of course, it was back up because I put the IP address in directly.) However, when I went back to using DNS SRV, it seemed to work fine … which really annoys me, because if there’s one thing I hate more than stuff that doesn’t work, it’s a stuff that breaks unpredictably and for no reason.

Having finished Gang Leader for a Day (someday soon I’ll get around to writing up some of my final thoughts), I’ve moved onto Jared Diamond’s Collapse for my sitting-in-airports reading. Although I’ve barely made it through the introduction, so far I’m impressed. Despite his tendency to be longwinded — the major criticism of Guns, Germs, and Steel that I agree with — he seems to have a good grasp of the complex issues underlying modern environmental issues.

There’s a choice quote in the first chapter that I wanted to highlight. Diamond quotes environmentalist David Stiller, writing about the nature of the corporation as an entity.

“ASARCO [American Smelting and Refining Company {…}] can hardly be blamed [for not cleaning up an especially toxic mine that it owned]. American businesses exist to make money for their owners; it is the modus operandi of American capitalism. {…} Successful businesses differentiate between those expenses necessary to stay in business and and those more pensively characterized as ‘moral obligations.’ Difficulties or reluctance to understand and accept this distinction underscores much of the tension between advocates of broadly mandated environmental programs and the business community.

(Text in square brackets is Diamond’s, in curly braces is mine.)

This is a good point and bears much repeating. Corporations aren’t immoral, they’re amoral. Asking corporations to act ‘morally’ is like asking water to flow uphill. We’d do better to make the behaviors that we want — protecting the environment, treating workers fairly, whatever they may be — profitable, either by creating genuine incentives, or by punishing noncompliance, than to ask nicely and cluck our tongues when our toothless requests are ignored.

On the other side of the coin, Diamond seems to also appreciate that as simple as corporations are, actual human beings are not.

Whenever I have actually been able with Montanans, I have found their actions to be consistent with their values, even if those values clash with my own or those of other Montanans. That is, for the most part Montana’s {environmental} difficulties cannot be simplistically attributed to selfish evil people knowingly and reprehensibly profiting at the expense of neighbors. Instead, they involve clashes between people whose own particular backgrounds and values cause them to favor policies differing from those favored by people with different backgrounds and values.

Together, I think these two statements could be applied truthfully and insightfully to a wide range of current issues. The motives of other people, including and perhaps especially those with whom we disagree strongly, are seldom as simplistic as they appear. The motives of abstract, non-human actors like corporations, however, despite being made up of people, are often relatively simple.

It’s a mistake to reify corporations, and it’s equally a mistake to treat other real people like automatons. Both mistakes may produce what seem to be good predictions at first, but will fail in the long run; corporations don’t have a moral center, and will frequently do things that nobody in them as an individual would ever consider doing themselves, and virtually no one gets up in the morning intent on doing what they percieve to be evil.

If we want to produce realistic, workable solutions to pressing problems, one of our first steps has to be eliminating fallacious assumptions, no matter how satisfying (for example, perceiving those we disagree with as evil morons) they may be.

After doing my due diligence, combing NewEgg and the greater Internet for more than a week, reading every blog review I could find, and even making a little comparison chart, I decided to take the plunge and ordered myself a VoIP ATA.

At the last minute, I passed up the favorite for most of my comparison, the PAP2-NA, ordering the slightly more full-featured SPA-2102 instead. Although it allegedly lists for $110, I picked it up from Telephony Depot for $58, with after shipping was the best deal I could find.

The 2102 arrived yesterday, and I got a chance to play around and set it up last night. Overall, the installation process went smoothly, although I did run into one significant hiccup. The 2102’s installation and setup documentation is sufficient if you’re planning on using it at the edge of your LAN, but if you want to have it inside the LAN, you’re mostly on your own. Furthermore, the paper documentation for the voice-prompt interface is flat-out wrong in several areas, giving incorrect values for options (a problem that I believe stems from a mismatch between the firmware revision on the box and the version the docs were written for).

After having to reset the box several times — it’s possible to get it into a basically un-configurable state by switching it into bridge mode, when combined with the poorly-document voice prompt — I began writing up notes on the ‘right’ order to change the 2102 from gateway to statically-addressed, internal mode.

To do it, you’ll need a laptop or other computer with an Ethernet port that you can disconnect from your home LAN. (You definitely don’t want to plug the 2102 into your LAN un-configured, since out of the box it acts as a DHCP server.)

SPA-2102 LAN Setup Notes - 2.27kB ASCII text

For reasons not really germane here, I ended up typing up a very long email a few days ago, basically comprising a very rough introduction to VoIP. It’s less of a “guide” than it is just a braindump, but I thought I’d toss it up online, let Google do its magic, and perhaps it would be helpful to someone.

It can be found here:
VoIP Infodump - 16kB ASCII text

While poking around on Wikipedia I came across this interesting graphic. It’s a map of the Regional Bell Operating Companies (RBOCs), the regional telecom monopolies — I’m sorry, I meant incumbent carriers — showing their coverage of the U.S. both today and back at deregulation in 1988. It’s worth taking a look at.

The color-coding represents their territory coverage today, while the shaded lines mark boundaries between RBOCs at deregulation.

Ironically, there are fewer of them today than there were in 1988. That’s right; for all the effort that went into deregulating Ma Bell, she’s putting herself back together again, Terminator-style.

Consider the southeast and midwest, which has been subject to the greatest amount of reconsolidation. Originally, there were three RBOCs: BellSouth, Southwestern Bell, and Ameritech. BellSouth had the southeast from Kentucky to Florida; Southwestern Bell had the southern part of the midwest from Missouri to Texas; and Ameritech had the Great Lakes region, from Wisconsin east to Ohio.

Today, you’ll find scant evidence of those companies — they’re all parts of the AT&T empire once more, along with the former California and Nevada RBOC, Pacific Telesis. The rest of the nation is basically split between Quest in the West and Verizon in the East.

It’s looking more and more like 1988 will be remembered as the high-water mark for telco competition in the U.S., with a total of eight regional operating companies. Now, we’re down to three.

It’s as though the U.S., with a few years to dull the bad memories of high rates and rented phones, has forgotten what life under a monopoly carrier was like. If we’re not careful — especially with the evisceration of many pro-competition policies in the fallout from USTA v. FCC (2004)¹ — we’re going to end up back in some places we’d probably rather not return to.

Footnote 1: One of the best summaries of the issues at play in USTA v. FCC was written in early 2004, before the USSC declined to take up the case. It’s “USTA v. FCC: A Decision Ripe for the Supremes” by Fred R. Goldstein and Jonathan S. Marashlian. Here’s the money shot:

Of course, we know that’s exactly what the Supreme Court did, or rather declined to do; the decision wasn’t taken up for review, the DC Circuit’s pro-RBOC decision stood, and years of progress in bringing competition to telecommunications at the local level disappeared virtually overnight.

One thing just leads to another around here. My search for a decent VoIP ATA (basically, an Ethernet to analog telephone interface box) led me to discover that I’m all out of ports on my current Ethernet switch that holds together my home-office network. Oops. Guess this VoIP project just got a little bigger.

It’s been a while since I’ve bought much home networking gear, and I was impressed when I fired up NewEgg to discover how far prices on Gigabit switches have fallen. But looking at the specs on them convinced me that not all are made equal — and some of them seem downright trashy. I’ve done battle in the past with crummy, low-quality “consumer” networking gear in the past, and swore never to buy hardware purely (or even mostly) based on price again.

My absolute requirements are:

• 8 ports
• Gigabit Ethernet (802.3ab) on all ports and uplink
• Jumbo frames (>9000B payload)

The major ‘nice to haves’ in a new switch are:

• 12+ ports
• Support for Spanning Tree Protocol
• VLAN
• Link aggregation
• 802.1p ‘Priority Queuing’
• Power Over Ethernet (PoE) injection

My requirements aren’t that stringent — pretty much any run-of-the-mill 8-port switch satisfies them — so really it’s an exercise in balancing cost against which of the ‘nice to haves’ I can get.

• Rosewill RC-410

  • $50 from NewEgg
  • 8 ports
  • Jumbo frames
  • “802.1p flow control” (means priority tagging?)
  • 802.3ad - Link aggregation
  • Limited QoS (per-port QoS bit flagging?)
  • Rosewill seems to be NewEgg’s house brand. It got mostly positive reviews, with the main complaints being about the heat, and that there’s no 12 or 16-port version available.

• Netgear GS108

  • $55 after rebate from NewEgg
  • 8 ports
  • Jumbo frames (9000B max.)
  • 802.3x - Flow control
  • 802.1p - Priority tags
  • Steel case
  • Looks decent, one of Netgear’s “ProSafe” series. Doesn’t do link aggregation, though, and the price before rebate is $70. However, the higher-end Netgear kit has performed well for me in the past, so that’s something it has going for it.

• HP J9077A

  • $80 from NewEgg
  • 8 ports
  • Jumbo frames (9216B max.)
  • 802.3x - Flow control
  • 802.1p - Priority tags
  • Full specs on HP site
  • Starting to get into “real” networking gear, rather than the consumer/home-oriented stuff, here. Only downsides to this unit are the lack of VLAN and link aggregation. HP has a similar unit, the J9079A, which does both and a lot of other tricks besides, but only has 10/100 on the client ports and a GigE uplink.

• Netgear GS108T

  • $105 from NewEgg
  • 8 ports
  • Jumbo frames
  • 802.3x - Flow control
  • 802.1p - “Class of Service” (aka ‘Priority tags’)
  • Port-based VLAN
  • Port and DSCP-based QoS
  • 802.3ad - Link aggregation
  • LACP - Automatic link aggregation
  • 802.1w - Rapid Spanning Tree protocol
  • Now we’ve moved from unmanaged switches into “smart” switches, and we bought ourselves VLANs, QoS, LACP, RSTP, Syslog/SNMP support, port mirroring, and tons of other fun stuff. For what you get, this seems like a good price — the question is just whether it’s necessary.

• HP J9029A

  • $156
  • This one seems to take the J9077’s feature set and add to it many of the “smart switch” features in the Netgear above, including LACP aggregation, 802.1Q VLANs, and QoS. One major feature it doesn’t seem to support is RSTP/STP.

Decisions, decisions. The J9029A is pretty tempting, but it’s leaning distinctly towards overkill for a home LAN. However, I really like the idea of being able to set up VLANs at some point in the future; say, to take all the VoIP devices and put them on a separate VLAN and subnet, and then put that whole subnet behind a separate NAT router and give it a separate internet-facing IP address. (Obviously this would cost money and require purchasing a second public IP from Comcast.) I’m not sure if this will ever be necessary, but it seems like SIP+NAT is just a bad combination, and the glacial pace of IPv6 means it’s a problem that’s not going to go away any time soon. Being able to just segment off all the telephone stuff from data (and maybe making SAN stuff separate from that) seems like a nice feature.

I’ve been doing a lot of working from home lately, and that means spending a lot of time on the phone. Since I don’t have a POTS landline, and my cellphone is both expensive to use and tends to run out of batteries just when I need it most, I’ve been thinking that VoIP might make sense.

The main features I need are the capability to have two independent VoIP “lines” — one for me and one for the S.O. — and to integrate without too much fuss into my current LAN. I also don’t want to be tied to a single provider (e.g. Vonage, Comcast, Skype) or buy hardware that will become obsolete too quickly.

• Linksys SPA-2100

  • Review at VoIPUser.org
  • Two ports for handsets/devices (FXS ports), no analog backup
  • Two 10Mb Ethernet ports (WAN and LAN) with optional NAT routing
  • Uses QoS bit
  • Discontinued; replaced by 2102, below

• Linksys SPA-2102

  • Info page on VoIP-Info.org
  • 2 FXS ports
  • Two 100Mb Ethernet ports with optional NAT routing
  • QoS
  • T.38 fax transport (this is neat!)
  • Compatible with SPA-2000 dial plans (source)
  • Replaces the discontinued SPA-2100
  • Uses Voxilla configuration wizard
  • Allegedly supports two concurrent G.729 calls

• Linksys SPA-3102

  • Review
  • One FXS, one FXO (one handset, one analog/POTS backup)
  • Can be configured using the Voxilla.com online tool

• Linksys PAP2-NA

  • The “NA” version is unlocked, some other versions are locked
  • VoIPUser Review
  • TechZone Review
  • Wikipedia article
  • Official Linksys Page
  • Based on Sipura SPA-2000 (at least generally)
  • Basically a rebranded SPA-2002 (source)
  • Large user community
  • Two FXS ports (device), no FXO (analog backup)
  • Can only do one compressed call (G.726 or 729) at a time
  • Discontinued, replaced by PAP2T, below

• Linksys PAP2T-NA

  • Official Linksys page
  • Replacement for discontinued PAP2 and Sipura SPA-1001
  • May also replace the SPA-2002
  • Allegedly same as PAP2, but can do two simultaneous G.726 calls
  • May be a cost-reduced version ofthe PAP2 (cf. bennyzhou’s post)
  • Blog review

After doing a lot of comparison work myself, I found a nice page comparing all of Linksys’ VoIP ATAs. It claims, contrary to other sources, that the PAP2T can handle two simultaneous G.729 calls. I’m finding this more and more doubtful as I read more.

Right now I’m leaning towards the PAP2T, just because it seems likely to have the most people using it in the near future.