April 30, 2003

Bad Customer Service Experience #12,812

How I love bad customer service. In this latest saga, a vendor takes a minor problem and turns it into a major one, through a cunning combination of poor customer communication and not-entirely-thought-out provisioning procedures:

I use Vonage as my primary business landline. I've been using the service for over a year and, despite ongoing growing pains, have been pretty happy with it. The ability to take my a landline with me, whether I'm in the office, at the lake or on the road, is valuable. This is especially true in hotels, where you can often get broadband for free, but they gouge you on calls. Add in the geek factor of folks being able to call you on a 773 number when you are in a hotel in Tokyo and you've got a pretty sexy product.

Recently, however, the Cisco ATA-186 that Vonage uses to deliver VoIP service to a normal telephone started making a high pitched whining noise. This is fairly odd, since the device itself doesn't have a speaker -- but there must be some digital component inside the unit that is screeching, quietly, at the upper range of my hearing. It was obnoxious enough that I finally decided to get in touch with Vonage and see if they would swap out the unit.

First I tried to call their customer service hotline. After 15 minutes on hold, I got a message saying "Sorry, we're too busy to deal with you right now. Please send us an email instead. Thanks." So, I sent an email. And waited. And waited.

To make a long story short, three weeks and several emails later, someone from Vonage finally called me on Monday and agreed that they needed to swap out the unit. I got an email confirming this, followed by an email with instructions on how to do the return. Then, a few hours later, my Vonage phone line went dead.

No dial tone. No incoming calls. Totally kaput. "No problem," I thought, "I'll just call customer support." Called. Waited 15 minutes. Got hung up on.

So, I mirrored the ATA-186's Ethernet port to a diagnostic port on my switch, broke out tcpdump and watched what the ATA-186 was doing. The device was booting up fine, but when it tried to get its configuration file from the Vonage servers using TFTP, it kept getting "file not found".

Ah ha! Vonage probably thought the unit was totally broken, instead of just misfunctioning in a mildly annoying manner, and figured they could shutdown service to the old unit entirely. So, I emailed Vonage's "tier 3" technical support, explaining my problem and got an email back around 7PM the same day. This fellow confirmed that my diagnosis was correct, but provided no further details on how they were going to solve the problem.

The next day, I gave Vonage a call at 7AM Eastern to try and talk to a real person before the customer service system started hanging up on people again. I got through fairly quickly, spoke to a gentleman who, instead of trying to solve the problem, kept trying to explain to me how their provisioning system worked. Finally, after a few goes at this, I was transferred, with a heavy sigh, to a supervisor. He listened to my problem, asked for a few minutes to talk to his engineering staff, and promised to call me back.

Much to my surprise, he did. And then explained that their engineering department was so swamped and/or undertrained that, while he could reprovision my old unit, he wasn't confident he could get it done before the new unit would arrive here. And, if he did do this, I would have to call back (ha!) and ask them to reprovision the new unit, once it did show up, a request that would need to pass through the same quantum uncertainty scheduling system again. By the time all of this was said and done, I would probably be better off just waiting for the new ATA-186, he explained. To add insult to injury, the supervisor was going to make me pay for this outage, until I asked him to arrange a service credit for the period of the outage.

I'm happy enough with Vonage that I am not planning on discontinuing my service over this incident, but it is a perfect example of how companies can piss off their customers, even when they think they are helping them out. The tech who arranged for the new ATA-186 should have told me that my service would be impacted and arranged for the new unit to be overnighted, to minimize my inconvenience. Since this didn't happen, Vonage should have gone out of its way to reprovision my old unit and make arrangements to have the new unit rapidly provisioned once it arrived. Instead, I spent 90 minutes on phone calls and emails trying to figure out what had happened and ended up, at the end of all this, without a real resolution to my problem. And posted a bitchy whine on patrick.com.

Posted by pmk at 9:26 AM | Comments (3)

April 29, 2003

Scary Things

I have this idea for a little website.

People used to mail out Christmas letters. Full of little tidbits on their family, job, illnesses, kids, etc. With email, people decided it would be a good idea to just send these things out electronically and save themselves the postage. And hey, while you're at it, why not just toss the old xmas letter up on the web for good measure.

The problem is people forget that google is their worst enemy. The next time they are looking for a significant other, job or home loan, all of these little tidbits will come back to haunt them.

Anyway, my idea is to compile a nice little gallery of embarassing online Christmas letters for the collective amusement and horror of passerbys. Imagine my surprise when I explained this to one Zack Exley and he told me that he had the same idea, but for wedding sites.

Looking at this doozy, I wonder if he might have a winner...

Posted by pmk at 12:15 AM | Comments (5)

April 28, 2003

Cup Stacking Redux

Emily Fox won the World Cup Stacking Championships. No times, so I figure she probably didn't break her world record, but still cool.

One interesting note is that Emily's father, Bob Fox, owns Speed Stacks, the folks who make most of the cups and other "gear" for cup stacking. Conspiracy theory anyone???

Posted by pmk at 5:49 PM | Comments (1)

April 27, 2003

Virtual Highrise

Want to get in on the ground floor (literally!) of a new high rise in Germany? Stop by Mr. Wong's Soup'Partments and design your own one-story apartment or two-story loft.

The building currently has over 200 residents and is growing daily. The amazing thing is the quality of the submissions. This is true community art, but without the pretention that our so-called "new media artists" bring to the table. Real people, putting together an unreal building. Very cool.

Posted by pmk at 5:38 PM | Comments (0)

vmailmgr, qmail and SpamAssassin

Getting qmail, vmailmgr and SpamAssassin to play well together requires a bit of glue. Theoretically, you should be able to change your dot-qmail file from:

| /usr/bin/vdeliver


| spamc -f | /usr/bin/vdeliver

and everything should work. Unfortunately, vdeliver expects its input to be seekable. So, you need to pipe through Bruce Guenter's filepipe.c first, to make the stream seekable. Once you've got filepipe compiled, you can change your dot-qmail to:

| preline spamc -f | /usr/local/bin/filepipe /usr/bin/vdeliver

and you're off to the races.

Posted by pmk at 12:24 PM | Comments (1)

April 26, 2003

Cup Stacking

The world championships in cup stacking are tomorrow, April 27th, in Denver. Cup stacking is a "sport" where the competitor must stack and unstack 12 plastic cups into pyramids.

The current world champion is Emily Fox -- Take a look at this video video of her most recent world record attempt in 2002.

Any post on cup stacking would be incomplete if I didn't mention that my neighbor, Doug Burton, dished up some cup stacking smackdown at our first (and last) competition.

Posted by pmk at 6:37 PM | Comments (1)

April 21, 2003

Electric Heli-thing

I just placed an order with Creek Hobbies for a Snelflight Hoverfly.

The Hoverfly is a small, electric remote-control helicopter. It is for indoor use only, since it plugs directly into the electrical mains and is designed mostly as a trainer for larger RC helis.

Still, it's a fairly cool looking toy. Lift is provided by three propellors mounted on the main rotor, instead of by the main rotor itself. Each propellor is controlled independetly, supposedly making the model very responsive to control inputs.

The general theory behind this thing is that you get started on this thing, so that when you get a real RC heli, you don't crash it into the ground straight-off. The Hoverfly is mechanically simple and fairly light, so a crash is less likely to cause expensive damage to the model (or you!).

I'll post a followup when I get this thing.

Posted by pmk at 1:18 PM | Comments (1)

April 16, 2003

802.1q VLANs, Linux and Me

802.1q VLANs let you pass multiple Ethernet networks across a single physical Ethernet cable. By adding a 4-byte tag to the Ethernet header, 802.1q-enabled equipment can distinguish between the various networks on the wire and treat each of them separately. These tags are the reason why 802.1q VLANs are sometimes called "tagged VLANs".

VLANs are an aggregation technology. I can take 10 Ethernet segments sitting out on the edge of my network and aggregate them across a single Gigabit Ethernet link to my core to be routed/filtered/whatever.

I use VLANs in my office to keep noisy equipment, such as servers, high-end switches, routers, etc. out of my work area. Due to the way the building is laid out, our fiber, SDSL and wireless connections all terminate in the same room as I work. If I had to keep my servers in here as well, I would quickly go deaf. Noisy stuff lives in a data closet on the first floor, keeping my office nice and quiet.

I recently removed an Extreme Network Summit 48 from the data closet. I had been using it to break out a bunch of 802.1q VLANs to devices that didn't have VLAN support. Summit switches are fantastic (especially when you get them for next to nothing on ebay), but they are also fairly loud and run hot. I wasn't really using any of the Layer 3 features that the switch offered, so I wanted to get rid of it. Since its primary use was breaking out my tagged VLANs to my Zebra-based BGP router, I figured I would try out RedHat 9 on the router, which ships with VLAN support and see if I couldn't get rid of the Summit entirely.

Everything seemed OK, at first. The VLAN stuff worked out of the box. The 8021q kernel module is included in the standard RedHat kernel build and the vconfig tool is available as an RPM.

Unfortunately, when the Zebra BGP daemon tried to suck down routes from my peers, it would get one or two routes, then stop. After a tip from RS, it became clear that Ethernet packets larger than 1468 bytes were getting dropped on the floor. With a default MTU of 1500, you should be able to get a 1472 (1500-28 bytes of IP header==1472) byte ping through. Interestingly, 1472-1768==4 -- the size of an 802.1q tag.

As it turns out, RedHat included 802.1q kernel drivers and 802.1q user space tools, but did not modify the Ethernet drivers in the kernel to support VLAN-sized Ethernet frames. Tsk tsk, RedHat. After digging around a bit, I found a patch for my 3c59x-based Ethernet card. But now I'm stuck patching the 3c59x driver everytime RedHat releases a new kernel. Someone has already opened up a ticket with RedHat on this issue, so hopefully they'll resolve it themselves some day.

As far as I can tell, none of the major Linux distributions have patched their Ethernet drivers to support VLANs. So if you're trying the same thing, dig around on google a bit and see if you can find a patch.

For the (even more?) technically inclined:

Configuring VLANs under RedHat 9 will be very familiar with anyone who has configured standard Ethernet interfaces under earlier RedHats. Just create a file called ifcfg-ethx.vlany, where x is the physical Ethernet interface that the VLAN terminates on and y is the VLAN ID. If you want to use tagged and untagged VLANs at the same time, just create a standard ifcfg-ethx file, with the IP address, netmask and other details for your untagged VLAN. If you are going to use tagged VLANs only, you'll want to create a ifcfg-ethx that contains:


to make sure the interface that your tagged VLANs are hanging off is brought up at boot-time.

Posted by pmk at 12:30 PM | Comments (2)

April 14, 2003

Pariser's Grand Plan Starts Here

Eli Pariser, wunderkind extraordinaire of MoveOn.org, has started a blog. His inaugural entry features a translation by yours truly of an article from die Tageszeitung describing Eli's plans to take over the world.

Go take a gander. Acccording to the German newsmedia, this is your chance to mingle with your eventual overlord!

Posted by pmk at 11:32 AM | Comments (1)

April 1, 2003

Andrew Curry discusses Iraqi war journalism

Andrew Curry of U.S. News and World Report fame has an email interview on knot magazine comparing and contrasting Peter Arnett and Geraldo Rivera's journalistic misdeeds.

It's comforting to see that Andrew is just as profane in an e-mail interview as he is in real-life. You can almost hear his frustrated sighing through the monitor...

Posted by pmk at 11:44 PM | Comments (1)

Rice Pudding Heaven

Ted Warin sent along a recent article (registration required) in the New York Times about a new Manhattan restaurant that serves nothing but rice pudding.

As Ted said:

I seriously thought this was an April Fool's Day joke. I still sort of think it is...

I don't know what I can add to that.

Posted by pmk at 11:29 PM | Comments (0)