Does the Internet Work? (A Rambling Rant)

One of the odd things about running a web site is even with a small site like this I end up giving a lot of money for services to people I’ve never met in person and whom live very far from me. My previous web host was based in Tennessee, while the current one is located in Connecticut. Occasionally I also paid a very helpful programmer from England to help me install and configure some CGI scripts on the old site.

As a small customer in this kind of environment, you really need to have friendly customer service on the other end of the arrangement. I’m pretty good at breaking things and it’s important to me if I think I’ve found a bug or a problem that somebody takes a second to ask “is he on to something?”

Which is why Dave Winer’s behavior continues to mystify me. The software this site uses runs on top of one of Winer’s products, Frontier, but the folks at Macrobyte deal with buying licenses, installation, etc., so I have no business relationship with Winer. And thank goodness for that.

Andrea and other users of EditThisPage.Com noticed that Google was no longer indexing their pages, and started posting rather mild complaints on their site. Userland (Winer’s company) didn’t know what was going on, but Seth Dillingham wrote a script that tried to access Andrea’s site posing as Google’s indexer, and lo and behold he got back a message saying,

Inktomi, your crawler is repeatedly hitting our servers getting the same WAP files over and over. Please stop pounding us, it’s hurting the service we provide to our customers. Thanks. [email protected].

Now if you read Scripting.Com more than casually, you know that Winer has complained about the behavior of indexing robots before, and for good reason. Some of the indexers do things that can quickly degrade web server performance. For example, on a few of my sites I have event calendars. If I forget to add the event calendar to my robots.txt file, a number of indexers will attempt to request the calendar page for every day from 1900 through near the end of this century. And they will request those files very rapidly, so the server goes from having maybe 500 to 600 requests an hour to 5,000 to 6,000 requests within 10 minutes. Now imagine a robot doing that at several hundred sites at EditThisPage.Com.

So it was perfectly reasonable for Seth to point out that Userland was blocking Google,

Because Userland didn’t like the load that Google’s indexer was putting on their servers, so they have prevented Google from indexing any of their sites.

In fact, Userland had intentionally turned off Google indexing a few weeks ago.

At first, Winer said that there was no way that Userland was blocking Google because one of Winer’s programmers said they weren’t (a common technical support answer — “what you’re describing is impossible”). Later, Winer conceded that Userland was blocking Google but put it down to a mistake or a but or “what ever.”

But my point here about customer service is that rather than concentrate on what seems to me to have been the primary issue — could Seth’s report of being blocked when masquerading as the Google indexer be repeated — Winer latches onto Seth’s conjecture that Userland is blocking Google because of the strain it puts on their servers and runs with that.

This is typical of how Winer repeatedly gets himself in trouble. He always ends up focusing on perceived or real personal slights while often ignoring the real meat and potatoes of the problem or issue that is being made.

The amusing thing is that Winer was complaining,

That Doc’s site has been de-indexed by Google completely punctures Seth’s theory, which unfortunately he didn’t state as a theory, and so now it’s become part of the folklore that UserLand is nasty or whatever, I’m already getting flames thanks to Seth’s post here.

Seth was completely right, but, thanks to Winer’s comments, Doc Searls went ahead and blasted Google on his site for their “censorship” and said Google’s actions prove that the web needs to move more toward Winer’s vision (I can’t wait to read the WinerLog take on that!!!)

And while I’m at it, two quick comments on Doc’s advocacy of Dave’s approach.

First, Winer seems to be moving to a model of having people use Radio Userland to serve web sites from the desktop. Maybe people who are comfortable with network security issues won’t have any problems with this, but the last thing I would want to do would be to add a web server to my desktop.

Second, Searls and Winer both conclude from this incident that, as Winer puts it, “the Internet doesn’t work,” with Searls saying the Internet needs a genuine directory infrastructure rather than a Google-style search engine. I disagree. In fact the Internet — especially Google — work a lot better than I ever imagined they would. Remember when there were all these stories about how the sheer size of the web would soon break popular search engines such as Alta Vista? Well, of course, technically it did, but then along came Google who decided to invest all of their cash in developing excellent search engine technology rather than throw away millions on buying a single domain name.

I continue to be amazed at how quickly I can find even the most obscure of information on Google. Which isn’t to say that there isn’t room for improvement or even new and better standards for searching and/or creating directories. But, still, Google does an excellent job of making Internet searching work well.

Microsoft Dreamin’

Dave Winer posted part of an e-mail that Charles Fitzgerald, a marketing director with Microsoft, sent him in reply to a query about the security model behind the .Net initiative.

The security system is a key part of protecting privacy of data. We think we have a very good and compartmentalized one that mortals can actually use (the weakest link in any security system is humans). And while theoretical browser holes and such make the headlines, the reality is we know how to run secure systems. We have some minor nits in the past (and hopefully we have learned from them), but the reality is we are probably the most attacked set of sites on the Internet. We get attacked tens of thousands of times a month – no better proof.

Minor nits? As a computer science professor I know puts it, Microsoft’s business is selling security holes disguised as applications.

Do Weblogs Promote Integrity?

On Scripting News, David Winer says he’s getting a lot of “pushback” for the last sentence of his recent article, The Web is a Writing Environment. Of course once you read the last sentence, it is not too hard to figure out why,

What’s said outside the barriers is already more interesting. Eventually we will shed our need for approval from the brand names of journalism. Today they look for teddy bears and warm-fuzzies, the cute stories that mask the real one — writers who work for others have less integrity to offer than those who do it for love.

Ugh. I think there is a lot wrong with contemporary journalism. The biggest problem, in my opinion, is that journalists and news organizations don’t do a very good job of living up to their own claimed ideals of objectivity and (especially) fact checking.

But simply taking the money issue out of the equation doesn’t necessarily improve the situation either. In fact the worst possible form of journalism is practiced by those small circle of people that I know and that you probably know as well who, out of nothing but the purest motivations, regularly forward e-mails filled with urban legends and myths that have found new lives on the Internet.

A friend of mine has a basic worldview that I don’t necessarily subscribe to, but which sometimes seems a lot more accurate than a lot of other efforts to describe the world that I’ve seen. His philosophy is this: People Are Stupid. A more complicated version might be: People Are Credulous.

One of the (many) things that continues to amaze me, for example, is the large number of people I know and respect who take psychics seriously. And it’s not just my friends and associates — look at the enormous popularity of the SciFi Channel’s “Crossing Over With John Edwards.”

For those of you who haven’t seen or heard about it, Edwards claims he can communicate with the dead. Basically he’s another fraud using classic cold reading techniques, judicious editing, and other techniques to make it seem like he’s accessing knowledge known only to audience members.

The first time I saw a SciFi channel promo for the show, I was contemplating canceling my subscription to the channel but figured the show would never be a hit — the fakery here is obvious, I thought. But obviously not as the show is so successful it’s rumored to be on the verge of jumping to a major network.

I’m not so sure how giving those true believers and fans of such a show will improve the integrity of journalism (in fact, I suspect the resulting web sites would be used as confirming evidence by print media of the problems with disintermediated journalism).

In this respect, easy to use web tools such as Winer’s Manila or Blogger or the Conversant software that powers this site is transformative only in the way that PageMaker and WordPerfect were transformative. All three forms of software made it easier than ever for people to produce professional looking documents in a variety of media. They did (and do) nothing, however, to prevent the resulting product from being crap.

What’s Up with RSS 0.92?

Like Steve Ivy I’m wondering exactly what Dave Winer is thinking with RSS 0.92. Okay I remember him complaining about how efforts to move forward with a new RSS implementation (or an extension of RSS 0.91) were getting bogged down and so he was striking out on his own. Fine — sometimes that’s the best way to innovate, and Winer certainly knows how to innovate.

But the RSS 0.92 that are showing up on Adam Curry’s web site and elsewhere are a move backwards in my opinion. As Ivy notes, Winer seems to have ditched the “title” and “link” tags in favor of shoving everything into the “description” field. Not only will that break existing applications that use RSS, but it also provides less flexibility when it comes around to displaying the RSS feed.

Is this where Winer intends RSS 0.92 to go, or is this just a small bug in the way Radio Userland or whatever application Curry and Winer are using to create the RSS feed?

Web sites on a Desktop?

At Scripting News, David Winer keeps saying he wants to see web sites on the desktop,

Last night talking with Brent about the scaling wall that Pyra is climbing I said what they should do is Blogger On The Desktop. Then everyone using Blogger could add their computer to the mix. Decentralization and P2P all the way. I’ve got to write an essay about this. Maybe in a few minutes. Desktop websites. It’s the cure for Dotcom Disease.

I hope he writes his essay soon because I have no idea what he’s talking about there. After all there are already a number of website on the desktop products. Dreamweaver will let do a nice web site on your desktop and then handle uploading it to an FTP site automatically.

Too complex? A tool such as Trellix does an excellent job of simplifying the process (or even Winer’s Radio Userland, which is impressive).

If Winer means actually serving pages from the desktop as well, that opens up a whole can of worms that I don’t think the average person using Blogger or a similar service wants to deal with.

Which brings me to the problems that Blogger is having raising money. Its venture capital partners decided not to offer additional funding, so Pyra is appealing for its users to donate money to buy new servers. New servers are needed because the service is slow in general, and more resources will be needed to launch the fee-based pro version of Blogger.

If I were a Blogger user, however, I might be concerned about Pyra’s long term ability to reach profitability. The market it is entering with its fee-based system is already getting pretty crowded and Blogger has an extremely narrow focus. I’m using Conversant to manage this web site, but if I weren’t and I wanted a solution that was going to cost real money, I’d opt for using something like Trellix, a hosted Manila site or even one of hosts that offers Slashcode before I’d go with Blogger.

Recount Almost Finished, What Happens Next?

David Winer argues on Scripting.Com that whoever loses the recount in Florida should just concede the election,

Here comes an opinion. The time to be partisan is past. If the recount says Gore won, he won. If the recount says Bush won, same deal. The election is almost over. Further, elections always have flaws.

Of course this ain’t going to happen. With 63 of 67 counties reporting in the recount, Bush leads Gore by only 359 votes. Lets be blunt about this — the recount has to be inaccurate, and any subsequent recounts will suffer from inaccurcies as well. There is simply no way to reliably count roughly 6 million votes with the sort of precision to capture such an extremely small percentage difference. At this point, election officials might as well flip a coin to decide who won Florida.

Whoever is declared the loser in Florida is clearly going to sue, which is going to cause a lot of headaches because both candidates have won other states by a very small percentage of votes, and minor election mix-ups are the norm around the country as Deroy Murdock documents well in a column at the National Review’s web site. Normally such problems don’t affect elections because rarely are they this close.

In Wisconsin, for example, Gore won by only 6,000 votes out of 2.5 million votes cast — slightly more than two tenths of one percent difference. And talk about election fraud — Democratic activists were handing out packs of cigarettes to homeless voters.

If Gore is successful in getting courts to order yet another recount or a hands-on rather than electronic recount of some Florida counties, Bush is almost certainly going to respond by challenging Gore victories in Wisconsin, Iowa, New Mexico and maybe elsewhere.

Winer is certainly right that given all of the fallout that happens if Gore or Bush lose the recount and then start mounting legal challenges, it might be best if the loser simply conceded for the good of the nation, but if Bush and Gore were the sort to go along with that idea they probably wouldn’t be in politics at this level (then again if Richard Nixon of all people could concede a tainted election to his opponent, surely Gore or Bush could muster up the moral courage to match Nixon!)