The Chatterbox

To API or not to API...

As you've read recently, I've become somewhat enamored (or enamoured for my Brit friends) with APIs. I've actually been discussing creating API applications for some time now with several close friends. Both to get an idea of what might be needed and if my thoughts on the whole API movement makes sense.

(The considered opinion across the board so far is that it does make a lot of sense and is a wonderful opportunity btw.)

Anyway, Debra O'Neil-Mastaler of Alliance Link and I have been talking about developing some applications that could benefit those who are trying to gain a grasp on certain aspects of getting a site --that is worthy of ranking well-- to the point that it can actually rank well.

The problem being that for the average business owner who wants to take their business to the Web, it's a very difficult and often confusing process. It's difficult enough for those with lots of experience in marketing on the web! Partly because there is so much information out there that is frankly complete and utter BS. Partly because "the competition" often has a several years head start.

Note that neither Debra or I think there is any formula which will automatically cause a site rank well, as it should be. And even if there was, such a formula would most definiately not necessarily be the best thing for the users of any site, which simply has to be the goal of every webmaster.

Even though one has to keep the end user firmly in mind, one has to also at least keep the search engines in the back of their mind when developing their online presence. And there are certain things which end up causing newer sites --that may in fact be much better-- from being able to compete with sites that have more age on them.

One of the things that is very difficult for new webmasters to get a handle on is the fact that links pointing to a site can have a drastic effect on where a site ranks. Over time, a truly worthy site will collect these links. No doubt about that. But by the same token, new webmasters simply don't have the expertise or experience to collect and analyze this type of data. And many give up in total frustration long before their site has enough age on it to be on an equal playing field with older sites.

The information to build these links is out there, but it's confusing, difficult to understand and very hard to come by.

So we had an idea to develop an API tool called Common Link (beta) which might help to simplify this process.

In a nutshell the concept behind Common Link was to look at the links that are pointing to the top ranked sites for a given search phrase. But instead of having to process the information manually, the idea we had was to look at how many of those top ranked sites have "common links". Or in English how many of those top ranked sites had links pointing to them that were housed on the same external sites.

The theory being that not only would the tool help someone with a new (worthy) site easily identify other sites which accept link requests for that market, but it might also help them to identify those sites which are considered Authority sites for the market.

Long story short, as long as the content of your site measured up, Common Link could give a webmaster a leg up on the linking side of things, hopefully assisting them in getting their site ranked highly enough to actually compete with these older sites in a shorter period of time. Before they became disillusioned and let their site expire.

The theory makes sense and I believe it's sound, even though it's difficult to test. I've looked at probably 50 different markets over the past few days and it's rather amazing how many of the Top 10 sites in most markets share common links from other sites. So it's our belief that the tool could be quite helpful.

But we've run into a slight problem.

The current API license that Yahoo! offers has a rate limit of 5,000 queries per day from any single server's IP number. That sounds like a lot, but it's not if: a) One person/company develops several API applications that are housed on the same server; b) Several someone's who happen to occupy the same server are developing Yahoo! API applications, since they all count, enmasse, against the same server IP's daily allocation; or c) You develop an application like this one that has the potential of doing a relatively massive amount of queries to the API from a single use.

That last one is the tricky bit.

You see, what the script does is query Yahoo! to see what sites link to Site A. Then it compares those against the links pointing towards Site B, Site C, etc, etc.

Normally it's not much of an issue since the "average" site may have anywhere from 50 to 200 links pointing to it. But if someone happens to run a query of a relatively competitive phrase, it's not at all unusual to have thousands or sites that share links to two or more of the top ranked sites.

In testing, even if we restrict the number of links to look at, it's still quite possible to get a list of 3,000 or more "common links", or sites that link to two or more of the top ranked sites. This of course produces a huge number of queries being sent to the Yahoo! API. And quickly puts the entire server over its daily rate limit.

Biting the bullet, I decided to write the email address provided by Yahoo! for those who wanted to create a Commercial application based upon their API technology. We really didn't want to have to commercialize the script, but because of the above scenerio's it looked like there was not going to be a choice in the matter.

As it turns out though, the Yahoo! API Developer's area does not yet have any method of overcoming the rate limit problem. They don't offer any sort of Commercial license at this time. The form letter I got back yesterday said as much.

So at the moment we're at a loss as to whether to develop this tool, or any tool using the Yahoo API.

The Common Link tool is certainly a No Go if a solution cannot be found, simply because there's no way to head off those competitive industry searches on the front end. But if the rate limit is going to be a problem, and it honestly looks like it's going to be at this point, it simply wouldn't make much sense to set about developing multiple tools for people to use. No matter how useful they may be.

It's a continuing story at this point. We're going to try to get in touch with someone at Yahoo! to see if there are any options. The whole API thing is so new that we may well have hit upon something that simply wasn't considered when the original license terms were developed.

More as I know it...