The Chatterbox: Search Engine Marketing Archives

August 12, 2009

New Google Preview

For those who don't actively track this stuff, Google is apparently getting set to release some major revamps of their code. Something they're calling Caffeine. You can see the preview at http://www2.sandbox.google.com/. You can also of course use this preview to compare rankings side-by-side with the current Google.

It's noteworthy to realize Google doesn't normally release such previews prior to an algorithm update. In fact it's never happened like this before. The fact that they are releasing a preview may indicate there is a lot more going on in this update than is at first apparent.

They're not saying much about what has changed exactly, as is completely normal and understandable. I'm sure those who watch these things will be espousing their opinions on the changes over the coming days/weeks. Note that you won't see any great changes in the user interface or how it looks visually. All of the changes are happening in the back end in the way they spider, index and rank sites.

I checked some of the markets and keywords I've tracked over a long time period and didn't see a ton of movement. One thing that I did notice is that without fail sites I knew were involved in a good bit of link spamming, where someone manages to get a ton of low quality links by blog/forum spamming, but where there are few or no legitimate high quality backlinks have taken a tumble. Not just a few places either, but many, many pages.

From this single observation it appears Google may be getting a better handle on preventing people from artificially pumping up their sites with these low quality links with spot on anchor text. I don't see it as a penalty, just that those links aren't conferring as much weight, trust or authority as they have been. Which is a good thing in my book.

In checking my own sites for their main (most competitive) keyword phrases they're all either holding steady in the New Google, or in many cases my sites have gained a bit more traction. This applies across the board, both for older more established sites and for newer sites that don't yet have a large footprint.

All in all I didn't see a lot of movement. But what I saw looks okay to me.

Additional Resources for more reading:
Matt Cutts' Blog
Google's Webmaster Central Blog
Vanessa Fox's initial thoughts on the new Google on Search Engine Land. (Vanessa used to work at Google for those unaware.)

Posted by Randy at 08:15 AM | Comments (0) | TrackBack

July 15, 2009

Google Rank Extractor (Part II)

UPDATES:
August 15, 2009 - GRE is now Ajaxified! Download links have been updated, documentation has been updated and there is now a dedicated page for the project that is located here.

The Google Rank Extractor, or GRE, is now officially a public Beta release. It may be downloaded from here.

I would like to thank those who volunteered to be Alpha testers and advisors. Your feedback was invaluable, as usual.

Several new features have been added since the Alpha version, the main one being that I've added an admin user interface to the package. This UI gives you several ways to sort and display the information that has been collected by the extractor.

Also added is some better documentation, which was definitely needed. Especially with the release of the user interface. The docs also include quick install instructions.

A couple of files have been added to the package and I've changed the structure just a bit so that the database connection info only has to be entered a single time and so the admin area can be easily password protected.

Lastly, I have removed the ability to save the data to a flat file. Starting with the Beta version GRE now requires a MySQL database to store the information being collected. This requirement simply makes the most sense given the amount of information that it'll end up collecting. Not to mention that one needs a sensible way to sort, process and display the information.

I'll be putting up a Demo (and will link to it from this post) sometime over the next day or two so that everyone can see what type of information gets collected and how it can be displayed in different ways. It'll take a little bit of time to get a working demo up and available since I need to build a database with real looking, but essentially fake information. I don't want to be handing out any proprietary information about my sites, and the tool does collect some very proprietary data. UPDATE: You can see the demo of the admin side of things here.

Enjoy !

Posted by Randy at 08:40 AM | Comments (3) | TrackBack

July 08, 2009

Google Rank Extractor

UPDATES:
July 15, 2009 - I've released the first public beta version. The announcement post is here and the download zip file has been updated.

August 15, 2009 - The GRE tool has been re-coded to be an Ajax application, removing the hard and fast requirement that the site's pages be php based. The announcement post of this update is here and I've also set up a dedicated page to the GRE project, which can be found here.

I've got a new little tool for you today. Yippee!

[If you want to skip the story, the download is here in zip format.]

Okay, what's all this Rank Extractor thing about. First a bit of history.

Forever people have been running (usually automated) queries against Google's search engine in an effort to try to figure out where their site ranks for its search phrases. These automated queries have always been against Google's Terms of Service.

Once upon a time Google would let you do this if you had applied for and used an API License Key in whatever automated rank reporting tool you used. Technically this is still the case, with one rather large caveat. The API License Key that's valid for this sort of thing is what is commonly known as the old style SOAP license. And Google stopped issuing new SOAP style API licenses some years ago now.

So basically if you weren't already doing this stuff years ago you had no way to get an API License to collect any ranking data for your site with regard to Google, while still remaining within the constraints of their TOS.

That's the quick history. Basically unless you were doing this stuff several years ago, you were SOL. Thankfully, all of that is about to change. And change for the better I might add.

A quick hit on the dirty details.

Though most haven't even noticed, Google began changing what shows up in the Referal (sic) area of hits a few months ago. It was originally confirmed and announced on the Official Google Analytics Blog back in mid-April.

The gist of it, and all you really need to know if you're not a tech junkie, is that Google is changing their referral string from the old style that looked something like:

http://www.google.com/search?hl=en&q=flowers&btnG=Google+Search

To a new style that embeds a lot information. The new style referral string looks something like:

http://www.google.com/url?sa=t&source=web&ct=res&cd=7&url=http%3A%2F%2Fwww.example.com%2Fmypage.htm&ei=0SjdSa-1N5O8M_qW8dQN&rct=j&q=flowers&usg=AFQjCNHJXSUh7Vw7oubPaO3tZOzz-F-u_w&sig2=X8uCFh6IoPtnwmvGMULQfw

The important part for our purposes, and something they didn't mention in the official release document, is that little bit in the referrer string that says cd= some number.

The really important part is that the "some number" value of this cd variable is the Ranking Position of your site for that phrase in Google at the time the link was clicked.

Say what!?!?! They're finally going to tell us exactly where we rank, for every single phrase that sends us traffic from Google.com? They're going to give us access to more information that we could have ever gotten from running some automated queries against their search engine, because there is no way we could ever extract every phrase that sends us traffic via Google, then guess whether what we saw was the same as what that user saw at the moment they clicked through?

Well, I'm here to tell ya that Google are going to do exactly that! And this is sooooo cool. It gives you relatively easy access to tons of information you've never had available to you before. And all you need to gain access to this new, quite important information is a little tool that extracts the information being sent to you during a Google referral hit.

Thus, my new little tool I've been testing on some of my sites for the last 6 weeks or so that I've coined Google Rank Extractor.

It's a really simple tool when you break down the code. All it does it review the referral string data, extract bits and pieces we want to record for posterity (namely the search phrase used, the ranking position, the version of Google searched, the date and time of the hit and the page of our site the user landed on) then drops it all either into a flat file or MySQL database.

Simple really. But also extremely useful.

I've not yet gotten around to developing a front end to help with the sorting of the data, but I can tell you already that I'm seeing all sorts of phrases I didn't even realize my site ranked well for show up with #1 or similar rankings. And apparently my little sites are kicking butt and taking names on some of the regional versions of Google (eg .co.uk, .com.au, etc) because on some of them my most competitive phrases in the worldwide search are consistently sitting at the #1 spot.

Cool beans!

And the Google Rank Extractor tool is incredibly easy to install if you have a .php based site. Especially if like me you use common header or footer files to drop some parts of your html template into each and every page. And to add to the fun, the tool is so lightweight that there is no discernible lag when the pages load. In fact, basically nothing in the php fires when it's not a Google referral hit.

Even though Google has not yet changed this referer string in all of their data centers yet (I'm only seeing it on about 5% of my Google hits so far) I can't stress enough how much data the tool will collect, nor how valuable the data can be both to your SEO and Marketing efforts.

Personally, I've found all sorts of nuggets that I've been able to leverage. Especially for the longer tail phrases where users are getting incredibly specific because they know what they want. Those are people ready to buy, if you just make sure you tell them you have what they want.

So I would encourage you to install the Google Rank Extractor on your PHP site. Let it run for a few days, then start reviewing the information. And check back here (you might want to bookmark the page, though it'll be mentioned in the files above too) in a few weeks. If nobody else gets to it first I plan on building a little front end that will help out considerably in sorting the data being collected so that it's both easier to consume and more actionable.

And if you're a developer please, please, please consider helping out. There are several ways to do so. You could build the front end, which shouldn't take long but my free time is pretty limited currently. Or you could port the code to other languages so that more people can make use of it. Or anything else that pops into your head. The GRE is released under a standard GNU/GPL license just like all of the other tools I release. So please run with it and improve it!

Posted by Randy at 04:18 AM | Comments (7) | TrackBack

September 04, 2008

Common Links Update

As you may have noticed, the Common Links tool was been down for maintenance the last few days. Thankfully Ed sent me a note last weekend to let me know it was returning gobbly-gook, so I took it offline to bring it up to date with the way Yahoo's API works these days, as compared to how they did things when the tool was originally released a few years ago.

Thanks for the heads up Ed !

While I had the tool down to bring it up to date, I decided to do a total rewrite of the back end code, adding a couple of features that had been requested. No time like the present, as they say.

So the tool still does the same basic thing. It allows you to enter a keyword phrase that it then uses to grab the Top 10 ranked sites for said phrase on whichever of the three major search engines you choose to query. It then uses Yahoo's API to grab backlink data for each of these top ranking sites, sorting and ordering it so that you can see which other sites link to two or more of the top ranking sites.

The idea being to give you a heads up regarding some sites you may want to approach to obtain links to your site. Hopefully links from sites that are already considered to be trusted and authoritative by the search engines.

The concept is pretty simple actually. Gathering, ordering and sorting all of the data is a bit more complicated. Almost impossible to do by hand in fact. Hence the reason I built the Common Links tool in the first place.

New additionals to this updated version of Common Links include:

1. The ability to manually insert your own site into the results table, even if it doesn't currently rank in the Top 10 on a certain engine. This feature was requested and I decided to add it into the mix since it can give you a quick visual reference to how your site stacks up against the top ranking sites.

The format you use when entering an Additional Site isn't all that important since I've added some checking into the back end code. You can use the http:// address, the www.domain.com address or even the hostname like domain.com. The tool will handle all of the above.

The tool also automatically checks for duplicate entries, which for the purposes of the tool means the same domain name. So if there are two listings for two pages of the same domain, the tool will automatically strip out the dupe and show the domain only one time. This applies for Additional Sites too. If you manually enter your site and it's already in the Top 10, it'll still only show up once.

Also, it's important to remember Additional Sites will always show up in position #1 in the tool. For our purposes the numbers aren't really the ranking of the sites, though it is if you do not enter an Additional Site. For the tool the numbers are used simply as a reference, not the ranking.

2. I've also added the ability to select whether you want the tool to report on backlinks to the specific page that ranks well, or show every link to the domain regardless of which page it links to. Personally I find the Domain choice gives me a lot better information, but a couple of people have asked for the ability to check backlinks to a specific page. So it's there now.

A couple of notable things to remember...

Sometimes you'll have a phrase that says only 8 or 9 sites were returned as being Top 10 sites. This happens because of the automatic pruning of dupes I mentioned above. Meaning if a certain keyword phrase in a certain search engine has two pages from one site showing in one of those indented listings, it's only going to show up once in the tool. This is done to eliminate needless queries for backlink data for the same domain multiple times.

Also, remember that this tool uses API licenses with each of the three major engines to gather its information. I've always been of the mind that it's a good thing to be a responsible Netizen, so I won't do any scraping, search engine or otherwise. A potential downside to this stance is that occasionally one or the other API licenses will max out its daily usage limits as set by the search engines. For Google that's 1,000 per day, for Yahoo and MSN/Live it's considerably more. And of course they Yahoo API gets a lot more use in this tool because that's the API that does most of the heavy lifting.

So if you get an Error, especially a 403 error, it typically means my license for a particular engine has hit its limit for the day. If you simply wait a day these issues should correct themselves.

I don't regularly monitor the use of this or other API tools I've developed, so I don't see when this happens 9 times out of 10. If you run into a problem where day after day you can't get the tool to work for you, let me know and I'll look into it. If I find someone abusing the free tool with automatic queries or overuse I'll simply cut off their access so they don't adversely affect anybody else.

That's it! Enjoy the new tool.

Posted by Randy at 08:55 AM | Comments (1) | TrackBack

July 11, 2008

Google All In Title Tool

I've added a new API tool to the site, based first upon a request from Jill and then a request or two from members of her forum. The new tool is simply a quick way to get an idea of competition a site may have by allowing you to run an All In Title search on Google. Just like using the normal operand on Google, except the tool allows you to load it with multiple keyword phrases and get them all reported at once.

The new Google All In Title tool uses their old SOAP API of course. I'm not about to release a tool into the wild that scrapes their results like some others seem to do all too often. If you have one of the old SOAP API license keys please plug it into the tool so that the queries will count against it. If you don't have one you can't get one anymore, so the tool will attempt to my Google SOAP API key.

Remember though, Google API licenses are limited to 1,000 queries per day and mine gets used by a few tools when someone else doesn't have one. So if the tool errors out it's probably simply that the license queries are used up for the day. Be nice if you have a license key. Use your own so that others who can't get a license anymore can get the data they want by using mine.

The tool is pretty self explanatory. Simply put a semi-colon after each keyword phrase so that the tool knows where one ends and the next begins. And after you've run you're query download the CSV file of your query to refer back to later, rather than having to run your phrases through the tool again later.

Enjoy!

Posted by Randy at 09:32 PM | Comments (4) | TrackBack

January 08, 2007

Common Link Update

I have just completed some updates and upgrades to the Common Link tool. The tool needed to be updated because of some changes that Yahoo has made since the last time I did anything with the tool. Basically, the address used by the script to connect with the Yahoo! API servers is migrating, as I noted back in November, so I took the time this past weekend to go through all of my Yahoo! API scripts to update them to the new address. (If you missed the original info about Common Link, it's here.

While I was in there I decided it would be a good idea to upgrade a few things.

1. I added the ability to search for common links in the top ranking sites on MSN/Live. So now you can run the tool against Yahoo!, Google or MSN/Live.

2. I have added a Link Depth selector.

3. A few other things things in the back end code to help it run more efficiently. These will all be transparent, other than that you may notice some performance improvements.

This second one is a bit more confusing, so allow me to take a moment to explain what it does and why I added it.

As you know, when the tool is run it checks for Common Links from other sites that link to the Top 10 (or whatever) sites in a certain search engine. It looks for sites that link to 2 or more of the top ranking sites in hope of aiding you in finding relevant quality linking possibilities it may be beneficial for you to persue.

The new Link Depth choice is simply a measurement of how deeply you want the tool to look for these common links. Now you can choose whether you want it to look at the first 50 backlinks being reported all the way up to the first 400 backlinks. This number is applied to each of the sites in your list.

Since there are always 10 sites you're checking against, choosing 50 would mean that the tool will look at the first 50 reported backlinks for each site. Or a total of 500 backlinks.

Whereas if you selected a Link Depth of 250 it would check 250 backlinks deep for each of the ten target sites, or 2,500 backlinks total. As you can see, it's an exponetial thing.

Previously, this Link Depth was something that was hard coded into the tool. You couldn't change it. (FTR, it was set at a high number.)

In some rare cases Common Link searches this was causing a problem. People were sometimes running into a problem where the tool would time out before it delivered the final results even though I have the script set up to allow it to run up to five minutes. Other times the server was running into an output memory buffer problem where it failed before the script could complete its run, even though I have it set up to allow up to the insanely generous amount of 64MB in the output buffer.

So now it's a setting you can change.

I've pre-selected a Link Depth of 150 as a default. In the testing I've done this seems to be a pretty good median.

The risk of using a smaller Link Depth size, if you want to call it a risk, is as far as we know the order of reported backlinks means little or nothing. eg The most valuable backlinks are not necessarily listed first. So if you were only checking the first 10 or 15 backlinks you might miss out on some really good candidates.

But on the other side of the coin setting a very high Link Depth usually means you have to wade through a lot of useless information when separating the truly valuable links from those that are simply link spam.

150 seems to be about right. In the tests I've performed changing this to the low end of 50 decreases the total number of reported Common Links by about 20-25% on average, but most of those were totally useless anyway. Increasing it from 150 to 300 increases the number of reported Common Links by about 20-25% on average, but again most if not all of the additional reported common links are useless as far as I'm concerned.

Bottom line, choose whatever Link Depth you want. Just realize the higher you set this number the longer it is going to take and the more queries you're going to be eating up from the limited stockpile we all have to use each day. If you run into a problem where the tool is timing out on your query, simply decrease the Link Depth setting until it goes through for you.

As always, let me know if you run into any problems or errors.

Posted by Randy at 09:21 AM | Comments (0)

November 06, 2006

Yahoo API Address Change

For those who haven't really been paying attention --like moi-- about a week ago Yahoo! announced a change in their API address we need to use. The announcement is here in the Yahoo! Developer Network.

Here's the quick and dirty skinny...

Before we used an address like api.search.yahoo.com. Or api.-product-.yahoo.com if you were using something other than the Search APIs. They're moving all of the API stuff to a different server location so that it's not sharing the same servers as the main Yahoo! applications do in order to improve performance for everyone. So you'll need to change any scripts you've made or any you got here or elsewhere so that the address reads search.yahooapis.com or -product-.yahooapis.com

That looks confusing doesn't it? Okay, time for some Plain Language...

Drop the api. subdomain off of the front of whatever api address you were using before and change yahoo.com to yahooapis.com. That's it, that's all, you're set.

So if you were using the Web Search Service like most of my APIs do, what before looked like:
http://api.search.yahoo.com/WebSearchService/V1/webSearch

will now look like:
http:/search.yahooapis.com/WebSearchService/V1/webSearch

Got it?

Word has it that all new updates and bug fixes will only appear at the new address moving forward. Plus they'll be phasing out the old address over the next little bit. The first one to be phased out on the old address will be Flash application support, which is set to hit on December 11, 2006.

Time to start tracking 'em all down ! I've got it on my To Do list for this week. I would highly suggest you do so as well.

Posted by Randy at 06:48 AM | Comments (0)

July 14, 2006

PDF Optimization Test

Feel free to ignore this post for the moment. I'm conducting a test with a series of pdf files, with a variety of meta data and security settings to see what gets picked up in each.

I'll report back on my findings as soon as the spiders have had a chance to grab all of the files.

PDF Test Case 1
PDF Test Case 2
PDF Test Case 3
PDF Test Case 4
PDF Test Case 5
PDF Test Case 6
PDF Test Case 7
PDF Test Case 8
PDF Test Case 9
PDF Test Case 10
PDF Test Case 11
PDF Test Case 12
PDF Test Case 13
PDF Test Case 14
PDF Test Case 15
PDF Test Case 16

Posted by Randy at 05:23 PM | Comments (0)

September 23, 2005

Google News (Part II)

As a quick follow up, Matt Cutts did post the facts about their new Webmaster Notification (pilot) program the other day. It's in his blog.

As I suspected, the whole thing about the host being sent an email came from their address being listed in the WhoIs contact information. Matt says in a comment that they've changed their internal procedures somewhat to keep this from happening in the future.

They'll try to find an email address on the site first, but they'll also look in the WhoIs. The difference being that now they will not send anything to either the Technical or Billing contact. Only to the Admin. If they can't find a valid email address they may also try generic email addresses such as webmaster@ and support@.

It's still a very good program IMHO. And yes Matt provided a sample of the letter they send out. And they do get into specifics. Also said that they only sent out email to 100 domains in the first run. So it's not a large-scale program by any stretch of the imagination at this point.

Posted by Randy at 12:04 PM | Comments (0)

September 18, 2005

Some Google News

There have been a couple of interesting developments in The Google Universe recently. One of which is quite noteworthy.

This past week Nick over at ThreadWatch broke the news that Google has begun a pilot program where they will (sometimes) notify a site owner when their site is in jeopardy of being dropped from Google's index for doing things that run afoul of their guidelines. The ThreadWatch report, with a ton of comments, is here if you're interested in reading the minute details.

Matt Cutt's of Google has confirmed the program.

There is a lot of hysteria by some in the above mentioned post. For those who have been doing things the right way all along have nothing at all to worry about.

FTR, the first thing that Google is apparently advising people about are the shady JavaScript redirects that a certain SEO company got hundreds (thousands?) of sites nailed for some time ago. I don't see this as a bad thing at all for Webmasters. There is no downside for them.

But it could be a very bad thing those SEO companies who are performing this type of rogue optimization.

Google has made it clear that their intention is to expand the program and let the webmaster know about other things like hidden text, hidden links, etc. All of which they will penalize a site for. To top things off, the first emails they are sending out is mentioning the specific things a site is being marked down for. Hopefully that continues to be the practice so that the average webmaster will no longer have to guess at what might be wrong.

Personally I think it's a very good thing. In fact, this type of feedback is something some of us have been asking for or wishing would be put into place for many years now. Hopefully, once it's up and running, Google will expand it further to include such specific information when someone sends in a question to them as to why a site has taken a significant drop.

Hey, a guy can hope! Even if it's something I very much doubt I would ever have to worry myself. I know how my sites are optimized after all. LOL I keep it all above board. I simply don't see anyone who follows common sense rules ever getting into trouble.

The whole thing does raise a very interesting question though...

Let's say you hired someone to optimize your site, turning over total control to the SEO company. You pay them, likely thousands of your hard-earned dollars, then a month or two later get an email --from Google no less-- that your site is going to be banned for something that the SEO company did.

How do you proceed at that point? If I were in those shoes I can't see letting the SEO company have access again to fix anything. They would have already lost my trust. Plus, if they're that stupid, I'm not certainly not going to let them have any control over my livlihood.

But what about the money you've already paid them?

Their efforts have obviously failed if your site is getting banned instead of climbing in the SERPs? And you're likely going to have to spend significantly more cash or a lot of your own time to fix their faux pas.

If you paid by credit card, and find out about the problem before 90 days have passed, the easiest thing to do would be to charge it back most likely. I can't think of much better proof to give to your credit card company --or a judge in court for that matter-- than something in writing from a search engine stating that your site was being removed.

Or do you confront them with the letter and ask for a refund? What happens if they refuse to give you one? Are clients going to start suing their SEO?

You'll hear me say this a lot... Personally I think it's a very good thing. A wonderful development. I can't wait for it to be expanded and become commonplace. Maybe the other major engines will start a similar program.

I also think that it's quite likely going to be a major factor in cleaning out the rogue elements from the SEO field. Those spammers and scammers simply won't be able to get enough to clear to stay in business after a time. Yup, that's a good thing.

One smallish point that some were worried about at ThreadWatch and other places it's being discussed...

Some have made claims of Google being a spammer by sending out the notices. That's complete BS as far as I'm concerned. The folks making those claims are quite likely the same supposed SEO's who are going to get nailed by this. I completely disregard those complaints. They're worried about getting sued and whining.

There is however a semi-legitimate complaint that someone reporting their Hosting company being CC'd on the email from Google. Matt is checking it out, but I'd be willing to bet that I know exactly what's happening.

Google is sending probably sending the notices out to the email addresses contained in the WhoIs information for the domain, rather than taking the time to see if an email address appears on the site.

If you've been paying attention the last several years you may have noticed that almost any domain that is purchased through a host almost always contains contact information for the host. If nothing else, their information always appears in the Technical Contact section; often it's also in the Billing Contact section; and sometimes it's even in the Admin section, even though this last one runs averse to the ICANN regulations.

So let's say that Google pulls up the WhoIs info for some domain to extract the email addresses. WhoIs records are notorious for having fake or bad email addresses in them. They're public records and can be scraped by the spamlist bots, so people often put totally fake contact info in there.

Knowing this, Google would want to send an email to each individual email address listed in the WhoIs, trying to make sure they get through to someone. And in the process the host ends up with a copy of the email.

Personally, I think it's a non-issue. Breaking Google's rules is not like breaking the law. So I can't fathom a host threatening a webmaster over a letter from Google. Especially not when they get used to seeing them.

Keep an eye on Matt Cutts' blog. He has promised to offer some more information on the pilot program as soon as he gets a chance to make sure he has all of the facts straight.

In fact, you might want to visit Matt's blog today! He's got a pretty interesting bit of information there as to how one should go about getting a previously banned site re-included in Google's index. Note all of the stuff he mentions as a reason a site can be removed or penalized. It was just posted today, Sept 18, 2004 in case you need to find it later in his archives.

Posted by Randy at 03:20 PM | Comments (0)

August 27, 2005

Powerhouse Linking Seminar

9-16-05 - I just received word from Debra that they're going to have to postpone the October Linking Seminar due to some personal issues. Not to worry, they're still planning to hold one, however it looks at this point like it's probably going to be next Spring before they can fit it into their schedules again.

You probably noticed that I've changed things around a little bit recently. The most important change is the graphic for Powerhouse Linking Seminar directly to the left.

Powerhouse Linking is an apt name, seeing as how the principles involved are Debra Mastaler and Eric Ward. For those of you who may not be familiar with the names, Debra and Eric are the top professionals in the world when it comes to creating and implementing link building campaigns that perform.

True powerhouses in word and deed. The proof is in the pudding with these two.

Not only do both regularly work with Fortune 1000 clients, but if you've ever attended any of the Search Engine Strategies conferences that Danny Sullivan and Jupiter put on, you will have noticed that one or both are always featured speakers when it comes to linking matters.

Yes, they're that good. To the point that it's getting quite difficult to hire either of them for your own link building campaigns because they're in such demand.

The link building panels at SES are always fun, but quite frankly they're not nearly long enough to allow for any depth or real teaching. There's usually only enough time to cover the basics and answer a few questions. Given the importance of link building --and more importantly doing the right way-- such a brief overview really doesn't begin to do the subject justice.

Eric and Debra have taken the decision to combine their rather notable talents in an effort to fix this oversight.

The Powerhouse Linking Seminars are two full days of face to face instruction with the elite of the elite. You'll find out exactly how they do it, what they look for, the tools they use, the secret places they find those primo links for their clients. There is going to be nothing held back. You'll even be able to go online to practice what you're learning, and ask any questions while doing so, right at the seminar!

As an extra bonus, Debra and Eric are bringing in Dan Theis, another well respected pro in the industry to answer any technical questions about linking (eg What redirects you want to avoid) and share additional resources.

If you're serious about your web-based business this one is a MUST. There is nothing else like it to my knowledge. And frankly if any others start to pop up, there never be another link building class featuring such credible instructors. You'll save money, time and headaches from the very first time you employ what you'll learn at this little beauty. Of that I am quite sure.

The only thing that I can equate the dynamics of this new seminar to is this...

It's like someone who is trying to start or grow their business into a world beater getting the opportunity to sit down to pick the brains of Jack Welch, the orchestrator of the successes at General Electric during their heyday, and Richard Branson, who started with a little mail order record retailer and built it into the empire that Virgin has become.

For two days.

Face to face.

Even if you knew nothing about business, can you imagine how much you could soak up from two such people who have had so much success?

That's the type of opportunity you're missing out on if you pass on attending the Powerhouse Linking Seminar. I would highly suggest you plan to attend if it is at all possible.

The date of the first seminar is October 27th & 28th. It is being held at the Sheraton Airport Hotel & Conference Center in Charlotte, NC.

Please, don't put it off too long thinking that another will come along in a month. At this point I don't know of a second date, and Debra would tell me if she knew. I hope there will be many more to come, but goodness knows both Debra and Eric are incredibly busy with their own businesses. So there may well come a day when time constraints simply don't allow for this type of intensive, hands-on instructive seminars.

Posted by Randy at 06:33 PM | Comments (0)

August 16, 2005

Search Engine Overlap?

There has always been something of sense out there that the search engines are falling all over themselves trying to deliver exactly the same 10 (or 15 or whatever) sites on their first page SERPs as the other engines do for identical searches. For example, since Google is supposedly the King of search, that Yahoo, MSN, Ask Jeeves and all the rest should be trying to duplicate what Google produces.

This accepted "truth" could not be farther from the truth. Each search engine has its own way of doing things and its own ways of measuring Relevancy, which is what they're all really shooting for.

To illustrate the point I'll mention a July 2005 survey conducted by Dogpile, the meta search engine. But first a little background and a bit of a disclaimer.

Dogpile is a Meta Search engine. What this means in English is that they get results from all of the big four (Google, Yahoo, MSN and Ask), including any paid placement ads, then try to amalgamate that data to "Show you the best of the best" as it were. Since they conducted the study some might ascribe a certain ulterior motive to the results.

I don't buy into this conspiracy theory. Sure Dogpile has motivations for studying and releasing such data, but it doesn't automatically follow that they would intentionally or unintentionally try to skew results of the survey. This is the second such study in the series --the last didn't include MSN since they didn't have their own search engine then. Frankly, the results provided in the Dogpile research match what I've seen with my own two eyes when I've looked at the question across some search phrases that are of interest to me. It also matches data compiled by others in recent years.

Okay, let's get on to the facts and figures. For your reference the full release of the study is here (30 pages) and the summary (1 page) is here. In the earlier report they released only the shorter Whitepaper edition. Warning: All are in PDF format.

I would encourage you to grab a copy of each for further study and to have a historical reference. Try to ignore the Dogpile hype, as that is to be expected IMO.

I rather like the fact that they released a fuller version of the second survey since it gives us a good overview of the methodology being employed, the analysis and even has a good discussion of the assumptions made and how conclusions were drawn. This alone gives the final data considerably more staying power in my view.

What the study proposes to do is try to set a benchmark of sorts in determining what percentage of the same sites rank on the first page (only) across each of the four most popular search engines for a random selection of search phrases. The current study researches 12,500 such search phrases. The reason they use only the first page results, regardless of how many results that may be depending upon each search engine's default, is that it's a pretty well proven point that the vast majority of searchers choose which site to visit from those first page listings. Note that the actual position on the page is not given any weight, only whether the site appears on Page 1 of the SERPs.

The results will likely surprise you, especially if you were under the impression that all of the search engines are chasing each others' tails..

When viewing the totality of all 12,500 searches, the same site appeared on the first page of all four engines a remarkably low 1.1% of the time. Remember, we're not talking about a #1 position correlating to #1 on another engine or anything of that nature. We're talking the First Page here. So it could be a case where a site was listed at #10 in one search engine and not at all in the others, or #1 in one engine and not at all in any others.

Instances where a single site appeared on the first page of three of the search engines for the same term came in at 2.6%, which is still less than the 3% noted in last year's report that only looked at Google, Yahoo and Ask.

11.4% of the time the same site would show up in two of the four engines for the same term.

And an astonishingly (to many) 84.9% of the time a web site's first page ranking was unique to a single search engine.

Let me say two of those again another way, just to stress the point.

If your site ranks on the first page in any of the big four, the statistical chances of it ranking well on a second, third or fourth engine are extremely remote, all things being equal. Almost 85% of all sites that rank on the first page of any engine will not rank that well on any other engine.

While the chances of even a well optimized site getting a first page ranking on all four search engines is a scant 1 out of 100.

What does it all mean?

Well for those of us who are in the SEM industry it means that very, very few sites are successful at getting a top (first page) ranking across all of the various search engines for the same phrases.

For searchers/users it means that they may want to use more than one search engine if they want to receive a better selection to choose from, because each engine is going to give you totally different results the vast majority of the time. Or use some sort of Meta Search Engine like Dogpile, which is of course the point they're trying to hammer home.

Now, what does it really mean for SEO's and SEM's?

It's quite simple really. The results of the study point out, in my most humble opinion, that far too many people have been focused on chasing the algorithm of XYZ search engine, thinking that will work for all of them. When nothing could be further from the truth!

Do you want to know how to rank well on all of the engines? I'm sure you do!

It's really quite simple, and is in fact the same thing some of us have been preaching for years now. Create good content that is geared towards your potential audience; you know, the folks that you want to visit your site because you know many of them will buy from you. Build your site to be the best it can be. The best it can be for users, not the search engines. And especially not for any single search engine.

Make your site a Truly Great Experience by paying particular attention to making it as user friendly as humanly possible. Then throw a well thought out link building campaign into the mix, one that will drive traffic to your site without the search engines, but will also give you an advantage with the engines link popularity measurements and you have it licked!

I know that sounds exceedingly simplistic, and that some of you will think it just has to be harder than this; that there has to be some secret I'm not telling you. You would be wrong.

I'm here to tell you that the above is exactly what I do and every single one of my sites is listed on the first page of at least 3 engines for the phrases that are most prized in their niche. Many of the sites are in that miniscule 1.1% that get listed First Page on all four of the major search engines. And even those where only three engines have me on the first page, the fourth engine always has me listed high up on the second page.

Go figure. Who would have thought it was really that easy and that simple? There's really no need to chase any search engine's algorithm, just build the best site that you can.

Oh, and to the above question about who would have thought it would be that easy, the answer is: Those of us who were around back in the Web's Stone Age when there were a lot more search engines to optimize for, but who realized that you didn't really need to do anything special to rank well across all of them. ;-)

Posted by Randy at 12:56 PM | Comments (0)

August 11, 2005

Search Goes Big Time

Do you know how you can tell that Search is really big business these days?

Forget about looking at profit and loss statements. Forget about looking at most everything in fact. The way to tell is when the major search engines start suing each other and making moves that are obviously designed to head each other off in some market or the other.

Within the last couple of weeks we've had Yahoo! jumping into the "Adsense" market.

More recently China's supposedly largest search engine (supposedly because I can't find any independent data) Baidu opened on the NYSE with a big splash. It ran up over 350% on its first day of trading. It's apparently known as "Google of China" and in fact Google holds a minority stake in Baidu. Then days later comes the announcement that Yahoo! has purchased a 40% of the 2nd largest Chinese search property Alibaba, hot on the heels of the Google/Baidu announcement.

The kicker is that Microsoft and Google are in quite a tussle (read: Lawsuit) over Google's hiring of Kai-Fu Lee to head up it's China R&D operations. Lee was under contract to MS and had a non-compete clause. Apparently things are going to get nasty since Microsoft found documents on one of former computers on the MS campus where he was corresponding with Google.

It's all very messy from what I can tell. And going to get messier.

When you get three behemoths, with basically unlimited funding, locking horns somebody is going to get hurt. Badly.

But it's a clear sign that Search is now a very big business.

Posted by Randy at 11:37 AM | Comments (0)

August 05, 2005

From the morning mailbox

Some interesting things flew into my mailbox overnight. So here they are:

Google Local

Do you have a bricks & Mortar store that you've always wanted to get into Google Local so that folks from your area can find you? Now you definitely can and it won't cost you a dime.

Two ways to go about it.

If you have 10 or more store locations they want you to use their Froogle feed capability. The details are in the Froogle Merchant Center, and if you're not familiar with creating a Froogle feed, some general instructions are here.

If you're a small business with less than 10 store locations you can submit your info directly to the Goole Local Business Center.

With both you control exactly what is displayed.

Yahoo Audio Search

I got an email last night from Toby Elliot of the Yahoo! Developer Network pointing out that Y! now has Yahoo! Audio Search in beta testing. Basically it'll allow you to search through the 50 million plus audio files that are out there on the 'Net. Music, speeches, interviews, sounds effects, podcasts, etc. If it's audio they want it!

If you're a producer of audio files you can even submit them to be included via Yahoo! Media (RSS) Feeds.

I got the email from Toby because I'm involved with the API side of things and in conjunction with the beta search launch they're also launching an Audio Search API. I've not tinkered with it at all, but from everything I read this morning at the above link and the Audio API Documentation pages, this new API is actually going to be more powerful and more flexible than the front end is!

This last bit is a very important development. To my knowledge it's the first time an API has been issued by any of the search engines that actually gives developers to extend their applications beyond what the search engine itself already offers. Sure other API's could be developed that gather information and display relationships/information differently, but never before have you been able to do something with an API that couldn't be done via normal search.

Cool beans Yahoo crew! Well done!

Hopefully others will follow your lead on the API front.

Posted by Randy at 08:02 AM | Comments (0)

August 04, 2005

Quick Hit

July 31, 2005

Common Link

This post is to announce that the Common Link tool has officially progressed beyond Beta status. Woohoo!

First, a bit thank you to the guys and gals at Yahoo! for their support. After some mail back and forth (well done on that Debra!) Yahoo have decided to allow us more queries than the normal 5,000 that comes with their API license. They've graciously upped that to 25,000 queries per day.

Note that unlike Google's API license, the 25k counts against the server IP number that the script is on. So all of the Yahoo API scripts on this particular server will count against the total. But none of the rest of them are nearly as query intensive as Common Link is.

Let's take a moment to explore what Common Link is, what it was created to do, how it can help, etc.

In a nutshell Common Link helps you to analyze the backlinks that are helping some sites rank well. First and foremost it gives you an easy way to get a list of the sites linking to the Top Ranking sites for any keyword phrase on Google or Yahoo. As a webmaster it's pretty helpful in that you can quickly identify sites that are already linking to sites in your niche. (read: Your Competitors)

Given that fact, those should also be sites that will accept the premise of linking to your site too, as long as it's up to snuff.

The second part of the equation is a bit of theory, which there is no way to prove or disprove. Though I personally think it's quite true. The theory goes like this...

Not all links are equal. Links from some sites --commonly called Hub/Authority sites in the SEO/SEM industry-- carry more weight than the average link. Because they've been identified by the search engines (via clustering technology?) to be good authorities on a given subject.

Add another loop to this base theory and it makes sense that the top ranking sites may have more backlinks from these Authority sites. More links from powerful sites = Better Rank.

Okay, got that?

What Common Link does is looks at the backlinks for the Top 10 (or whatever) sites for a given keyword phrase and compiles the data in a way that is different from anything I've ever seen before. The script looks at those linking pages to figure out how many of the top ranking sites that particular page links to.

The idea being that if Page X from Site Y links to four sites who all happen to be in the Top 10, it may well be the case that Site Y/Page X has been determined to be an Authority on the subject. And that link is helping those four sites to rank as well as they do.

Mix, roll and extrapolate. If Page A from Site B also links to multiple top ranking sites --maybe the same four as above, but not necessarily-- it may also be an Authority site.

So there's the brief description of what Common Link does. It tries to help you identify which of those other sites may be Authorities, which of course you should try to get a link from. And if you can identify more than the current top ranking sites do, those backlinks should help you (somewhat) to obtain a good ranking for your own site!

Hopefully, that all made sense. Like I said, there's a lot of theory in there.

Now, here's the caveat...

There are no filters whatsoever being applied to the Common Link results. So it's still going to require a good bit of human judgement to determine if a given site is a Quality site or just another fake directory or scraper site. You'll want to avoid those like the plague!

But in the meantime, if you can get as many or more links from those Authority sites as the competitors, it should put you on equal footing with them where this one ranking factor is concerned. In my very limited experience during testing of the script and its results, it certainly looks like the theory will prove itself out to be true. Those backlinks from Authority sites do indeed appear to carry some extra weight.

Something to note: Common Link needs to run several hundred queries against my Yahoo API license each time it is run. So it's quite possible that you may get an error if the script starts to be heavily used. Even with the additional queries Yahoo have granted.

That's the reason why when you get to the last step it allows you to save the data as a .csv file for later viewing. If you save that file you won't have to run the data again the next time you start to work on it. Plus you'll be able to keep notes in the Excel file. eg what sites you've decided to approach, the date you asked for a link, the results of your request, etc.

Also, because of the license limit, I'll be making the code available for download and installation on other people's server. I need to find some time to clean up the code a bit and comment it much better than it now is before I make it available, but that's the plan. When that happens I'll be looking for people who want to mirror Common Link. The sole requirements are that your server must have PHP installed and must have DOMXML installed and enabled. You can tell if that's the case by running a phpinfo(); query. If it's enabled you'll see a box for DOMXML giving the details.

Another approach, which I'm considering if I can find anyone who wants to do it, is to convert the current PHP code into Java or VB script. With this approach the script would be a self-contained module that you can run from your own computer. Which means that the daily license limit will count against the 5,000 queries per day that is allowed for your computer's IP number.

If anybody out there is interested in converting it to something that can be run straight from a users computer, contact me. I would actually like to convert several useful API scripts into self-contained applications. Java would probably work best, since it would run on either PC's or Mac's. Unfortunately I'm well out of the Java development loop these days, and simply do not have enough time to take a refresher course.

Anyway, there it is. Have fun with it!

Posted by Randy at 10:03 AM | Comments (0)

July 02, 2005

Click Fraud Goes Mainstream?

According to Reuters in this story Google is being sued in Federal Court over their handling of Click Fraud. The plaintiff in the case is a company by the name of Click Defense, which sells software to help online merchants detect and fight click fraud.

Because of their business, some are saying that the suit, which is also trying to gain Class Action status, is somewhat dubious. After all, it gets Click Defense some free advertising, and if there were no Click Fraud to deal with Click Defense would be out of business. In the various other stories I've read about this suit the folks at Click Defense say their main reason for filing is that they want the courts to force Google to release the real numbers detailing exactly how prevalent Click Fraud has become.

I'm willing --at this point-- to give Click Defense the benefit of the doubt and hope that some good comes out of it all. Hopefully more of the mainstream press will get on the bandwagon and that pressure will force the PPC Providers to make changes in how they report and share information. Keeping it all a big hush-hush, no-I'm-not-telling, secret --especially from the people who are paying for those ads-- is not only silly but leads to a lot of angst and mistrust.

Generally speaking, most in the industry ascribe a 20% number to how much Click Fraud is out there. That's one in five. Click Defense says they've seen it be as high as 38%. Personally, when I ran a lot of Adwords ads and set up my own little Click Fraud detection for a few sites the Click Fraud number came in as around 1/3 of all clicks were fraudulent.

The problem, as Click Defense right puts it IMHO, is that neither Google nor Yahoo! (via Overture) will release any numbers about the amount of Click Fraud they detect. They refuse to release this info for obvious reasons, since the mainstream press would probably latch onto those numbers and it would be quote harmful to their reputation. And I'm quite sure if they did release their internal data the number would still be on the low side. If they were catching all of the fraud, then companies like Click Defense, aQuantive and DoubleClick would not exist. There would be no need.

The new suit follows on the heels of another Click Fraud suit that was filed a few months ago in the State Court in Arkansas, as outlined in this AP story. The earlier suit takes on Google, Yahoo, AOL and Ask --all of whom display PPC ads-- and was filed by Lane's Gifts and Collectibles, which is a local gift shop that advertises on the Web.

Frankly, this one is more of a feel-good story since the plaintiff obviously isn't looking for free advertising. But by the same token, they probably don't have the same resources or information to back up their claims that Click Defense has available. Plus the notion that all of the defendants are involved in some type of conspiracy is a bit much in my opinion. I honestly don't think any of them are actively participating in or allowing Click Fraud. They would rather that there were none, I'm quite sure.

The Arkansas group is trying to get their suit moved to the Federal Court system, but you can bet that all of the defendants are going to challenge that.

Is this the first shot across the bow for the PPC Providers? Can one or both of these lawsuits help in the effort to force PPC Provider to supply more information to their advertisers --at the very least-- so that the individual advertisers can become more proactive in fighting Click Fraud?

Personally, I don't care if the Click Fraud number is the generally accepted 20%, Click Fraud's number of 38% or my number of around 33%. All of those are waaaaaaaaaaaay too high. Even given those numbers, I would hazard to bet that if you asked the average PPC Ad Purchaser if they'd ever received any type of refund, let alone a 20% credit every month for fraudulent clicks, nobody would say they had received such credits.

The couple dozen PPC advertisers I did ask have never knowingly received any type of rebate. Ever. So it may be a case that the only advertisers who are consistently getting refunds for Click Fraud are those squeaky wheels who are doing some type of Click Fraud Detection on their own, outside of what the Providers supposedly offer. Given that, it would make sense that none of the numbers are close to matching.

Putting those numbers into perspective, and using credit card merchant accounts as a comparison, if you were a merchant and had a 20% fraud rate for any month, you would be dropped by your merchant bank so quickly that your head would spin. In fact, if you had an ongoing fraudulent rate of as little as 2% you're likely going to lose your ability to charge fees to anyone's credit card. Forever.

No matter how you slice it, Click Fraud is rampant.

Until the PPC Providers take it seriously enough to include the people who pay for those ads as part of the Solution, it's going to remain a problem. Being so secretive about how big or bad the problem is, as they've always been, does nothing but convince the average person that they've got something to hide. Something sinister.

People are going to fear the worst if you give them no choice in the matter.

Posted by Randy at 04:21 PM | Comments (0)

June 23, 2005

Yahoo! looks to go head-to-head against Adsense

Sure Yahoo! owns Overture, to compete with Google's Adwords, but they've never had anything like AdSense where they can create a network of advertiser sites.

But news comes from Marketing Vox that Yahoo! have teamed up with Revenue Science in a beta test before launching their new system onto the unsuspecting world.

It looks like Google are going to have a lot of competition in the area where the vast majority of their income is derived from. MSN is said to be working on at least an Adwords equivalent already, so it probably won't be long before they enter the fray into the PPC advertising market.

It's bad news for Google, because of the competition factor, but I'm not sure yet what it'll mean for Webmasters. Either from the advertisers point of view or for those Webmasters who would like to place Adsense-type ads on their sites. Hopefully the added competition will bring the rates down a bit for advertisers. And hopefully more competition on the provider (Adsense) side of things will mean that Webmasters get a bigger cut of the revenue generated by each click.

It's too early to tell how it's all going to shake out though. I really don't expect any major movement if/until Microsoft enters the fray.

For surfers it can mean only one thing... You're going to see a lot more ads on a lot more sites you visit. That will likely bother some, but it doesn't me. Mainly because I have all of those ads blocked at the "hosts" file on my computer. LOL So I never even see them in the first place!

Posted by Randy at 03:11 PM | Comments (0)

Recent MSN News

MSN looks like they're really going to step up to the plate.

In a recent blog entry the touched on how determined they are to make sure their SERPs are delivering the most relevant results possible. To the point of not just wanting what a person would feel to be the most relevant site ranking on the first page of the SERPs, like some other search engines currently do, but to make sure that most relevant result ends up in the first position of the SERPs.

Also noted in the same blog entry is that they've been working to improve their Local Search.

And lastly they've added some advanced operators into the mix: FileType: InAnchor: InURL: InTitle: and InBody:

Something new to play with! Now if someone could just convince MSN to offer an API to use in connecting to their data...

Posted by Randy at 02:36 PM | Comments (5)

Google Training Docs

Some interesting reading for you today if you're a glutton for punishment. LOL

Just kidding. It's really quite interesting though not exactly earth-shattering news. And can give you a good insight into what Google considers to be a good or bad site.

Recently Henk van Ess stumbled across and published some internal documents that confirmed what most have known (assumed?) for a long while now. That Google actually hires people around the world to help it gain some human feedback on its relevancy. Googleguy even confirmed that the program and documents are real.

Those who watch this type of thing closely already knew that Google hired people to offer such feedback, as you'll see the occasional listing from third party employment places that state as much. It was also known that the eval.google.com sub-domain actually exists, even though you can't really get to it with your browser. That's because of their internal security procedures that protect this sensitive area.

But the internal documents are somewhat interesting.

This first document is sort of a Welcome letter from December 2003 that explains how the testers can/should use the Random-Query Evaluation works. It goes into how the testers should apply different ratings, what each means and makes mention that sometimes very general queries could have totally different meanings to different searchers.

The second link I'll provide leads to the Evaluator's Spam Guide, which purports to help the tester determine what is and what is not spam. It also goes into redirection schemes a bit, which is something some struggle with correctly identifying as either okay or a no-no.

Lots of other stuff over on Henk's blog if this sort of thing interests you. Just scroll back a week or two.

Posted by Randy at 02:04 PM | Comments (0)

June 04, 2005

Google Sitemaps

I should sub-title this one Paid Inclusion Without Paying, since that's basically what it amounts to. ;-)

In case you haven't run across it yet --and most haven't since it's still in Beta-- Google Sitemaps is a pretty cool new method you can use to let Google know about all of your pages so that they can spider them.

Seriously it's much like the the Paid Inclusion services at Yahoo! and other places, with two major differences.

1. Google Sitemaps doesn't cost you a dime.
2. Because of this you can submit every single URL of your site instead of having to choose just a few pages that you hope will convert well for you.

Now you're probably wondering why Google would do this, and more importantly why they would do it for free when others are charging for the same type of service. Well, it's a culture thing.

Hunh? Culture?

No, I'm not kidding. Google, pretty much since its inception has had a culture inside the company that they want to collect information about everything, becoming essentially a huge catalog of every scrap of information on the Earth. This has been their defining principle since the very beginning, and continues to be what inspires most Googlites. This is why you often hear about Google doing things like scanning entire libraries at their own expense.

Google and everybody else knows that there is a huge Undiscovered 'Net out there that they cannot spider for one reason or another. Mostly because of technical issues that their spiders cannot deal with. It is not unusual to see very popular, well respected sites that only have 50-70% of their pages indexed. In Google's eyes this small fact is a crime.

Most times this isn't a huge problem for webmasters because there will be other pages that are similar enough in concept to still pull the traffic the webmasters want to receive.

So it's not a big deal unless, like Google, your aim is to know about everything about everything. For them, missing even miniscule, perhaps inconsequential information equates to failure. If for no other reason than that they're not following through on the entire purpose of Google existing in the first place.

Yes, as a webmaster this concept is tough to wrap ones head around. Webmasters are concerned only about getting quality traffic and converting the real people behind that traffic into customers. Most search engines are concerned with driving traffic in such a way that they can make a profit from their advertisers, and to an extent Google shares this concern.

But when you get down to the base level of why Sergey and Larry first created Google, they wanted to father the foremost encyclopedia the world has ever known. They wanted to know Everything about Everything.

To do that, they need to find some way to crack the vast Undiscovered 'Net.

This is what Google Sitemap is all about. What it was designed to do.

A side benefit for we webmasters is that we now have an easy way to let Google know about every page and file on our site(s), even those which have been troublesome for Google to find and crawl.

Paid Inclusion without having to pay.

I like it!

Posted by Randy at 08:57 AM | Comments (0)

May 27, 2005

The Changing Landscape of Search

There has been much speculation over the past year or so as to what the Search industry will look like when it all shakes out. The real answer is that no one knows for sure what the landscape will look like 12-24 months down the road.

So rather than take a guess --yes I have an opinion, just like everybody else-- let's take a look at the pro's and con's for each of The Big Three.

Google

Google is of course still the reigning leader in search engine traffic, though they are nowhere near the dominating force they were less than 18 months ago.

Pro's: Google had a large head start while they were the only major search engine for years. Their name is basically synonymous with Search for many users. After their IPO they are cash rich, especially the employee's, who all had stock options as part of their compensation package.

Con's: The vast majority of Google's income is produced by selling advertisements via Adwords. Right now they hold an even more dominant position in this area than they do/did in Search. Yahoo! is trying to catch up with their Overture ownership and Site Submit, but the general sense is that Google owns the pay per click (PPC) market for the time being.

The problem with being so dependent on a single source of income is two-fold.

First, there is beginning to be a more generally recognized perception that Click Fraud is rampant in PPC advertising. My sense and experience agrees with this perception. Click Fraud is very difficult to detect unless the fraud is being done on a massive scale by an individual or oganization who doesn't understand how to hide themselves. Small time Click Fraud is almost impossible to detect unless you already suspect it. However a lot of small time Click Fraud --ala AdSense advertisers click on their own ads or having others click on them so that their own cash register rings up a princely sum-- is still a very large ghost in the closet to deal with.

Second, being so very dependent on a single source of income has a very definite potential for disaster. This is true of any business, not just Google and not just on the Internet. Remember the old axiom about not putting all of ones eggs in a single basket?

Whether this potential disaster comes from mainstream news organizations finally publicizing the scope of the Click Fraud problem; or if it comes from some competitor with deep pockets deciding that it's worth a short term sacrifice in order to put a major dent in Google's ability to maneuver in the longer term.

Simply put, a bidding war for their Adwords business has the potential to have an effect on a lot of the flexibility that Google now has.

Another con for Google was formerly a big pro. Being the only game in town for so many years ended up painting a huge target on their collective backs. This issue of being the sole focus gave the less-than-scrupulous folks out there plenty of time to figure out ways to fool Google and scam Google users. Google has been struggling to rid their index of these scammers and spammers, and to regain their once renowned search relevancy for well over a year now. Frankly, by all outward appearances, they don't seem to be gaining much ground.

Yahoo!

Since getting back into the Search game Yahoo! has been making some pretty significant strides. Not only is their Search relevancy much better than it was a year ago, but they're also making some inroads into the percentage of searchers who use Yahoo! as compared to Google. Based upon all of the sites that I watch the stats on, Yahoo! is up a good 10% on the Traffic Delivered scale from February 2004 when they re-launched their own search product.

Pro's: Though not many seem to realize it, Yahoo! makes a lot more money on Internet endeavors than Google, and has for years. In fact, they even make more on Internet Advertising than Google does by a large margin according to a Wired magazine article I read some months back.

They've got a good portal and are a respected brand. They also have strategic alliances all over the place that none of their competitors can currently dream of matching. Additionally, with the growing perception that Yahoo! is much more Webmaster-friendly than the standoffish Google; and with new releases like their MyYahoo! RSS feeds and their API back end, Yahoo! stands to gain a lot of favorable reaction and word-of-mouth advertising from those who have to deal with them the most: Webmasters and other Internet Professionals.

Yahoo! is old school. They're a huge corporation with deeper pockets than Google, and they simply go about their business quietly and effectively. Better yet they don't seem very interested in getting into hand-to-hand combat with either of the other major players.

Con's: Being a Portal is both a good and bad thing. I honestly don't think it's true, but there seems to be a perception out there that Yahoo! doesn't care about Search as much as Google does. I don't personally find that to be the case. In fact, I've found Yahoo! to be easier to work with when something goes wrong (by error, not by design) simply because they don't seem to have the same "You need us -- We don't need you" attitude that many report as infecting the Googleplex.

That said, Yahoo! is still seen as something "less" by most search marketers, mainly those who haven't been paying attention. But they're still thought to be less of a threat to Google than MSN, for what are fairly obvious reasons.

MSN

MSN is truly the sleeping giant. Not only do they have basically an unlimited warchest to throw at their entry into Search, but they are under a mandate from Bill Gates to catch up, and catch up quickly. When Bill sends out something like that, everyone in Redmond is said to feel the hair on the back of their necks stand at attention.

In the relatively short time they've been running their own search engine they've made some pretty significant strides. They're certainly not there yet, but are getting better pretty much every week.

Pro's: Money, and lots of it. And the willingness to spend lavishly if need be. MSN also has a lot of talent to draw upon, both in house inside Microsoft and via outsourcing or purchasing smaller companies if need be.

Additionally, MSN seems to be taking what can best be termed an Anti-Google approach to others who have a vested interest in Search. Early on in the creation of MSN Search they started out listening to Webmasters, Marketers and users. Amazingly they have continued to keep these lines of communication open.

No matter how you look at it, MSN has an ace-in-the-hole that frankly no one else can begin to trump. Their parent, Microsoft, just happens to be the Operating System of choice on a crushingly large majority of the world's PCs. Anybody who is under estimating this single fact should move to another universe.

The basic premise goes like this...

When Microsoft releases their new Windows update (known as Longhorn and expected sometime in late 2005) it has already been announced that Windows will integrate search boxes into everything. Certainly all of those often used MS Applications like Outlook, Word, Office, Excel, etc will have a search box.

Ostensibly these search boxes will connect to MSN Search. The Convenience factor alone stands to pump up the volume at MSN a good deal, likely bringing them into striking range of both Yahoo! and Google.

Now, the nay sayers will tell you that this will never fly and that Microsoft will be sued by Google, etal. Which is absolutely true.

The problem with the premise that getting sued is going to stop MS from doing exactly what they've stated they're going to do is that all of the folks who think it will evidently missed their Internet History lessons.

When MS introduced and integrated IE into Windows, making it the default browser for every Windows user, Netscape owned a solid 85% + of the browser market. By the time Netscape had (sort of) won their suit, the percentage was exactly the opposite, with IE owning 85% of the market. Netscape may have won that battle. But at the same time they were winning the battle the lost the war. Big time.

Expect something of a repeat. No, Google isn't Netscape, but Google also has a very soft underbelly that MSN can attack at the same time they're hitting them on the OS ownership side of things. Remember Google's reliance on Adwords dollars I mentioned above? Do you think for a moment that Gates & Co won't be going after that business, knowing how crucial it is to Google?

I have no doubt they'll go after it and go after it with gusto. Probably keeping less for themselves just to drive Google nuts. MSN's Adwords/Adsense clones can actually be a loss leader for them, which is something that Google simply can't afford to to for long.

Con's: MSN got a very late start in the Search game. They also have some baggage coming into it that they need to dispel, namely being Big Bad Microsoft. Many also have what could turn out to be very unrealistic expectations concerning what MSN can do with all of their talent, technology and sheer dominance of the PC market. The unrealistic expectations could cause some to think that MSN is failing, when in fact they're simply doing things on their own time table.

----------

So there's our short(?) look at the pro's and con's of the each major players.

What does it all mean?

I'll let you decide for yourself. Each of the Big Three has a lot of work to do, each in different areas.

Personally I would like to see it all end up with there being three engines that have roughly the same traffic base to work with. I kind of doubt it'll end up that way though.

Two of the three players seem to really have it in for each other, so it could easily turn into a blood feud. I would almost like to see what their advertising departments could come up with if it ends up getting into something like the beer ad wars between Budweiser and Miller. That would be a hoot IMO!

Take your pick as to who is going to win the all-out war between Google and MSN. Then try to figure out what happens with good old Yahoo! as who simply keeps plodding along, staying above the fray and out of harm's way.

Posted by Randy at 08:04 PM | Comments (0)

May 07, 2005

Yahoo! Patent Application

Just a heads up since I'm short on time today...

If you haven't already you may want to read the latest Patent Application filed by Yahoo!

This one was filed on April 14, 2005 and is quite interesting. It's looooooooooong though, weighing in at 23 pages when printed.

I'll be posting what I take from it in the near future, as soon as I can find some time. I've already read through it several times, so now it's just a matter of gathering my thoughts, condensing it down and trying to figure out how to put it into plain English.

Posted by Randy at 12:22 PM | Comments (0)

May 05, 2005

Yahoo! encourages web businesses to Think Big!

Yeah, yeah... I know it's been too long since I've last written here. Sorry, but I've been busy !

A quick hit today that is time sensitive. I'll try to catch up on all of the other goodies over the next week or so.

Yahoo! has partnered with Richard Branson of Virgin (Records/Airlines) fame in a new contest.

They announced a contest on April 28th to give away 10 million ad impressions across their network to one lucky entrepreneur or small business. The deadline for submissions is May 20th, so if you're interested make up your mind fast !

The rules/criteria for entrants are here and there are several, including the business must be US-based, can't be "seasonal" and must be able to conduct their business from Times Square (or similar location) for a day, can't display any ads from Yahoo! competitors (ie Adsense), etc.

Read 'em all to make sure you qualify.

If you've ever wanted oodles and gobs of free publicity for your site this is just the ticket! So you'd better enter quick. The deadline is fast approaching, and it looks like they're going to name the winner pretty quickly too since the rules state that you have to be able to travel in the June 12-16 time frame.

Posted by Randy at 01:52 PM | Comments (0)

April 12, 2005

To API or not to API...

As you've read recently, I've become somewhat enamored (or enamoured for my Brit friends) with APIs. I've actually been discussing creating API applications for some time now with several close friends. Both to get an idea of what might be needed and if my thoughts on the whole API movement makes sense.

(The considered opinion across the board so far is that it does make a lot of sense and is a wonderful opportunity btw.)

Anyway, Debra O'Neil-Mastaler of Alliance Link and I have been talking about developing some applications that could benefit those who are trying to gain a grasp on certain aspects of getting a site --that is worthy of ranking well-- to the point that it can actually rank well.

The problem being that for the average business owner who wants to take their business to the Web, it's a very difficult and often confusing process. It's difficult enough for those with lots of experience in marketing on the web! Partly because there is so much information out there that is frankly complete and utter BS. Partly because "the competition" often has a several years head start.

Note that neither Debra or I think there is any formula which will automatically cause a site rank well, as it should be. And even if there was, such a formula would most definiately not necessarily be the best thing for the users of any site, which simply has to be the goal of every webmaster.

Even though one has to keep the end user firmly in mind, one has to also at least keep the search engines in the back of their mind when developing their online presence. And there are certain things which end up causing newer sites --that may in fact be much better-- from being able to compete with sites that have more age on them.

One of the things that is very difficult for new webmasters to get a handle on is the fact that links pointing to a site can have a drastic effect on where a site ranks. Over time, a truly worthy site will collect these links. No doubt about that. But by the same token, new webmasters simply don't have the expertise or experience to collect and analyze this type of data. And many give up in total frustration long before their site has enough age on it to be on an equal playing field with older sites.

The information to build these links is out there, but it's confusing, difficult to understand and very hard to come by.

So we had an idea to develop an API tool called Common Link (beta) which might help to simplify this process.

In a nutshell the concept behind Common Link was to look at the links that are pointing to the top ranked sites for a given search phrase. But instead of having to process the information manually, the idea we had was to look at how many of those top ranked sites have "common links". Or in English how many of those top ranked sites had links pointing to them that were housed on the same external sites.

The theory being that not only would the tool help someone with a new (worthy) site easily identify other sites which accept link requests for that market, but it might also help them to identify those sites which are considered Authority sites for the market.

Long story short, as long as the content of your site measured up, Common Link could give a webmaster a leg up on the linking side of things, hopefully assisting them in getting their site ranked highly enough to actually compete with these older sites in a shorter period of time. Before they became disillusioned and let their site expire.

The theory makes sense and I believe it's sound, even though it's difficult to test. I've looked at probably 50 different markets over the past few days and it's rather amazing how many of the Top 10 sites in most markets share common links from other sites. So it's our belief that the tool could be quite helpful.

But we've run into a slight problem.

The current API license that Yahoo! offers has a rate limit of 5,000 queries per day from any single server's IP number. That sounds like a lot, but it's not if: a) One person/company develops several API applications that are housed on the same server; b) Several someone's who happen to occupy the same server are developing Yahoo! API applications, since they all count, enmasse, against the same server IP's daily allocation; or c) You develop an application like this one that has the potential of doing a relatively massive amount of queries to the API from a single use.

That last one is the tricky bit.

You see, what the script does is query Yahoo! to see what sites link to Site A. Then it compares those against the links pointing towards Site B, Site C, etc, etc.

Normally it's not much of an issue since the "average" site may have anywhere from 50 to 200 links pointing to it. But if someone happens to run a query of a relatively competitive phrase, it's not at all unusual to have thousands or sites that share links to two or more of the top ranked sites.

In testing, even if we restrict the number of links to look at, it's still quite possible to get a list of 3,000 or more "common links", or sites that link to two or more of the top ranked sites. This of course produces a huge number of queries being sent to the Yahoo! API. And quickly puts the entire server over its daily rate limit.

Biting the bullet, I decided to write the email address provided by Yahoo! for those who wanted to create a Commercial application based upon their API technology. We really didn't want to have to commercialize the script, but because of the above scenerio's it looked like there was not going to be a choice in the matter.

As it turns out though, the Yahoo! API Developer's area does not yet have any method of overcoming the rate limit problem. They don't offer any sort of Commercial license at this time. The form letter I got back yesterday said as much.

So at the moment we're at a loss as to whether to develop this tool, or any tool using the Yahoo API.

The Common Link tool is certainly a No Go if a solution cannot be found, simply because there's no way to head off those competitive industry searches on the front end. But if the rate limit is going to be a problem, and it honestly looks like it's going to be at this point, it simply wouldn't make much sense to set about developing multiple tools for people to use. No matter how useful they may be.

It's a continuing story at this point. We're going to try to get in touch with someone at Yahoo! to see if there are any options. The whole API thing is so new that we may well have hit upon something that simply wasn't considered when the original license terms were developed.

More as I know it...

Posted by Randy at 09:08 AM | Comments (0)

April 08, 2005

Yahoo! Modification Date

A question was raised by Kalena Jordan of Search Engine College the other day on Jill's Forum regarding how to figure out when Yahoo! last spidered a site or last updated their index.

This is one bit of information Yahoo! hasn't included in their SERP display for a long time now, though no one could ever figure out why. It never seemed to be sensitive information so not including it as a norm never made any sense to me.

Oddly, I'd just started playing a bit with Yahoo! Developer stuff a few days earlier because I plan to create a few little API applications for fun and hopefully to help folks out. In doing this I happened to notice that the API version of Yahoo! searches does in fact include a "ModificationDate" field that gets returned.

Technically, this ModificationDate isn't intended to show the last time Slurp came by for a visit. Instead it shows the time and date (in unix timestamp format the actual file was last updated or modified. In other words, the time and date the server the page resides on supplies. Or at least that's the way I read it in the documentation.

In practice though, this info is still pretty useful.

To begin with, sites that use server side scripting (e.g. PHP, ASP,etc) are always going to send a current day date/timestamp even if the files do not contain any scripting. That's because the server still has to parse through the file before delivering it. So these files are always going to show as being brand new even if they've not technically be modified in months or years.

If you have file types that will report the true date of upload (e.g. plain HTML files) it's a bit more problematic if my understanding of ModificationDate is correct. Still, if you find that your server reports an old date you could always re-upload exactly the same file to your server, thus updating the date/timestamp the server has stored, and check the tool in a few days to make sure Slurp is coming by to grab the file(s).

Anyway, I'm rambling. Back to the main point of this entry.

Since Kalena asked, and since I was feeling inquisitive, I spent a few minutes --literally less time than it's taking me to write this post-- to tweak their example API code and register an application with them that will show this ModificationDate information. Simply because it is interesting and maybe even valuable if you ever run into a situation where your Yahoo! rankings go wonky and/or you need a quick way to figure out if Slurp has been by lately.

Using a tool like this is much easier than pouring over log files, that's for sure! Plus even if you see the spider hitting a certain page in your logs you still couldn't be sure that Yahoo! updated it in their index unless the content of the page had changed pretty significantly.

I'm going to release this little tool with a GPL (GNU) license so that others can slap it up on their site if they want to. Because of the way Yahoo's API works --they use REST, not SOAP-- your server will need to support DOMXML via PHP. That means your PHP installation will need to be at least version 4.3.x and be compiled with DOMXML support.

You should be able tell if your server supports this by creating a phpinfo() file. Or simply unzip and upload the mod_date file and try to run it. If DOMXML is not supported you'll get an error right away telling you that one of several built in functions is not available.

Or if you would prefer to simply use the version on my server, the live mod_date script can be found here. Yes it's free to use. You'll probably notice that I pared down the information it displays a bit, so it doesn't show things like the title of a page or even a snippet. That seemed to clutter things up so I simply removed them from the display portion of the equation.

Part of the reason I'm making the code available for others to upload to their site is because there is some confusion in the way Yahoo! has their application license worded.

Unlike Google and their API license, Yahoo! has you actually register an Applicaiton ID with them. That's how they identify the scripts. Then instead of the 1,000 queries Google allows via API, Yahoo! allows 5,000 queries per day, per IP address. They don't state clearly if it's the IP number of the server that is recorded against the rate limit, or if it's the IP number of the person using the application.

I could see it being either way, so I'm going to try to get some clarification on that point. If it's 5,000 queries per user it's no big deal. It would be very difficult for one person to send through that many queries in a day. On the other hand if it's 5,000 queries per server, that's a whole other story.

If it is keyed to the server IP as I suspect, it would be better IMHO to set the limit as per day, per application, per IP. I can envision myself and others creating all kinds of little applications. The problem as I see it is that while each of these applications would not even begin to approach the daily query limit, when you group together the number of queries from a couple of dozen of applications on a decently active site/server, the combined total could easily top the daily limit.

Especially when you factor in the idea that many of us use Shared Hosting, where there could be hundreds of sites on a single IP number.

My way of limiting queries would certainly encourage API Developers to create more, and more useful applications. So hopefully Yahoo! will consider such a change to their license if they indeed count queries against the server IP number.

The reason I'm releasing the code is that even if the version on my site runs out of queries for a given day, the same script being run from a different server could have another 5,000 queries left to go.

So there it is. Enjoy the little tool and look for more to be coming in the very near future. Also, please let me know if you upload it to your own site or if you improve on the concept. I'll certainly link to your version from here somewhere to give people options.

Posted by Randy at 10:51 PM | Comments (0)

April 02, 2005

The New (Marketing) Frontier

There is a very interesting development that has been trudging along in its infancy recently that is about to explode on the 'Net in my opinion. Collectively this new opportunity is known as API, or Application Programming Interface(s).

What are API's?

Basically they are nothing more than a bit of coding that can be used to connect to databases in order to retrieve information. Nothing too complicated actually. Rather similar to how you can use PHP in conjunction with MySQL databases to retrieve and display information.

The major difference is that API's are used to connect to someone elses data, that they have of course given you permission to retrieve and use. So instead of making a connection to your own local database, you're connecting to and manipulating the data being offered by some other company.

The trick, and in fact the opportunity, is in utilizing an API to provide more data, or better data, or data in a more usable format than is currently available. And for we marketers, monetizing our use of the API movement.

Okay, that's all nebulous and intriguing, but it's kind of vague still. Especially the part about how this may be the next frontier for the internet marketer.

In a nutshell, several very large web properties have jumped firmly on the band wagon. They're offering you access to a lot of proprietary data which you can build API tools to sort, organize and display in a way they have yet to offer. It's almost akin to the Open Source movement since you can get ahold of the data completely free, at least for testing purposes. (Note that they all put some restrictions on the "free" side of things, but if you develop something that you can monetize most also offer other types of licenses that allow greater access and the ability to commercialize your creation.)

Who are we talking about that is allowing this unprecedented access to inside information? Amazon, eBay, Yahoo!, and Google to name a few.

That's quite a list, isn't it? We're talking the biggest of the big are already firmly on-board the API movement, with more to follow surely.

I've been looking at this and mulling it over in my mind for several months now. What spurred me to write about it here is an article in the April 2005 issue of the Business 2.0 magazine that hit my doorstep this past week. For the record, they don't publish the whole article on their web site, but it's one that I recommend picking up for this one artile alone. Or subscribing to since I seem to get a lot of really solid information from lately.

The article contains several quotes from industry leaders, pointing to the fact that all of them are very happy with the way the API movement has helped them and their customers in the relatively short time a small group of API developers have been taking advantage of their new openness.

Some of the current API tools that are making money for their developers include:

Hive Group, which has developed custom browsers that integrate with and display Amazon products and iTunes songs

ScoutPal, which brings Amazon products to your cell phone, practically turning it into a wireless bar code scanner

LivePlasma, which interacts with Amazon's database to provide visual search for music and movies

Bonfire Media, which bring eBay auctions to your mobile phone

Channel Advisor, which uses API's to assist sellers on Amazon, eBay and other sites manage and automate their operations

Super Pawn, which condenses and organizes the inventory of a 46 pawn shops to sell it on eBay

The above is just the tip of a very large iceberg.

The superior access API technology offers to developers is unprecedented and almost boundless. Even if something you need, as a developer, is not currently available from one of these portals, they've been very proactive in listening to developer requests. So if you have a plan for a tool that would be useful, but which needs another nugget of information or deeper access, it's a good idea to approach the portal to explain what you need and why. They've all been very forthcoming, especially if it's going to help their bottom line.

What makes this all intriguing to me is that it's a 180 degree turn from the way traditional businesses treat their proprietary information and internal systems. Those on the leading edge have taken much more of an Open Source approach, and been very happy with the additional revenue the API movement has generated for them.

As you can see from the above examples, there are more possible uses than one can begin to imagine. And it goes both ways. Meaning the profit potential can come from several business models.

Can you imagine building some little tool that is found to be a Must Have for a large percentage of those millions of eBay sellers? You would be a millionaire overnight.

By the same token, can you imagine getting together a group of Overstock suppliers (or any other brick and mortar business for that matter) and offering to help them sell their inventory through one or several of these portals or via Pay Per Click advertising on the search engines themselves in exchange for a small cut of the profits? Completely removing the "tech boundary" and campaign management issues that so many struggle with?

Or what would happen if you built a tool set that made use of the various API's made available by the search engines to analyze what elements of the competitors sites are pushing them into the Top 10 ranking positions for the most competitive search terms across all of the search engines. Something that would break it all down into a sensible plan of action to help other vendors compete for that traffic?

(Side Note: Yes, I already have a couple of SEO/SEM tools in mind that I'll be working on in the very near future.)

I'm 100% serious when I say that the possibilities, and the possible markets one could target are endless.

With the expansion of the API Movement, the stage is set for entrepreneurial developers to become quite wealthy, while providing truly helpful applications, across every conceivable market.

The New Fronteir Indeed...

I, for one, do not plan on missing the API boat. I would encourage you to get on board also, before the cabins start filling up. Now is the prime time to take advantage.

Posted by Randy at 10:10 AM | Comments (0)

New Google Patent

In case you hadn't heard, Google has applied for a new patent that is quite interesting. It's a rather long read, but it's pretty much devoid of tech-speak, so the average person should be able to understand it all.

Basically, it points to the fact that Google may be looking at the Age (or history) of pages and links. The interesting point is that they do not state if new links will be giving a big bump, or less than they have historically. Or if old links are going to be better trusted, or be considered "stale". Or if they're going to use some sort of sliding scale where new links start off not giving a lot of weight, then more and eventually fading into the stale category.

It's a good move on Google's part not to give out the specifics on that front IMHO.

The document also portends to look at the history of each document/page in their index. Note that this doesn't necessarily mean they'll be looking at the If_Modified server header that gets sent when a document has been re-uploaded with or without changes. The way I read it they're going to be looking at if solely from Google's perspective. When the document first became known to Google via their spider crawls, if the page has changed, how much has changed, what elements of the page has changed, what percentage of change, etc.

Personally, I rather doubt that on-page portion is going to make a lot of difference in ranking. My guess would be that they'll be looking a lot more closely at link history since links are such a force with their algorithm.

FWIW, I also believe this is pretty much what has been going on since at least November 2004 in the now infamous "Florida" update. Parts of it actually goes back much farther than that, to at least the April 2004 time frame. Don't believe me? Stock, aka Grumpus, made a post on Jill Whalen's HR Forums back in April of some new crawl behaviors he had noted. Stock studies this stuff constantly and has been doing so for years. I trust his observations and assessments because he has a very good track record of not only pulling out the important bits of information, but also of not screaming "The sky is falling, the sky is falling" every time something changes. Like so many seem to do. ;-)

So the question is why would Google file a patent application? Especially one that is quite broad in some areas? Double especially when they've already been using it for months at the point the application was filed and they'd been working on it for at least a year?

I honestly have no clue since it's something they've already been doing. The only thought that comes to mind is that maybe, just maybe they're getting ready to dump the old PR (PageRank) name, which is a patent held by Stanford University (a non-factor IMO), opting for some new name.

But what would that gain them?

Well, a few things come to mind immediately.

First, it would definitely throw all of those PR chasers/abusers for a serious loop. My sense is that Google has never much liked the idea of a whole link brokerage springing up with everyone being so focused on how much PR the linking page had to send. Remember that it's always been Google's stance that links should exist solely for the good of visitors, a concept some of us have been pushing for years now.

Second, it would make a break from Stanford's patent and be a clear signal that Google is a lot different now than when it was originally developed back in the 90's. And forestall a lot of the conspiracy theorists out there who complain that Google shouldn't be able to use either the PageRank name or algorithms since they belong to Stanford. FWIW, this is pure bunk. I don't remember where anymore, but I read a statement from one of the higher ups at Google a long, long time ago that clearly stated that Google had what amounted to a perpetual license to use PageRank however they wanted.

Third, and possibly most importantly, simply filing the application (for something they're already doing) and giving it a name is likely to generate a huge amount of positive press for Google. Seriously, I can see the headlines in all of the major publications already. "The New Google!" "Google Enters The 21st Century!" "Google Boys One-Up MSN and Yahoo!"

If I were a betting man, that's where I'd be placing my wager. But then again, I have no crystal ball.

Posted by Randy at 08:59 AM | Comments (0)

March 16, 2005

AutoLink

This is a subject that I've refrained from writing about here for awhile now, because frankly it's likely to turn into a friggin' novel. LOL So I might as well apologize for this little tirade right off the bat, even though this is not going to be nearly as long or viscious as it would have been if I'd penned it when AutoLink first crossed my radar.

So let me just state that I don't care at all for this new "feature" of the Google Toolbar. No more than I would agree with any other third party software that has the potential take over a page that it has no right to. Long story short, they can do whatever they want to with their own pages and the traffic that goes to their pages, but LEAVE MINE ALONE!!!

Unless you want to start paying my hosting bills and pay me for every bit of traffic you divert.

Okay, now that I've got your attention... What is AutoLink exactly?

It's an extension Google have created for their Google Toolbar that, if enabled, will place links on potentially every page you visit that aren't there at the behest of the webmaster. It doesn't take over links you already have on your page(s), however it will key on certain things that you may have in the visible text of your page and turn it into a link. A link that leads somewhere else that Google has chosen. Without your consent or knowledge.

This concept if fraught with issues. Anybody who was around the 'Net back in the will remember the whole MS Smart Tags fiasco that was put forward by Microsoft. It did the same basic thing to webmasters, with the (fairly sizable) difference that Smart Tags required the active participation of the webmaster.

If you, as webmaster, didn't add a very specific tag to your page headers telling the browser to apply the Smart Tags nothing happened.

So it was an Opt In service.

Even still, so much hell was raised and so much public pressure was brought to bear over the concept and the slippery slope it led down that Microsoft was forced drop the idea. For all the right reasons I might add.

AutoLink takes it a step farther.

There is no Opt In. There's nothing you can put in your page to signal to the GTB (Google Toolbar, but I'm going to tire of typing that quickly) that you wish them to place additional links on your page.

There is also no Opt Out being supplied by Google at this time. In my mind that makes AutoLink, and in fact the GTB by association, nothing but Scumware. No better than any of the other browser hijacking programs out there that are the bane of all evil.

(As an aside, does anybody remember when you could actually believe it when Google said their company was built on the premise of "Do No Evil" ? They've certainly managed to fall a long way since those days!)

Okay, okay, so you want a concrete example. Here's one that is not only a good one, but is a high profile one.

AutoLink turns the ISBN numbers of books into links to Amazon.com. That's who they've decreed is the *only* book seller worth mentioning apparently.

Unfortunately, every other site out there sells books is probably going to have the ISBN number of every book they carry on their web site somewhere. The high profile example I mentioned was that AutoLink caused Barnes & Noble to re-tool their entire site so that every instance of every ISBN was a link, for no other reason than to keep Google from sending visitors who were already on their site off to Amazon.

What was Google's quasi-official response to this problem as clearly evidenced with B&N? As reported by Jill Whalen from Matt Cutts (of Google) who happened to be sitting on the same panel as Jill during the recent NY SES, Matt's response was "Well that's good, they should have been link anyway!"

(You can read the rest of what Jill said and what Matt said and several pages of what webmasters are saying in this thread from Jill's forum.)

I would hate to guess at the expense of that job or reconfiguring all of B&N's site. After all, according to Google itself BarnesAndNoble.com contains over 10 million pages! Frankly I hope one of the big boys like this sues Google for the cost of this type of silly change was required by their wrong-headed software. They deserve nothing better for sure!

Quite frankly, whether they like it or not, Google has no say and shouldn't have any say over whether they think any site passes their internal Usability Test. And even if their "They should be links anyway" mantra had any truth to it, why didn't they assure themselves that Amazon.com also linked the ISBN numbers on their site, thus providing decent usability? Amazon doesn't link ISBN's either for what it's worth, and they're often pointed to as a site to pattern oneself after on the usability front.

So a totally bogus argument by Matt Cutts IMHO. He was trying to convince Jill to take Google's side in the matter because he knows that her voice carries some considerable weight in the webmaster community.

Another interesting tidbit or two.

How are these sites that Google is giving a boost to being chosen? They won't say. Well, that's not exactly true. Their PR people spew out some fluff but never give a real answer.

Now if I were a conspiracy theorist I might start pushing the concept that perhaps the fact that Jeff Bezos, Chairman of Amazon and one of the original 5 outside investors in Google back in the late 90's, and who supposedly one of the few to have gotten Preferred Stock, reportedly for pennies (I've read the figure 6 cents per share a lot)...might just have some bearing on their decision to choose Amazon over Barnes & Noble or any of the others. Well, you can see where a thought train like that could easily go if one wanted to persue it.

If Google were thinking this thing through, selecting Amazon and Amazon alone was one of the worst decisions they could have made. Sort of a: "Yeah, we're going to release this new thing to 'help users' and anybody who looks for a book is going to be sent to the site of our Angel investor, because they're the biggest and the best and nobody else counts."

The word Collusion comes to mind. Quickly.

(FTR, I don't think anybody has the full details on the number of Google shares Bezos owns or owned. Or exactly what he paid for the stock. Or if he's sold any and if so how much profit he made from that sale. All that can be said for sure is that he owns enough to be noted in the Google IPO filings, and that he got some Preferred Stock (also from the filings with the SEC) and that he didn't own over 5% of the total Google stock at that moment, because then by law the issue how much of it all would have had to been disclosed.)

Let's see, what else before this turns into War and Peace. LOL

1.) Webmasters are in an uproar. Of course.

2.) Google is supposedly considering offering a way for webmasters to Opt Out of AutoLink, which is completely backwards. It should be Opt In only, not force webmasters to Opt Out. Just like their Adsense stuff. If the webmaster does nothing to send a clear signal that they want Google to add something to their page(s), Google should do nothing.

3.) Someone started a little anti-AutoLink petition you can sign if you want. I signed it, but as mentioned above I think they too have things a little bit backwards. It should be Opt In, not Opt Out.

4.) None other than Walter S> Mossberg penned an article that appeared the other day in The Wall Street Journal panning the idea of AutoLink. For those who don't know, Mossberg was one of the early and vocal opponents from MS Smart Tags many years ago too. Hopefully his article is a signal that the mainstream press will finally start to pick up this story, as that type of public pressure is probably the only thing that will stop them.

5.) Some of the folks that frequent ThreadWatch.org have developed and released several Javascript and combination Server-side/Javascript routines to detect and defeat AutoLink. It's better than nothing considering the current state of affairs I suppose. I just hope this kind of thing doesn't end up giving Google an easy out in living up to their responsibility to be a good Netizen. I fear that it will.

6.) Some are supposedly considering starting a Class Action suit against Google over it all. The basic thought right now is to approach it from a similar angle that eBay did when they sued Bidder's Edge. The case isn't the same because in that one eBay was mad about Bidder's Edge scraping their auction pagess and putting the data up on their own site where they compared items and prices on various other auction sites. But hey, at least with that one if someone chose an eBay auction they went to eBay, not some other site that Bidder's Edge had chosen.

FWIW, I'm not a lawyer but I think the basic premise may well be close enough to make a dent. eBay alleged Trespass to Personal Property (their site), Unfair Business Practices, Copyright Infringement, Misappropriation, False Advertising, Trademark Dilution, Injury to Business Reputation, Interference with Prospective Economic Advantage and Unjust Enrichment.

Some of those would definitely be a stretch. But the Trepass has a decent chance since the judge in the above case actually found that the domain eBay.com, and the content it provided, was Personal Property. That would, in theory, make all of our domains and the content on them our personal property.

I can also see something of a case for Misappropriation (stealing traffic that has been paid for one way or another), False Advertising (because their links will look like your links), Trademark Dilution (obvious), Injury to Business Reputation (since we webmasters would have no control over where those AutoLinks lead off to), Interference (since they're diverting traffic without explicit permission of the domain owner), and Unjust Enrichment (since they're sending our traffic off to enrich someone else and you'll never convince me that Google isn't receiving any benefit from their partners. Add in Collusion too for good measure, and include their partners.

I'd really like to see some of the big companies not fold as I suspect they will as soon as Google agrees to include them in the AutoLink choices. Get a handful of large e-Tailers heading up the list the the courts are much more likely to take the suit seriously. Whether Google does or not. I'd still like to see Barnes & Noble sue Google for the cost of having to tweak the ISBNs on their site for the sole reason of foiling AutoLink. Frankly, I think Google should bear the cost of that update since they forced the reaction.

The saga continues. And I'm sure much more will be written about it. I'm not sure I'll write about it anymore here though. The whole thing makes me sick to my stomach.

My only hope --if Google continues down this slippery slope that will, I say with 100% certainty, open things up for every other Scumware manufacturer out there-- is that they get theirs in Spades for being so stupid and such bad 'Net neighbors. In other words, if they go forward I hope that Microsoft builds the same type of thing into their new IE7 browser that is supposed to be out this summer. Only instead of it affecting every site, let's have it specifically target anything that is displayed on Google.com and their other Internet properties, effectively taking their users off to MSN. Hey, MS did just announce their new adCenter to compete directly with Google's adwords. Might as well get it off to a rip-roarin' start! :::sweet smile:::

That would be ironic justice in my mind. Even if MS does it only to prove the point that they learned the hard way back in the MS Smart Tag days.

Do No Evil my foot... Fight fire with fire I say!

Posted by Randy at 03:12 PM | Comments (0)

February 26, 2005

Google & AutoLink

So Google has this "new" thing in conjunction with their Toolbar that they're testing called AutoLink. You can read one of many, many stories about it on the BBC site.

Basically, it's pretty similar to the premise of the old MS Smart Tags or even TopText, with the exception that, as a webmaster, with SmartTags you were automatically Opted Out of their program unless you added some specific code to each of your pages.

What they're wanting to do is to dynamically make certain phrases of text on your web site clickable so that they lead off to other sites. In other words, people surfing your site with the Google Toolbar installed and AutoLink active will have no clue whether the link they're clicking on will keep them on your site or send them off on some wild goose chase. And every page of your site is going to end up looking like link spam, so we might as well all link every word of text on our site to somewhere.

I have major issues with this sort of thing, especially when it's being pushed by someone with as large as Google or Microsoft. Microsoft, in fact, was forced to put the halts to SmartTags because of the concerns raised by several people. And that required you to be an active participant in order for the additional links to show up!

My understanding of Google's AutoLink is that it's going to be active all the time, no matter whether the webmaster wants it to be or not. That, I have maaaaaaaaaaaaaaajjjjjjooooooorrrrr problems with.

Here's the deal from my perspective...

Google, across the board, supplies well less than 1/4 of the folks visiting my various sites these days. That includes traffic that comes from natural listings and Adwords traffic sent from Google or one of their partner sites. Yet they want to be able to take any and all of the people visiting my site away to somewhere else.

Now, as a business person, I have issues with this. First, Google has no concept of what my Cost of Acquisition is for every visitor to any of my sites. Second, Google does not pay any of the domain registration/renewal costs, or hosting costs or any of the other costs associated with running any of my sites.

I pay those bills, not Google. Therefore only I, not Google nor anybody else, has any say in what is to be done with my traffic once it hits my site. If I suck as a designer and my site is so ugly that nobody can stand to stay for more than 2 minutes, that's for me to decide to change so that I can keep those visitors. If my navigation structure is so bad that people get frustrated by not being able to find what they want, again that's my problem to deal with.

But by the same token, if Google wants to siphon off and divert My traffic to somewhere else, they should be prepared to pay me for that traffic at whatever rate I want set. I don't care if they're making money by diverting the traffic or not. I'm paying for acquisition and losing potential customers, so they need to be prepared to pay for the traffic. Just like I pay them for click-thru's via their Adwords program. They have the traffic there that I want, so I'm willing to pay for it. The same should be true when it's I who has the traffic.

I don't know what's gotten into the boys and girls at Google since their IPO, but they really need to take a step back. Their "Do no evil" policy has definitely flown (or been thrown?) out the window over the last several months! You know it's bad when a corporation, especially a media darling like Google, can't even follow its own principles anymore. Sooner or later the press is going to turn on them. Hopefully sooner rather than later if they keep up stuff like this.

To quote Google themselves

At Google, we put a lot of thought into improving your online experience. We're alarmed by what we believe is a growing disregard for your rights as computer users.
...
As a provider of services and monetization for users, advertisers and publishers on the Internet, we feel a responsibility to be proactive about these issues. So, we have decided to take action. As a first step, we have outlined a set of principles we believe our industry should adopt and we're sharing them to foster discussion and help solve the problem. We intend to follow these guidelines ourselves with the applications we distribute (such as the Google Toolbar and Google Deskbar).
...
Applications that affect or change your user experience should make clear they are the reason for those changes. For example, if an application opens a window, that window should identify the application responsible for it.

Now I would really like to hear from anyone at Google how the principles put forth by them on the above referenced page and on several other Do No Evil diatribes (lies?) can possibly jive with they've already stated is going to be in AutoLink.

Bad idea Google. Very, very bad idea. Sounds to me like you're trying really hard to become as hotly hated as Microsoft is in many people's eyes.

If you wanted to destroy your corporate image, you're certainly on the right track.

btw, if you decide force this through and not to make the default action of AutoLink to be disabled unless the webmaster expressly and explicitly enables it for his/her, expect to get sued. I'll be at the front of the line waving to you most likely.

Unless you want to pay me for my traffic, at the price I set for it that is. ;-)

Posted by Randy at 08:34 PM | Comments (0)

February 15, 2005

Why Google became a Domain Name Registrar

News came recently that Google is now an ICANN accredited Registrar for domain names in the .com, .net, .org, .biz, .name, and .pro sectors.

The question is Why would they want to do this? Especially given the information provided by this News.com article that quotes some anonymous Google respresentative as saying that they won't actually be selling domain names.

There are a lot of very nebulous statements in the above article about how Google intends to use their Registrar status to improve the search experience for their users --and granted the $10,000 application fee is peanuts to them-- but still the question begs to be answered.

Here's my take on it, though this is complete speculation on my part based upon things I would be looking at if I were in Google's shoes and put as much emphasis on links as they do.

I think they're doing it to help them identify link farms for the most part. And to help them protect against domain selling for PR only.

Let's looks at it a couple of ways.

First, there has been something of a burgeoning industry for some time now in the buying and selling of expired domains. Mostly domains that already had a good number of links pointing to them and a deceent PageRank number. This idea behind those employing this somewhat shady tactic being that they would get an immediate boost by purchasing expired domains that had a good PR rating because the old domains had built in link popularity.

This type of domain name manipulation becomes an even larger issue with Google's relatively new aging delay, a pretty sound theory put forward by Scottie Claiborne. Basically, snapping up an older domain could easily overcome this penalty-that-isn't-a-penalty if Google couldn't tell that the ownership of the domain had actually changed.

Google of course doesn't like either of the above scenerios much. They want each site to stand on its own merits and make sure everybody starts off from the same point, ie from the ground floor with no built-in advantages.

Frankly that makes sense for them to guard against, if for no other reason than Google does give an awful lot of weight to backlinks. (Too much weight IMHO, but I'm not Larry or Sergey either and it's their call.)

The second issue is a more recent one.

Over the last year or so a lot of services have sprung up which allow domain owners to basically remain Anonymous for all intents and purposes. Please note that these services were not begun in some attempt to fool the search engines. They were developed because WhoIs data that shows domain name ownership is, by law, supposed to be Public Information, available to anybody. As it should be IMO.

The problem with requiring WhoIs domain registration/ownership to be public information --including a valid email address for obvious reasons-- the email spammers have been out there for a long, long time harvesting the WhoIs information so that they have plenty of email addresses to spam.

Hence the privitization of domain ownership as shown in WhoIs records.

It was started as a method to keep your email address from being added to all of the spam lists. An email address was still provided in the WhoIs record, but it is a fake one provided by the service, which forwarded to you and could be changed from time to time. The kicker was that no other personal information is provided to the public if such a service is employed.

A side effect to this though is that Google could no longer tell from the public WhoIs records if the same person owned 100 or 1,000 or 10,000 different domains and had created their own little link farm in an attempt to defeat Google's link-centric algorithm via nefarious means.

I don't know this for a fact since I'm not one myself, however becoming a Registrar is likely going to give Google a means to see the actual ownership information. Rather than the anonymous crap that we mere mortals get to see from the privitization services.

In other words, instead of seeing 100 or 1,000 or 10,000 fake email addresses and no other information they could use to detect these hidden link farms, Google will be able to see (at least) that Registrar X's client number 2345 has registered all of the domains. It wouldn't surprise me if they'll be able to see all of the personal information that we can see in normal WhoIs records now, as well as other stuff like what IP number was used to purchase a domain. All of which could be used give Google plenty of clues in being able to sort the wheat from the chaff and immediately raise a red flag if a significant number of the domains are cross linking to each other.

Again, it's just pure speculation on my part. But it's the only scenerio that makes sense to me.

So there you have it. My opinion is that Google became a Registrar so that they have a means by which to protect their link-centric algorithm from those trying to trick their way to the top.

Personally, I think it's a good move on their part. And one that will have no effect at all on those who do things the Right Way, including registering their domains in a lawful manner.

Posted by Randy at 07:52 AM | Comments (0)

February 12, 2005

Friendly Navigation (Usability Part III)

Seems like I'm on a roll today!

Next, let's take a look at what makes your navigation friendly for your visitors.

First, whether we like it or not, people have been conditioned over time to see the main navigation for a site in one of two places. Either at the top of the page or vertically down the left hand side of the page.

As much as you might like to put it somewhere else, I would still advise having your Persistent navigation block in one of those two places. To do otherwise can throw some of your less savvy users for a loop and make them think right off the bat. That's something we're trying avoid remember?

Now you may be asking yourself what I mean by the term Persistent Navigation. All that is is the navigation that leads to the main pages or sections of your site., The links that will normally appear on every page of your site.

Often, especially with larger sites, you will have a small set of links that need to be on every page and other subsets that are specific to a particular section of your site. The Persistent links let people move effortlessly from section to section of your site. And also to move back to the Home page if they get lost. (Hey, it happens to prepare for it!)

One the "problems" with surfing a web site is that it's hard to convey the feeling of movement. Plus you need to remember that people will click on the first thing they see that looks like it might be what they're looking for. There's no penalty for doing that, as it's easy to hit the back button a time or two, so you can expect this trend to continue.

In the grocery store (to use our earlier example) you have tangible things around you to tell you when you've just walked in the door, when you get to the coffee aisle and where exactly you are in the store at all times. And you're far less likely to head off to the milk or sugar aisle looking for coffee simply because you like both when you're having a cup.

Gaining this sense of movement, and the location you are currently at in the grand scheme of things, is much more difficult to attain with web sites. But you really do need to do it.

There are several ways to accomplish this task by using some visual clues to your visitors. You can change the link that leads to the page when someone is on it, so that when their on your Coffee page the link to that page is a different color than the rest. You can provide breadcrumb navigation, where across the top of the page it'll say something like Home > Coffee. You can also state plainly in a headline across the top of the page where they are in such a way that it's obvious to the surfer what section they're in.

I tend to favor using at least two of the above methods to make sure I'm getting the message through. I always try to make sure the first line and title tag of the page has some connection to the link that was clicked. So when someone clicks on the Coffee link in our example they arrive on a page that has "Coffee" as both the Title of the page and as a Headline right at the top of the page.

The same goes when they click on my Gourmet Coffee (subset) link.

This approach not only helps users of the site know where they are --and what any page is about-- but it also helps the search engine spiders determine exactly what the page is about. So it's a win-win all the way around.

The main things to remember regarding your navigation in my mind are:

Keep your Persistent (main) links consistent throughout your site. They should be essentially the same on every page of your site, with the possible exception of your Home page.

Make sure your links are in a prominent location on the page and where people would expect to see them. Normally either at the top or in the upper left of the page.
Make sure the text shown on your link says what it leads to. Don't get too cute and use a link that says "Mmmm-mmmm good!" for a link that leads to your coffee page!

Give your users a couple of visual cues to indicate where they are currently at on your site.

Make sure your title tag and headline of each page corresponds to what the link to it says. It doesn't have to be exactly the same text, but it should be very obvious.

Always give your users an easy and effortless way to get back to your home page in case they get lost.

Most importantly, construct your links for your real human visitors, not for the search engines. In other words, don't load them up with long keyword phrases just for the sake of doing it. If your pages and links are constructed properly they should naturally contain your keywords.

That should cover the basics of creating a friendly navigation scheme for your web site.

Posted by Randy at 07:03 PM | Comments (0)

Usability Part II

Let's now delve a little deeper into the subject of Usability and start to tie it all together with your site design, navigation and copy. While we're at it we will also dispel a few myths, just for fun.

First, it is very important that all of your links tell the user exactly what they lead to. It is also very important that you follow up on that promise when they reach the page. I know this seems quite obvious, however you would be amazed at how many pages/sites out there do not follow this common sense approach.

Put into terms everyone can understand, let's relate the user experience for your web site to that of a customer in a grocery store. It sounds silly, but is a very valid comparison.

When you walk into a grocery store you're already in Buying Mode. Meaning if you can find what you want without having to expend too much effort you're going to buy it. So what is the process and what have the grocery stores done to help you find what you're looking for?

For instance, let's say you were going to the store to pick up some Mocha Java Gourmet Coffee. How do you find it?

Well, you start by looking at the signs that hang over each aisle to find the Coffee section. Then you walk down that aisle looking for the Gourmet coffees. Lastly you narrow it down to the specific kind of coffee you're looking for, Mocha Java, and read the packaging to decide which brand/version we want to purchase.

Notice how none of this process requires deep thought until the very last step? It's a natural progression.

Also notice that when we do start thinking about it and asking ourselves questions it's not a "Do I want to go to another store?" type of situation. More often than not it's a "Do I want this brand or that brand?" type of question. Or maybe "Do I want to try something other than Mocha Java this time?"

Regardless, we're still going to buy from the same store because we didn't have to think the moment we walked into the store. This is something all of us do every day, even though we don't realize it on a conscious level.

I'll give them their props, the grocery stores have done a really good job (for the most part) of removing obstacles from our path to make sure we buy from them. That's why it's a very good model to follow for our web sites. I'm here to tell you that 90% of the process of making a sale to every (legitimate) visitor to your site is nothing more than not making them think until it's time to hand over their money.

Here's something most people never realize... As with grocery stores, the vast majority of people will enter your e-commerce site in Buyer Mode. Your job is to remove as many obstacles as possible from their path, just like the grocery stores have. Doing so keeps them in Buyer Mode, which is where we want to keep them.

So let's transfer now to your web site. Let's say you have an e-commerce store that sells all kinds of coffee, tea, cocoa, etc. How do you accomplish this goal of making the entire process a natural progression that people are already conditioned to accept?

Now this is going to seem quite obvious at this point, but you model your site's navigation after that used in grocery stores! You make it easy for people to find a logical starting point and drill down to find exactly what they're looking for.

So on your home page (and every page for that matter) you have main navigation links near the top of the page that read something like Coffee, Tea and Cocoa.

Once someone reaches your Coffee level, assuming you carry lots of coffee type, some gourmet and some not, you offer them additional navigation links to go to different categories. Such as Normal Coffee (the big brands), and Gourmet Coffee. Depending upon your products you may even break this down further at this level. For instance you may break it down to Espresso, Cappuccino, Fruit Flavored Coffees, etc.

Then once you get to the Gourmet Coffee level, you list the details of (and/or link to if you need another level) all of the types and brands of gourmet coffee you carry so that your customer can make their choice. By this point they're already 90% sold on the concept of buying from you and they've not had to think at all !

If your copywriting is good at describing the various types of gourmet coffees you carry, you can bet that you're going to convert a lot of the people reaching this level into customers. In their mind, just like in the grocery store, it's not an If/Not question anymore. It's an Either/Or choice. They're going to buy something from you. The only question is which brand(s) and how much.

Now, doesn't that make sense? You know it does!

I challenge you however to go look at some of the top ranked gourmet coffee sites out there and study them for usability.

How many of these sites make the mistake of making you Think too early in the process. How many make it plainly obvious from the very first page which link to click on to get to their Coffee section? And how many times do you have to think about (or guess) which link will take you to the Mocha Java Gourmet Coffee that you're intersted in?

I just looked at each of the Top 10 in one search engine. Of those, only 1 was doing it substantially correct, though it had a couple of issues mid-way through the process where I had to think or guess. The rest failed my little usability test horribly.

If I had a Gourmet Coffee site (I don't) and managed to get it into the Top 10 I would hazard to guess that I'd be able to steal a lot of the other guys customers, simply because my site would be a lot easier to use. Once they found my site and realized that they didn't have to put a lot of thought into finding the type of coffee they wanted, those customers would never go back to the other guys. They would be mine forever.

While we're at it, let's dispel a few myths.

Some would say that you should have links directly to the lowest level (the one that displays the actual products) right on the home page. That may work for very small sites, but once you're carrying 15-20 products it just doesn't pan out.

You may think you're doing a good job of giving people exactly what they want on the home page, but in reality all you're doing is cluttering things up, creating a lot of noise to wade through and making people start thinking right from the get go.

Here's the deal...

People surfing the web don't read the entire page first, then make their choice as to which link would be the best to follow. Instead they scan the page and choose the first link they see that might provide what they're looking for!

For this reason you need to take a very simplistic approach to your site. Make the links to the things you offer clear and prominent right at the top of the page. Give people a way to drill down for more specific information.

It is a total myth that people will only click 3 or 4 times within a site to find what they want. That only applies when they have to think every time they make a click. Literally hundreds of usability studies have shown that as long as the progression is a natural one and people don't have to give a lot of thought to the next step they'll gladly drill down many, many levels to find what they're after.

Moral of the story?

As simplistic as it sounds, don't make them think until it's time to choose which item they're going to buy! If you can manage this one thing your conversions are going to skyrocket.

Posted by Randy at 07:22 AM | Comments (0)

February 11, 2005

Usability and Selling on the Web

Today let us start to explore the concept of how the Usability of any web site has a direct correlation to how well that site converts Visitors into Customers.

Note that when I talk about Usability, my definition has nothing to do with the Accessibility Standards for handicapped users. People often get them confused since many writers interchange the terms as if they are the same thing. They're not!

Usability is simply the ease with which average surfer can navigate a site to find what they're looking for.

Now, first is Usability the Holy Grail and the only thing that has an effect on conversions? Certainly not. The graphical appearance of your site still has to be clean and visually pleasing. The words on the pages, or your Copy, must still speak to your target market in a way that will encourage them to perform the actions you want them to take.

Usability is however a large part of the equation, and one that is often overlooked. Good usability will work hand-in-hand with your Design, Navigation and Copy to convert many more visitors into customers. Which of course leads to more dollars in your pocket.

I fall into the camp that says Usability should dictate that a web site should be Intuitive to your target users. In other words they shouldn't have to read every single word and actually think about what to do next. Instead, it should be a natural progression.

If you could be a fly on the wall watching as people surf through your site you would probably see many of them having to actually think about what to do next.

If they do, you've failed the most important Usability test.

Since I'm going to assume you know who your target market is and have written engaging copy for your site, you should be able to incorporate sufficient usability into your site where your visitors can get from Point A (initially entering your site) to Point Z (where they convert from being a visitor to a customer) without ever having to think. When, not if, you get to this point you will have captured the Holy Grail of web marketing.

If we were all rich we could hire a usability expert to test our sites and offer suggestions so that they performed flawlessly. They would have a number of users who fit our target market visit our site and give us feedback on areas that we need to improve.

Very, very few have this luxury. I certainly don't.

That's the bad news.

The good news is that it is quite possible, even easy, to conduct your own usability study. The key is getting people to tell you what they really think, instead of being too nice so that they don't hurt your feelings.

In the coming days (weeks? months? LOL) I'll be delving into the details of how I approach the subject of Usability, including tips and tricks(?) you can use to improve your site and its conversion to sale ratio. There really are no tricks. As with most things relating to marketing via the web, it's all simple common sense.

But by the same token, it is also something that the vast majority of webmasters either completely ignore or could certainly do a much better job of.

Posted by Randy at 07:57 AM | Comments (0)

February 07, 2005

Yahoo! Contextual Search

Yahoo! have relased the beta version of their Contextual Search, which goes by the moniker of Y!Q. Personally, I don't care for the name much as it would be easier to simply call it YQ or even YCS! I understand why they have the exclamation point in the middle of it (it is part of the Yahoo! trademark after all) but I think it looks silly.

But that's neither here nor there. Just my personal opinion.

Now you're probably asking yourself, What the heck is Contextual Search?

Basically it's a form of Plain Language Search, such as you see at AskJeeves or Vivisimo, where instead of typing in a search term you type in what you're looking for just as you would speak the question.

AskJeeves itself doesn't offer much in the way of clustering or contextual search, just the plain language portion of the equation. A little bit of clustering. In fact, AskJeeves doesn't even offer as much of the clustering effect as Teoma does, even though that's the data source that AJ uses. I've always thought that is just a bit odd. On the other hand, Teoma isn't really good with natural language searches.

Vivisimo does a bit more with Clustering, where related categories are shown as possible options. But frankly their pages get so cluttered that they're hard to work with. There is such a thing as too much information.

Yahoo's (sorry, no room for a ! hehheh) contextual search works on a similar principle. But instead of offering you potential related clusters, they strip certain portions of your request out and allow you to enable or disable those on the search results page. When you choose to enable or disable certain contextual phrases the results change, which could be valuable if they picked up on a word or words that you're not interested in.

What's it all mean? Not much really, since the technology is still in its infancy. And honestly not many people use it.

The bottom line is that computers still aren't nearly intuitive enough to correctly parse out the pertainent parts of your plain language questions. They do a pretty good job if you manage to get your most important words next to each other, but if you're going to do that you might as well simply type those keywords instead of an entire sentence!

Let's take a quick look at a couple of very similar searches to illustrate the point. Here's my (novice to search engines) plain language statement/question:

My daughter is having a birthday and her party is next week. How can I make and print invitations for the event?

Sounds logical doesn't it? Almost like what you would ask someone if you were standing in front of them and needed help or guidance.

The problem is that all of the methods mentioned above fail on this relatively simple request. Clustering doesn't work and neither does contextual search. The statement/question is too complicated for them to get the gist of it the same way a human would.

A human would automatically latch onto the most important words. Print, Birthday, Party and Invitations. However computers struggle with this logic since not all of the words are connected. Clustering comes up with no options if you search on that plain language phrasing. The best contextual search can do is realize you're asking about invitations. Nothing else though.

If however you leave out the extraneous details and keep it really simple by asking something like:

How can I make and print birthday party invitations?

they all do at least a decent job of sorting out what you want.

AskJeeves will return some pretty relevant sites, with the only related cluster being "Invitations For Free". Teoma will give you that one cluster and a few more, none of which are all that relevant frankly.

Vivisimo does okay by returning some relevant results and giving several related clusters.

YQ (sorry, no room for a ! LOL) actually does a better job that the above two. It gives you some relevant results and has managed to sort out three possible contextual portions of your search. Those being: birthday party invitation, party invitations and birthday party.

Not bad, not bad at all. Except that they all missed the fact that I was looking for a place that would allow me to print the invitations myself. A human wouldn't miss that tidbit of information, and IMO the search engines shouldn't either since it's actually connected to the phrase they're all picking up on.

Which leads us back to the original problem.

If I'm savvy enough to realize that I want to look for a way to print birthday party invitations, I might as well simply stick those four little words into the normal Yahoo!, Google or MSN Search. I'll find sites that offer what I'm looking for a lot more easily and considerably quicker. And type less to boot!

Will Contextual Search be something we'll see more of in the future? Maybe, maybe not. The underlying technology is going to have to improve an awful lot in order to make it viable.

In the meantime it might be easier to simply teach people how to search. ;)

Posted by Randy at 12:08 PM | Comments (0)

February 01, 2005

MSN Search Is Here !

Just a quick note for those who have been following the MSN Beta search engine. Well for that matter those who haven't been following it too! LOL

MSN Search is officially out of beta and is Live from here on out on MSN.com. To go directly to MSN Search so that you can play with all of the tweaks you can make while searching, go here.

I can spot a few improvements that need to be made (the page count thingee is downright wacky for one!) but overall I'd say they done a pretty decent job of it to this point.

Expect the TV Commercials to start any day now. ;-) I'll lay you odds that they'll start soon. If not before, the Superbowl would be a good time for Mr. Bill to spend some cash and make a big splash.

Want a sneak preview of the commercials? Well, not exactly a preview but I think several elements this little tour are probably going to be included.

Welcome to the real world MSN Search! I, for one, am glad you're here to provide some competition to the other search engines. A serious player like yourself should spur everybody to start innovating again.

Posted by Randy at 05:15 PM | Comments (0)

January 20, 2005

MSN Search Ramps Up

For those who haven't been paying much attention to the new MSN Search (Beta) that is fed data by MSN's own spider and uses their own algorithm, now is that last chance to get in on all of the fun.

Over the last few days there have been numerous reports of people getting the MSN Beta results at MSN.com instead of the old Yahoo!-fed results. I had seen this appearing and disappearing over the last couple of months off and on. They're giving you an easy graphic to tell which you're connected to when you conduct a search. Simply look for "MSN Search (beta)" on the graphic in the upper right corner after you search for something.

Rumor has it that MSN will be gradually upping the number and location of people who use the Beta search product, with an expected full, worldwide release in the very near future.

If you haven't been paying attention to MSN's new search capability you'd better start in a hurry! I would expect that once the full release has been issued Microsoft will start a media blitz in an effort to cut in to the marketshare of both Google and Yahoo!

Posted by Randy at 01:13 PM | Comments (0)

January 16, 2005

Is Google Tracking Clicks?

Here's something I noticed when gathering links for yesterday's blog entry that was interesting.

It appears that Google is now (sometimes?) collecting data on individual users, the searches they perform and the links they click from the SERPs.

It is not terribly new that Google sometimes tracks clicks. They've been doing that off and on for some time now. But before it was simply a little javascript routine that recorded the SERP position number that you clicked on.

The new version I saw yesterday (and still today) is apparently associating those clicks with me, as an individual searcher. I'm not sure what is triggering this, but it may well be that I have the Google Toolbar installed and running.

Here's what I see...

When I conduct a search on Google use my IE browser with the toolbar installed, every entry is returned with a jump link that looks something like http://www.google.com/url?sa=U&start=5&q=http://www.somedomain.com/&e=XXXX

Where the start= number is the SERP position for the link and the &e=XXXX is a four digit number that never changes for any link or even with new searches in different IE instances. The four digit number appears to be some type of individual tracking code at first glance.

Some others who have checked this see the same type of jump links, however have a differnt four digit number. Strangely, others do not see anything like this at all. They either see a clean link (no jump) or the old javascript position tracking as an onMousedown event if they view the source code.

In fact, if I perform the same search in my Netscape, with JS and cookies enabled, I get a completely clean URL. No JS, and no jump link. This leads me to believe initially that the id number may have something to do with the Google Toolbar being installed on my IE browser.

What does it mean? Who knows.

Maybe it's Google's first foray into the Personalization of Search, though if that's the case they're simply collecting data right now. All of the folks I checked with who are seeing the jump/tracking links see the same Top 10 sites as I do for some specific searches.

It'll be interesting to see what they do with the data in the future. Right now it doesn't seem to be doing much, if anything.

Posted by Randy at 07:10 AM | Comments (0)

January 02, 2005

Google and Mulitiple Site

There are some indications that Google may be looking much more closely at the relationship between sites that are in the same market.

Please note that from my observations this appears to be Keyword Specific, especially on reasonably competitive (or money) keyword phrases.

What does that statement mean? Well it means that sites are not being penalized just because they happen to be sitting on the same IP number or same IP range as another site that is doing something rather iffy. It does however mean that if there is a connection between two sites that are both competing on the same highly valued search phrases, one may show up and the other not.

It is difficult to ascertain at this time whether Google is looking at IP/Server information, WhoIs information, Linking between the sites or a combination of the above and a lot more.

If I had to hazard a guess, I would say that they're looking at a multitude of things, including the above. We know that links are important to Google, so it would make sense that they are analyzing that portion.

It would also make a lot of sense that they would be looking at the IP number or IP range of two or more sites that are competing for the same competitive keyword phrase(s).

It certainly wouldn't be out of the realm of imagination that they would also be looking at WhoIs data, or even text blocks within certain sections of the page content to determine if two sites were closely related.

I ran across this one quite by accident. I run two sites that provide a very similar service to two completely different markets. Even though they are targeted at different market segments, the sites of course compete on a few very general, and quite competitive, keyword phrases.

I made a boo-boo on the server side of things that caused one of these sites, the one that has always ranked pretty well, that caused Googlebot to have a few heart palpatations. I caught it right away, but Googlebot had already been by and freaked out.

So that site that always had a decent ranking was dropped out of Google's index for a few days, while I waited for them to re-spider the site and see that all was well.

A very interesting thing happened when this Site #1 disappeared.

Suddenly, the other site that we'll call Site #2, started to appear for certain reasonably competitive phrases in almost the exact same SERP positions that Site #1 had been dropped for!

Let me say that again so that it sinks in.

Site #1 has a temporary problem and gets completely dropped.

As soon as that happens Site #2 starts to show up on those competitive phrases that it has never been in the Top 100 for before.

The moment that Site #1 gets re-spidered and re-indexed, returning to its normal position, Site #2 gets banished again to position 130 or so.

Let me note here that the fact that Site #2 doesn't show up for these very generic phrases. It is targeted at a very specific market segment and I really don't want it to show up under the general phrase. You know me, I'm all about ROI. I want people who make it there to be the exact people who would most benefit by the service.

In the search engine's eyes they may be competing sites. In mine they're not since they target different market segments.

That said, it was very interesting. And something I posed to some friends in the SEO field.

Some have seen the same thing, though we've not yet been able to put our finger in what the trigger might be. I have a feeling it's a multiple trigger.

My sites absolutely set off all of those triggers. Both sites sit on the same IP number/range, both sites target (in the search engines mind) some of the same phrases and both sites have identical WhoIs information. They also link to each other once or twice so that I can try to get potential customers to the place they actually belong, instead of us having to sort it out after the fact.

But that's it. The other links pointing at the two sites are totally different. They are different markets, even though the service is very similar in a general sense, so the links pointing to the sites are totally different.

Is Google over-reacting in their hope of removing sites that are owned by the same person, and have a close relationship, but offer the same thing?

Perhaps. But I'm not going to complain about it too much. They really do need to knock down all of those fake sites that are built as an attempt to increase PageRank and deliver more traffic. Whether it is qualified traffic or not.

On the other hand, in a perfect world both of the little sites I noticed this on should probably show up. Both are legitimate, full-service sites that happen to offer a similar service to two different market segments. There is zero duplicate content, save a copyright notice. The design of the two sites is totally differnt.

A human reading the two sites would immediately realize that the sites are about as different as two sites can get while offering something similar.

Which just goes to prove two points...

1. The search engines are not nearly as smart as people are.
2. Legitimate sites are once again paying the price for Google trying to come up with a machine-oriented solution to the problems they have created themselves.

Posted by Randy at 11:53 AM | Comments (0)

December 18, 2004

The Personalization of Search and what it means to You

I've written before about the Personalization of Search, but it's always been very general stuff. Today let's dig a bit deeper into the subject to explore what it means to You, the web marketer. And how it may affect the choices you make in designing your site.

Search Engine Marketers, or SEMs, tend to fall into one of two camps.

There are those who concentrate almost totally on getting the search engines to rank their site well, come hell or high water. They'll spam the search engines with links from totally irrelevant sites, stuff their keywords in every conceivable nook and cranny and basically hope that having a high ranking site alone will convince people to buy from them. There is little thought or consideration to the actual human visitor, and it shows.

The other camp is almost totally customer-focused, spending their time and energy on building a site that is easy to navigate, chock full of rich content and provides real value to visitors. The search engines are a secondary thought to these marketers, though still important. But if the choice ever comes down to doing something just for the engines that will not benefit, or could even hurt, the offering when viewed from the customer's perspective; people who fall into this camp will always choose customer satisfaction over gaining another few search engine positions.

Recent history has shown that it is sometimes very difficult for the customer-focused marketer to compete with Sammy-the-stuffer. At least for search engine rankings, though their traffic converts at a much higher rate. In some markets there are so many sites doing the spammy stuff that it's difficult to even crack the Top 20 positions for moderately competitive keyword phrases.

That's all about to change...

With the debut of Personalization of Search, or POS in Randy-speak, I expect there to be a major shift. Here are the how's and why's of my opinion.

For POS to be effective for the search engines and their customers (the searchers) there has to be some way for the individual user to provide feedback to the search engine. That's been the missing element in Search forever. There has been no fairly automated method for the individual searcher to tell the search engine "Yes, this site provides exactly what I was looking for" or "These guys are duping both of us!" The only feedback the search engines got was how many people used their service, but it never got down to the individual searcher or site level.

POS is going to have to introduce a new level of searcher feedback to be effective. Why? Take a look at this brief scenerio.

Let's say you went to a search engine and conducted a search for "Blue Lederhosen". In that search you get 10 sites on the first page to choose from that the search engine algorithm has determined are a match your request.

Reading the snippets provided you decide to click on the site in the #3 position, because it matches the idea that you have in mind. When you get there you look around a bit and make your own decision as to whether it provides what you're looking for or not. If you're happy you buy some blue lederhosen. If not you go back to the search engine to try another site. Just like with what happens now.

The difference with POS is that the search engine is going to want to provide some way to find out if the site you chose was the right one for you or not. If it was, they'll want to make sure they give that particular site a boost in your personal searches the next time you look for the same thing or something similar, like Red Lederhosen. In fact, the search engine will likely want to bump this site up from the #3 position to the #1 position for you.

By the same token, if you got to the site and it was totally mis-advertising itself, the search engine will want to know that too! So that the next time you search for the same or similar phrase this particular site will be pushed down far enough to allow another possible match to rise to your attention.

That's what the Personalization of Search is all about, at the individual user level. The search engine being able to collect enough data about You and what You want to help you find what you're looking for more easily. Even if the first search starts out being exactly what we see today, and it likely will in the beginning, they need a way to fine tune the results they show you.

Now let's extrapolate things and expand on the concept a bit.

Since the search engines are going to be receiving a lot of feedback from a lot of people, they should be able to use that information to improve their overall service for everyone after some time has passed. Let's work with small numbers to show you what I mean.

Taking the same scenerio as above, let's say that 100 people from around the world searched for "Blue Lederhosen" wanting to buy some. Let's also say that all 100 of these people went to the #3 site as above.

Now, for the sake of discussion let's say that the #3 site is using a lot of dodgy tactics to attain that lofty position. They have hidden text in their code that never reaches human eyes, they're cloaking, they've run a linking campaign that gives a false impression to the engines what the site is really about. In fact, they don't sell Lederhosen at all! Instead they have a fakey directory site that is targeting the phrase for no other reason than to get traffic and hope someone clicks on one of their ads so that they get paid a few pennies. The only thing they sell themselves are ebooks on how to scam your way to the top of the search engines.

That's definitely not what you were looking for. But I'm sure we've all run into something like that in our surfing. ;-)

So now let's say that 40 (or however many) of the people who 100 people who searched for Blue Lederhosen use whatever method is provided to send feedback to the search engine that this particular site has duped them into showing highly for the phrase.

Based upon this real human input, the search engine can now choose to remove this site from showing in the Top 100 or Top 1,000 for the phrase we found it on. Not just for us, but for everyone. The engines could also send it into oblivion for related phrases with reasonable assurance of making a correct decision. In fact, if they got enough bad reports, as a percentage of unique users, about the site on multiple phrases it appears for, they can drive it down in the rankings across the board!

It has been said by some for many years that search engine algorithms simply aren't smart enough to make some of the decisions that need to be made in order to clean up all of the crap that shows up in some markets. And those people are right. There needs to be a human element in the mix somewhere. However it would take a huge labor force to introduce human judgement in the equation, more than any single company could afford to hire without a doubt. That's why it's never been done.

With the debut of the Personalization of Search the engines will have this huge labor force at their disposal, without having to pay a dime for it. And searchers will also be happy, because as POS is used more and more the SERPs will get a lot more relevant. The spammy site tricks simply will not work anymore, because the human element will be part of the criteria of how well any site ranks.

It's a Win-Win situation for both the engines and searchers. And in my opinion it's also a Win for search engine marketers who go to the time and trouble of conducting their business in a legitimate manner.

Will it take some time to reach this Search Nirvana? Yes, there is no doubt there. POS isn't even a reality as I write this. But it's getting close. Whichever search engine comes out with a well thought out and workable version of Personalization will have a huge leg up on their competitors. I fully expect to see some form of POS within the next 12 months.

Once it arrives it is only a matter of time before those who concentrate solely on the search engines start getting their comeuppance.

And in direct correlation, it is only a matter of time til those SEMs who focus on customer needs and wants will begin to reap huge rewards.

Posted by Randy at 06:51 AM | Comments (0)

December 11, 2004

Doing what's right

One of the questions I often get asked about marketing web sites via search engine optimization is What is the difference between White Hat and Black Hat techniques?

Personally, I don't care for the terms themselves because everyone seems unable to come up with a good definition of what methods constitute any of the various Hat colors. To me it's more about doing what's right or not doing what's right. In other words, if the technique is designed to deceive anyone --be it human visitors, search engines or whatever-- then you're simply asking for trouble.

That said, some of these riskier techniques can have an effect on search engine rankings. As has been proven over and over again. It's something the folks who run the search engines combat constantly. Lately they've been losing that battle.

When asked to explain this phenomenon in plain English I've taken to using an updated version of an old fairytale.

Do you remember singing the old song about going over the hill and through the woods to Grandmother's house?

Simply put it into 21st century terms. Instead of walking down a path you're driving your car along a two-lane, twisting and winding road on the way to Grandma's house. It's got so many curves and blind hills that most of the 40 mile journey is a No Passing Zone with double yellow lines.

To make matters worse, you keep getting caught up behind other cars that are going 20 mph in a 55 mph zone. And there is a fair amount of oncoming traffic. And you're running late.

Now, to get around the slower cars you can pick your spots and try to pass each one in a relatively safe manner, by waiting for a passing zone. But there are so few you know you're going to arrive later than planned.

Or you could take more risks by passing against the double yellow lines. Something you could probably get away for a time.

It's a risk vs. reward situation.

Is it really that important that you get to Grandma's on time, no matter that you're risking a head-on collision to do so? Or would it be better to arrive 5 to 10 minutes late, but be reasonably assured of getting there safely?

Most people would take the sane approach, realizing that the possibility of an untimely death is not worth the risk. The reward simply isn't enough to justify taking the chance.

Choosing which method to use with your web site is really very similar. Take dangerous shortcuts and you risk a death penalty for your site. Take a sane, common sense approach and it will take you longer to get there, but when you do you'll be assured of a long and profitable life for your site.

The real key for webmasters and e-marketers in my mind is what type of business you're trying to build. If you're out for a quick buck at any costs then cloaking, spammy pages, link manipulation, etc seems to be a viable option. Using any/all of the above can help to push your site up the SERP ladder more quickly.

But if you follow that path, you've got to have a Plan B and Plan C because eventually your tactics are going to get discovered and your site will be toast. And you'll be starting over again from scratch.

Whereas if you keep everything clean, stay in your own lane and keep plodding away it's going to take you longer to get there. But you'll be able to enjoy reaching your destination more and for a much longer time.

That is your choice. I have already made mine. I prefer living a long life, and give my sites a shot at the same.

Posted by Randy at 01:40 AM | Comments (0)

December 05, 2004

Google and Directories

Been meaning to write about this one for a couple of weeks, but been too swamped to find 5 spare minutes.

It appears that Google may finally be doing something to combat the (somewhat) recent explosion directories. Mainly what I call fakey directories that people have been adding to their web sites in an effort to trick the engines into ranking them higher on their main terms and hopefully to get some traffic on supplemental terms.

It's not clear exactly how they're going about sorting out which directories are real and which are fake, but some early indications I've seen lead me to believe they may be looking at the URL structure.

What I've seen looks to be combating the default URL structure of a certain directory creation script that is sold specifically to decieve the search engines. Namely, if a site has a large number of pages that have a number as a directory in the URL, many sites are dropping out of the SERPs for the most competitive keyword phrases. These sites aren't being banned per se, and there hasn't been any PR fluctuation... They simply stop showing up anywhere in the top 200 or more for their most competitive phrases.

Whatever Google is doing it's certainly not perfect yet. In fact, some innocent sites who happen to have a certain URL structure are getting caught up in it all. That said it is still good to see Google at least making an effort combat these spammy fakey directory scripts/sites for a change.

I know I for one am tired of seeing spam for 1/2 or 2/3 of every moderately competitive search term.

Posted by Randy at 10:15 PM | Comments (0)

November 16, 2004

Another MSN Search Quickie

If you want to stay on top of things in the coming/new MSN Search, here's the place I watch: MSN Search Blog

You can pick up some info there and even leave comments if you see something that doesn't look quite right.

Don't miss the "Near Me" button info if you're interested in geo-targeting.

Posted by Randy at 12:00 PM | Comments (0)

November 10, 2004

MSN Search Rumors

Quick and dirty news.

Dateline: November 10, 2004

MSN is apparently set to release their new search technology upon the world. Speculation is that the official release could be as soon as tomorrow! That's what a couple of newspapers are saying this morning, including the Seattle Times and the New York Times.

No word yet on whether they will be releasing their new Desktop Search that will use the new MSN Search at the same time or not.

They have interviews with various reporters scheduled throughout the day today, with an official announcement expected to come at 9pm tonight (Seattle time).

If you haven't looked at the MSN Preview area to see how your sites stack up, here's a link to it: MSN Search Beta. I don't expect that one to remain active much longer.

I rather like the MSN Search results, but I may be biased because it seems to love all of the sites I work on. LOL

It's time for Google to really start worrying now...

Posted by Randy at 07:19 AM | Comments (0)

July 12, 2004

MSN Again

MSN has released what looks to be a real alpha test of their new algorithm. It's all still in the very early stages, but it'll be something to keep an eye on. The URL address is http://techpreview.search.msn.com/

I would caution everyone not to put a lot of stock in anything you see there yet. They only have about a billion pages in the index that's being searched currently. And they don't have a lot of things "in play" yet that surely will be before the final release.

They have a link there to their Community/Forum where you can supposedly ask questions about the new MSN Search, but it doesn't seem active at all yet. Most of the posts in the MSN Search are aren't even about search. LOL Looks like someone needs to do a bit of housecleaning...

Posted by Randy at 04:56 PM | Comments (0)

June 19, 2004

What's up with MSN Search?

Two of the most oft asked questions these days in the Webmaster community are "What's up with all of this spidering by MSN's bot?" followed quickly by "When will they start using their own data?"

The first answer is easy. Those MSNbot entries you see in your log files is Microsoft's prototype spider gathering information to use in their database of web sites. Both to build it, since they're starting from scratch; and to build something they can use for internal testing.

MSNbot certainly appears to be a voracious little critter, snapping up everything it can find on a site. Especially the first few times through a site. There have been numerous reports over the last few months of MSNbot crawling like a radoactive spider, jumping from page to page quickly and eating up a lot of bandwidth.

Some webmasters have complained about this, mainly because it's putting them over their bandwidth allowance. There have even been a few reports of MSNbot hitting a site so hard that it's slowing down the server. Most of that IMHO is because those sites are on crappy servers. ;-)

MSN has allowed you way to better control how MSNbot behaves, short of simply blocking it from your site altogether. Basically, they've appropriated a non-standard robots.txt directive that you can use to slow the bot down. Usage is as follows in case you need it:

User-Agent: msnbot
Crawl-Delay: 20

The above instructs MSNbot to pause for 20 seconds between pages. So rather than grabbing several pages in 20 seconds time, it will retrieve only one. You can of course use any numerical value in the Crawl-Delay entry.

Personally, I've had no problems with MSNbot's crawl rate. Even with my dynamic sites its not been putting a strain on my server. For a few weeks there it did grab an AWFUL LOT of information, but bandwidth is cheap these days so that's not an issue for me. Now that it has a good idea of what my sites are all about and how they're structured, things seem to be pretty normal. Still a little above what the other search engines are grabbing, but not out of line by any stretch of the imagination.

Last week's bandwidth stats from one site are:

MSNbot: 11.58 MB
Googlebot: 5.75 MB
Yahoo! Slurp: 4.05 MB
Teoma/Ask Jeeves: 1.09 MB

I can certainly live with that, especially considering this particular site uses over 700 megs of disk space. Plus, I'd be much more worried if MSNbot wasn't spidering my sites completely ! From the looks of it, they're certainly trying to make sure they have a large database when they start rolling out their own search technology.

Next comes the question of when that will happen.

Speculation has run rampant. Even prior to MSNbot's first appearance earlier this year. Since its arrival on the scene, the rumors have reached an almost fever pitch.

Early on there were hints that it would be coming in 2005. The most recent official mention of a time frame I've seen is a post by MSNdude (a Microsoft employee) over at Webmaster World that simply stated "later this year."

If I were a betting man I would say it's going to be sooner rather than later, assuming all of the technology parts and pieces are in place. It's a many pronged problem.

First, the database has to be in place. Microsoft has obviously spent the last few months scouring the web building this database. With the varied reports of how active MSNbot has been I have to believe that they're reasonably comfortable with that portion of the equation, but who knows. Maybe they want to make a really big splash by being able to say that they have twice as many documents in their index as Mighty Google.

The second part of the equation is the actual Search Technology. The algorithm and the speed with which search results are returned. And can it stand up to the massive usage requests it will likely see from Day 1, both by normal users and search engine marketers who are trying to get a handle on what MSN Search needs for a page to rank highly.

To my knowledge no one outside of Redmond has seen this side of things yet. However it would not surprise me one bit if they're very, very close to having the search side of things ready to rollout. You have to remember that Microsoft has already been working on this for some while. Not only for MSN Search, but also for their Longhorn project, the next iteration of Windows.

They've got skilled programmers and they certainly have the funding to hire others if needed. And there are other factors in the mix that may spur a sooner-than-originally-predicted release.

Google's pending IPO. Yahoo's recent rollout of their own search technology. The AOL contract that should be expiring soon.

My thought is that if there is any way Microsoft can get MSN Search ready for prime time early, they'll do it. This isn't like rolling out a new operating system. Not only is it considerably less complex from the programming side of things, but there is also competition.

In the O/S market there is really no pressure for them to rush production because there is basically no competition. The same is not true in Search. There is plenty of competition, and the competitors are Hungry to beat MS to the punch. If you don't believe that one, read Google's IPO filing where they specifically mention Microsoft and the search tool they're building, and how this could have a detrimental effect on Google's business and corporate valuation.

Microsoft is coming into the search race in 3rd place, as opposed to being the unchallenged champion they are in the O/S market. Being late to the game MS certainly doesn't want to let everyone else steal all of the thunder for the rest of this year.

So while I don't have a specific date in mind, my gut tells me it'll be sooner rather than later.

Posted by Randy at 05:41 AM | Comments (0)

May 31, 2004

Google PR - Backlinks Update

I just noticed that Google is starting a new PageRank and Backlinks update. I haven't been watching things too closely this past weekend, it being a holiday and all, but it appears that it just started.

If you watch that sort of thing don't freak out when you see your site moving up and down and all over the place in the SERPs over the next few days. Generally speaking, you'll want to give it at least 4 or 5 days for things to settle back into some semblance of normalcy again.

Posted by Randy at 08:31 PM | Comments (0)

May 27, 2004

The Personalization of Search

So we know that Google isn't nearly as dominant as it once was. That Yahoo! has taken the first baby steps to be a real competitor. And that MSN is ramping up their own search technology. (Sidebar: Google is still providing search for the AOL Network as of this writing. I fully expect that this may change in the very near future, as soon as the contract is up. The AOL search traffic is likely going to end up going to the highest bidder.)

That's all just Marketshare stuff though. It's important, especially to the investors of the different companies, but it doesn't really tell us where search is headed. So what's the next big thing on the horizon?

What I have come to call the Personalization of Search, or POS in Randy-speak. This concept is going to be difficult for some to wrap their brain about, so let me try to explain it by example.

Let's say that in my searches a fair percentage of them are for information related to Apache web servers. Meaning, I'm not really looking for an e-commerce site to purchase something from. Instead I'm looking for reference material related to how to do this or that on Apache. Also, I'm certainly not looking for Apache Indian Artifacts to purchase because I'm a collector.

If the search engine issued me a cookie that contained an anonymous identifier, or better yet if they had profile information I have provided them (Yahoo! and MS could do this easily) they could track my searches, which results I clicked on, if I stayed on a site or came back to refine my search, etc.

Over time, the search engine could easily build a sort of Search Profile for me, with one of the things noted being that I used a lot of "Informational" type sites.

Now, let's say that I have a friend named Bob who is a collector of Indian Artifacts and is constantly looking for good deals to build up his Apache Artifacts collection.

Bob uses the same search engine I do, and many times our searches contain at least one word in common (Apache) even though we're looking for *totally* different things.

When the Personalization of Search becomes a reality, Bob and I will be able to conduct a search on the exact same or very similar phrases and get totally different results. Mine will be about Apache Servers; Bob's will be about Apache Artifacts.

If you take this one (rather smallish) example and apply it to every search by every searcher you'll begin to get an idea of the scale of the change POS is going to bring to Search, and specifically to Search Engine Marketing. In a nutshell, there will no longer be a #1 placement that is there across the board for all searchers using a certain phrase.

Instead, there will be one site that is #1 for a certain category of searcher and a totally different set of results for another category of searcher. As an SEM it is going to become more important than ever for you to "speak" to your target audience. If you don't, not only will you have difficulty getting listed in the search engines in the first place, you'll never stay in the top spots because the search engines will be able to sense (to some degree at least) if the average searcher was happy when they reached your site or not.

That sounds scary doesn't it?

What we're talking about here is the next generation of search engine technology. Instead of them taking a shotgun approach, they will attempt to become sharpshooters, picking off one target at a time instead of shooting in a general direction and hoping.

What does this mean as an SEM?

That if you've been trying to use some "formula" to obtain high rankings it is now time to stop and consider a new Customer Centric approach. Instead of building a site and then trying to tailor it to the type of people who visit it, you're going to need to understand who your target audience is from the very beginning.

Personally, I like the concept POS. But that's likely because I've always placed a lot more effort to building sites for Visitors vs building them for Search Engines. Many, many of the spammy sites that are out there with decent rankings right now will simply disappear from the SERPs over time. If everybody who clicks on a site immediately comes back for another try, that demographics can and probably will be factored into where it appears in the search results after a time.

When is POS going to be a reality?

It's hard to say. All of the majors are already working on it. They've also already been doing some limited tracking of their users for some time. If you don't believe me take a look at your cookies and you'll see at least one has been set for every major search engine, assuming you allow cookies to be set.

If I had to take a wild stab, I would guess that we'll see the first version of POS released by somebody before the year is out. I definitely expect to see all of the Big 3 have some sort of Personalization in effect by this time next year.

The smart search engine marketers are already considering the possibilities, and preparing for the change as well as they can. Those are the folks who will take the most advantage of the changes when they hit, so I would encourage you to start now.

Posted by Randy at 04:55 PM | Comments (0)

May 26, 2004

Search 2003-2004

The last entry brought us to the last couple months of 2003. Let's take a look a the major changes that have occurred in the last few months.

First, back in mid-November of 2003 Google conducted what can best be described as a pre-emptive strike by making some major changes in the algorithm they use to rank sites. Needless to say, it is difficult to predict an outcome when you make changes to 1's and 0's and those changes affect billions of web pages. I have no doubt that Google was trying to improve the relevancy and search experience of their users, however this is far from what ended up happening.

Some very good sites that had worked hard for months, or in some cases years, to obtain a much treasured high ranking position for their search terms suddenly fell off the map. We're not talking small changes here. Not like a #2 site dropping to #20.

In many, many cases Top 5 sites that were very good and very relevant to the search term being used fell as low as position 1,000 or below!

You can imagine the outcry. Especially since it happened at the height of the Christmas buying season. Needless to say a lot of eMarketers got hurt badly. And conspiracy theories ran rampant.

I really don't believe Google was doing anything sinister. Instead they were simply changing the weight different elements used in their algorithm when ranking sites. And it didn't go well.

To this day Google still doesn't provide search results that were as god as those pre-November in my opinion. They seem to be putting an awful lot of weight on links and linking text... Too much weight IMO.

I make this statement as a searcher, not as a search engine marketer. I used to use Google for all of my searches and were pretty happy with the results they returned. These days I don't use Google nearly as much as before, simply because I don't have the time to wade through 30 or 40 sites to find what I need, and what used to come up in the Top 5 every time.

As far as optimizing a site for Google at this moment in time, it's laughably easy to totally skew their results if you want to. Simply get a lot of links pointing to your site that use the phrases you're trying target in the link text. It doesn't necessarily matter if the page is optimized for that phrase. It's all about links and linking text with Google at the moment. That will trump everything else.

I'm sure they'll keep tweaking things until they get it right. Or hope they will anyway, because it's far from perfect at the moment.

Now, a normal searcher would wonder why Google would make such a major change since they were the only game in town.

The reason is simple. They had to prepare for two major competitors entering the market to contest their dominance. And they knew they were going to be becoming a Public Company.

All of these changes were going to happen over the course of a few short months, so they were trying to get a jump on everything. And find some way to differentiate themselves from everybody else.

The first shoe fell in mid-February 2004 when Yahoo's contract with Google expired. When it did, they began (finally!) to use their own search technology to produce their search results pages. It's not like they didn't have the expertise, considering they'd purchased several search technology companies in the previous 18 months or so.

The moment this happened Google lots a good portion of its marketshare. The way I view marketshare anyway. To me, if Google supplies the results that's Google marketshare. I don't care if people are searching on AOL, it's still Google data.

So a new player was there that had the technology and wherewithal to compete head-to-head with Google. Which is something they'd not had for years. Most of their life in fact.

Truth be known, Yahoo! even has some advantages over Google. Google has a very recognized brand name, but so does Yahoo! The difference is that Yahoo! is more of a Portal than strictly a Search Engine. They have a lot more data (hopefully anonymous data!) about their users and their habits than Google could ever have with their business model. Which will be an important factor going forward, but that's another discussion.

Next, around the same time, MSN (which now uses Inktomi data, which is owned by Yahoo! ... yes it's convoluted) publicly announced that they were developing their own search technology. This had been known for some time by insiders, but the announcement made it official.

MSN, which is of course owned by Microsoft, is the proverbial 800 pound gorilla. They have both Google and Yahoo! quite concerned. Not only do they have significant cash to throw at the market, but they are also marketing wizards! Whether you like the folks in Redmond or not, you have to admit that they have done a very, very effective job of marketing their products.

The part that is the scariest for G and Y, and everybody else for that matter, is that Microsoft dominates the PC market. It's their operating system on the vast majority of the world's computers. It would be laughably simple for them to integrate their search product directly into the operating system and make it the default search product for most users.

And while Yahoo! has a fair amount of data about their users, the information they have at their disposal pales by comparison to that MS has available by owning the platform.

MSN, at this moment in time, is still using Inktomi data to seed their search results. I expect this to change within the next 30-60 days though. I really expect that MSN will have their own search technology in place on their site before the end of July 2004. And start a major marketing blitz right after it goes Live.

If I were Google or Yahoo! I would be very scared, just like I'm sure they are. Especially if I was Google and had an IPO worth billions set to roll out.

That brings up up to today. Next we'll take a look in Randy's Crystal Ball to see what the next "killer app" in search will be about.

Posted by Randy at 09:10 AM | Comments (0)

May 24, 2004

Search Engine Schtuff

Let's get started on some of these search engine marketing posts. First a (very) brief history of Search.

In the stoneage (mid-1990's) there were a whole slew of search engines. Altavista, Lycos, Yahoo, etc, etc. None of them were all that good since Search was in its infancy. Heck, the world wide web itself was still a baby!

They all did okay sorting things out, but you have to remember that back then there were merely a million or so documents on the web. Nothing like today when the various search engines tout that they have 5 or 6 Billion pages indexed, and freely admit that they're indexing less than 1/10th of the pages available.

Then along came the little engine that could, better known as Google.

Sergey and Larry came up with a new method of determining which pages should be listed before other pages. At the time their new method, or algorithm, was pretty revolutionary.

Google's ability to return relevant results to searchers, and the sheer size of their database of sites when compared to the other search engines gave them a very definite advantage over everybody else. By the late 90's Google absolutely dominiated the search market.

In addition to the searchers who used Google.com, anyone searching on Yahoo.com, AOL.com and several smaller search properties all saw Google results. For all intents and purposes search became a one-horse town, with Google directly controlling approximately 3/4's of the market.

That brings us up to the end of 2003 for the most part. I've left out a lot of niggling details over the years, but that'll bring you up to speed.

Next time... We look at the *Massive* changes we have seen since the beginning of 2004, and a bit leading up to the new year.

Posted by Randy at 10:49 PM | Comments (0)