The Chatterbox

Download Google Rank Extractor

Copyright and Trademark Disclaimer: Neither I nor this tool have any connection with Google, other than the fact that the tool utilizes referral strings from Google to process some ranking information. Google Inc of course continues to own the term "Google" and I make no claim to the word. Its use with regard to this tool is simply as a reference.

What is Google Rank Extractor?
Google Rank Extractor, or GRE for short, is a small ajax/php/mysql application that lives on your server and watches for traffic that shows certain Google information the referrer string.

What does it do exactly?
When it sees one of the Google referrals that contain ranking position data it extracts and stores information related to the search and hit to your web site.

What information does it save?
Well, the search term the user typed into Google; The ranking position of your site for that phrase at the time the user conducted their search; What landing page of your site to which the user was taken; The date and time the hit was seen; What version of Google the visitor was using (eg Google.com, Google.co.uk, Google.com.au, etc) and if it was one of the regional versions of Google whether the visitor had selected a Global or In Country type of search.

The easiest way to see what info it collects is probably to view the Demo of the tool I've set up here on my personal blog. Though if you're going to use the demo you'll probably want to read the section below entitled Usage of the Google Rank Extractor

Wow, that's a lot! Will it bog down my server if I use this Google Rank Extractor thing?
No it won't bog down anything. From the beginning GRE was built with speed and flexibility in mind. In fact the two files that fire if there is a positive hit are just 693 bytes (as in tiny, less than a kilobyte) and about 5 kb. Neither you nor visitors should notice any page load time difference at all.

Okay, that's good to know. But I've heard that Google doesn't like people using automated tools to see their rankings. Will Google penalize my site if they notice I'm using Google Rank Extractor?
You're 100% correct in saying automated rank checkers are against Google's TOS and have been known to take action against those using them. But they're after the ones that run automated queries against Google's site because these sort of automated tools tend to use up a lot of resources Google would rather be used by real searchers and not bots.

GRE is different. It works on the other end of the spectrum. And it never, ever, ever queries Google. Instead it gobbles up the information Google has so graciously started providing in their URL Referral Strings and records what it needs directly from this referral string as sent by users browsers. In other words, the tool gives you ranking information in real time from real users without ever once connecting with Google. Thus it is completely within Google's Terms of Service and completely safe to use.

Great! How hard is GRE to install?
Well, that's a bit of a How long is a piece of string question now isn't it? It sort of depends upon how your site is constructed and mainly whether you have any common files, like a header or footer file, that gets automatically included in each page of your site. If you have one of these common files, installation from beginning to end should take you less than 5 minutes. If you don't have a common file it will probably take a bit longer.

The best way to answer this question is to say that installing GRE should take you no longer than it takes to install something like Google Analytics. The trigger mechanism in both is a simple call to some javascript, so the installation time for the two is comparable.

Alright, you've convinced me. Where to I get this Google Rank Extractor thing?
You can download the php/ajax version right here.

If you would prefer a Perl version (it also incorporates Ajax calls) Craig Chant has ported it and that's available here. Craig's version has some visual improvements from the original version, namely he's added utilized Fusion Charts to add some graphic based charts to aid visualizing the data.

Okay, how much does it cost?
GRE costs nothing. It is free software that carries a standard GNU type of license. Meaning you can do pretty much anything you want to do with it. The only exception being that you cannot steal my code and then try to charge for it. ;-)

You're kidding! It's even free? Wow! Are there any instructions I need to know?
The installation and usage instructions are included in the README.txt file in the zip file I linked to above. Or if you would prefer to read them online, keep on reading. I've put both the Installation and basic usage documentation here too.

Wonderful! I'll give it a whirl right now!
Have fun with it!

I've installed GRE and having some trouble getting it working properly. What can I do to get this fixed?>
Though there is not full support offered, drop Randy an email via the GRE Contact Form. I'll try to help you get it all sorted and working for you.

I'm a developer and have questions or have an improvement. How to I get this to you?
As above, use the GRE Contact Form to drop me a note.

---

Files Included In This Distribution
rank_extractor_ajax.php - The file that collects the data.
gre_ajax.js - The javascript file that kicks off the Ajax processes to extract and save info.
README.txt - A file with instructions and documentation.
gre_db_setup.sql - The SQL instructions to set up your database.
index.php - The admin user interface.
gredb.php - Contains database connection info for the tool.
gre_admin.php - The php file that provides the admin interface with its data.

Quick Installation Instructions
1. Download and unzip the zip files.

2. Upload the rank_extractor_ajax.php file to the root level of your web space. (Note: You can use it for sites where the root level isn't available for use, however it will require one small edit to the gre_ajax.js file on line 24 to provide the correct path.)

3. Upload the gre_ajax.js file to the same location as the rank_extractor_ajax.php file.

3. Create a MySQL database via your hosting Control Panel to store the data. You can name this whatever you would like.

4. Run the SQL commands included in the file named gre_db_setup.sql to set up your database.

5. Edit the gredb.php file with your database information.

6. Upload the index.php, gre_admin.php and gredb.php pages to a subdirectory at least one level below the location of the rank_extractor.php. This will be your Google Rank Extractor Admin area. By default this subdirectory is named /gre/ though you can change this if you would like.

7. If you changed the default /gre/ admin location edit the $admin_path variable at the top of the files rank_extractor_ajax.php file so that it points to the correct subdirectory. Upload your changes.

8. It is highly recommended that you password protect the admin directory. If you don't you're potentially leaving a fairly large hole in your security for competitors to exploit. Do you really want to hand the competition all of the keyword and ranking information for your site? I thought not. So please password protect the admin subdirectory.

9. Insert a script reference in each of the files of your site, or in all of the pages you wish to track. If you have a typical blog or other site that utilizes include statements (php include, ssi include, etc) you probably have a common header or footer file you use to populate some basic html information for each of you pages. For easy installation simply put the script reference in one of these header or footer files. It doesn't matter where in the page code the gre_ajax.js reference appears in your code, just that it's there somewhere.

A typical php include would look like:

<script language="javascript" src="/gre_ajax.js"></script>

To see it on a web page view the source of this page. It'll be towards the top of the page in the head section. You don't think I'd not have GRE installed here to see what search terms this page ends up ranking for did ya? :::giggle:::

10. That's it. You've got everything installed. Sit back and relax. Give the tool a day or two to start collecting information, then visit the Admin area to see all of the data.

Usage of the Google Rank Extractor
GRE uses the new Google Referral String that first appeared in April 2009. Without going into all of the gory details, Google has made a change to how they structure referral urls that get sent when a searcher clicks through from a Google search to a site. As part of this change Google has started to include the ranking position of the keyword phrase.

In a nutshell, GRE does exactly what its title says. It extracts this ranking position information, as well as some other data, from each referral hit sent to your site from a Google search. The end result being that you no longer have to query Google to figure out where your site ranks for its various keyword phrases. In fact, the GRE gives you a lot better and more consistent information that you could ever get by querying Google, mainly because you get data that tells you exactly what the searcher saw at the time they clicked through to your site. GRE does not in any way violate Google TOS like so many of the automated rank reporting applications do. It is completely Google friendly.

--GRE Functions--
When you log into the admin area after letting the tool collect information for a couple of days you're going to be presented with a basic html form. This form will allow you to retrieve and sort the data in several different ways. No doubt you will be surprised at some of the data once you start to delve into the details.

The GRE admin user interface makes use of a of PHP and Ajax to retrieve and display the information. Below is a list of each of the admin user interface form, followed by a brief explanation of what each area of the form does.

1. Maximum Postion Filter
2. Phrase Filter
3. Page Filter
4. Google Version Filter
5. Date Range Filter
6. Sort By data display sorter.

--Leaving All Forms in their default or blank state--
Leaving everything blank will return the maximum results. In essence it will return every single hit your site's pages received from a qualified Google Referrer.

--Maximum Position--
The Maximum Position filter allows you to place a limit on the results being returned, based upon the ranking position of your site for the search phrase each user entered to find your site. For instance, if you only wanted to see the hits you received from your site's Top 5 ranked keyword phrases you would put the number 5 in this field before submitting your query. This would exclude all referrals that came through with a ranking notation of position 6 or higher.

--Phrase--
The Phrase filter allows you to narrow down the returned results based upon the keyword phrase used by the visitor. As a default this field conducts a MySQL "LIKE" type of search on the database, not an Exact type of search. In English this means if you plugged in a phrase of "red lederhosen" the tool would of course return any results where the user searched specifically for the phrase "red lederhosen". However because the default is a LIKE type of search the results would also include phrases like "pretty red lederhosen", "german red lederhosen" and "red lederhosen germany".

A slight caution is in order here. Because of the way MySQL's built in LIKE function works it is not a true Full Text search. What this means in practice is that there is no scoring mechanism to the phrase you enter, unlike a normal search. So if we use the "red lederhosen" example above there are several variations of the phrase that would NOT be returned in the display results. For instance, a search for "red lederhosen" would NOT return a result for Google searches of "lederhosen red". Nor would a result be returned for a phrase like "red and green lederhosen" or "red goat hide lederhosen".

This is an issue that simply cannot be overcome without a fairly major update to both how the data collected by the extractor is saved and also how the database is constructed. It can be done, but doing so would increase the complexity of the back end process significantly. If enough people want this type of Full Text search feature I'm willing to code it, either for a future version of GRE or potentially by maintaining two versions of the tool, however I'm not sure it's necessary at this point. Especially since I expect there would be unintended side effects that would have to be dealt with because of the added complexity.

If you're careful in how you enter your phrases, sticking to one word or commonly used two word phrases, you shouldn't need the more complex capabilities of Full Text types of searches.

Okay enough of that.

At the suggestion of one of my Alpha version testers (Jill Whalen suggested it first for those interested) there is also a toggle to switch the Phrase filter to Exact Match mode. Exact Match means exactly what it says. Using the above example again, a search for "red lederhosen" in the tool would return ONLY results where the original search was for exactly this phrase. Nothing more, nothing less would get returned.

This is useful if you have a specific phrase you want to track. Especially if you're trying to optimize your site for a certain phrase and want to track its improvement over time.

--Page--
The Page filter is quite a useful piece of data, in that it tells you quickly and easily which pages of your site people land on when they conduct certain searches. Not only does this show you which are your most important landing pages, but if you use the information wisely it will allow you to tailor the message you're communicating on each of your landing pages so that it better matches what visitors are looking for when they hit those pages.

This one can have some important implications not only on your SEO efforts, but also on your conversion efforts. Because knowing exactly what people were searching for when they landed on one of your pages can give you a major hint at what you might need to say to them in order to improve your conversions.

--Google Version--
This was a suggestion from several of my Alpha testers. In a nutshell it tells you which version of Google the visitor used to find your site, as well as where your site ranks for search phrases in each version of Google. By version of Google I mean the global and regional or geo-location versions. eg Google.com, Google.co.uk (United Kingdom), Google.com.au (Australia), Google.ca (Canada), Google.co.in (India) and so on.

The results of this one surprised me a bit with the sites I used to test the original GRE code. I was frankly amazed at how well my .com web sites that are housed on US based servers rank all over the world. For their most competitive phrases in many, many cases my sites seem rank better on several of the regional search engine versions than they do on the .com version. Which makes no sense at all given the search engines general push towards personal search results.

Additionally, the tool will tell you if someone searched one of the localized versions of Google with a Global type of search, or if they had selected "Show sites from XYZ Country". At the time of this writing the Google Local versions are defaulting to Global types of searches. It takes an additonal action by the searcher to change to a Country-Only type of search

Note: There is nothing built into the back end that lists all of the various Google versions. Instead the tool looks at the actual referral string to see where the hit came from, and this information is used to populate the Google Version dropdown box. So if you see a regional engine listed in the dropdown box, your site has received at least one hit from that version of Google.

--Date Range--
Another suggestion from the Alpha testers. This one does exactly what you would expect it to do. It allows you to filter the results being displayed by the date of the original hit. So if you wanted to only look at results from a certain one week period you can.

A fairly important note: As things stand right now I don't have any error checking built into the Date Range function. And given how MySQL's built in BETWEEN function works you'll want to make sure the date selected in the left hand box is always the earlier date and the date selected in the right hand box is always the later date. You won't break anything if you switch them around, however you will not receive any results.

--Sort By--
Sort By is NOT a filter and does not exclude any results. Instead it simply changes the order in which results are displayed. So it allows you to group the results in different ways, to hopefully make it a bit easier to use. The currently available sort options are as follows.

a. By Date in Ascending Order (the Default action)
b. By Date in Descending Order
c. By the search phrase entered by the user, Alphabetically in Ascending Order
d. By the search phrase entered by the user, Alphabetically in Descending Order
e. By the Ranking Position in Ascending Order
f. By the Ranking Position in Descending Order
g. By the Google Version used by the user
h. By the Landing Page name, Alphabetically in Ascending Order
i. By the Landing Page name, Alphabetically in Descending Order

--Combining Filters--
Though it may not be evident at first, you can combine the various filters and sorting functions available in the tool to focus in on more specific data. This type of combining is quite powerful if you need to drill down to a granular level. As a for instance, let's say I was optimizing my site for the phrase "red lederhosen" and its similar offshoots and wanted to track how it was doing both in Global and Regional searches.

I could simply plug red lederhosen into the Phrase area of the tool and then toggle through each of the Google Versions listed to see if and where my site ranks for those phrases. And see quickly and easily if it's ranking is improving over time because of my optimization efforts.

The same general concept can be applied when combining any of the GRE filters. The more filters you combine, the more focused results you'll get.

That's pretty much it for now. If you have any suggestions for further improvements, questions or comments please leave them as comments on my blog at or drop me a note via the GRE Contact Form. If you find the tool useful I would appreciate a link back to the above page or directly to the Google Rank Extractor info/download page at http://www.randycullom.com/google-rank-extractor.php.

--Suggested Improvements-- If you're still reading this far down this very, very long page you're really a glutton for punishment aren't you!?!? What does that mean to me? It means chances are pretty good you might be a code head with an inquisitive nature. So you might well be interested in improving the Google Rank Extractor tool!

Improvements I'd like to see but simply don't have the free time to implement at the moment:
1. Port the code to other server side scripting languages, like ASP, Perl and ColdFusion. This would involve porting two php files: rank_extractor_ajax.php and gre_admin.php. Shouldn't be too difficult to do, however I just don't have the free time right now to support additional scripting languages.

If you make such improvements I would love to hear about them! In fact, as long as the improvements are useful I would be happy to host your improved version right here on the same page as the original so that more people will see it. Please let me know about your improvements via the GRE Contact Form.