How To Get Pages Out Of Google's Supplemental Index (SI)

About Google’s "Supplemental Index" – and how to get pages (URLs) out of it.

If you don’t know what the "SI" is, crack open your copy of SEO Fast Start and begin reading at page 51.

If you have pages in the Supplemental Index, here’s how to remove them, step by step. These instructions assume you have fewer than 1000 pages on your site. If you have more than that, read them, follow them, and then watch this space for further instructions on how to troubleshoot a larger site.

1) Start by going to www.google.com, and clicking the "Advanced Search" link to the right of the search box. In the advanced search form, select "100 results" from the drop down menu, and type your domain name in where it says "Only return results from the site or domain." Then click the Google Search button.

2) The search result page (SERP) will probably say something like "1-100 of 682 results." If they’re showing you less than 100, skip to step 4.

3) Scroll to the bottom of the page, and look for the numbers (1 2 3 4 etc) under the Gooooooooogle graphic. Click on the largest number (for most sites, the number 10). This will take you to the last page of search results.

4) Scroll down to the bottom of the last page of search results, and look for a link that says "repeat the search with the omitted results included" – click that link.

5) This will take you back to page 1, showing 1-100 of however many results.

Now, as you move through these search results from page to page, you will notice that some of the pages are labeled as a "Supplemental Result." Bummer, eh?

Introducing The "Buddy System"

Just kidding… we learned all about the buddy system at camp, right? If you ever let your buddy get out of sight, one of you will be eaten by a bear. What we’re going to do next is similar to the buddy system anyway.

You have two kinds of pages (URLs) – either they’re in the main Google index, or they’re in the supplemental index. Actually you may also have some pages that aren’t in either index, but that’s a whole other problem that I am not going to solve today.

If you have the time, make a list of all the "supplemental" pages, and all the "main index" pages. I’m sure there’s some easier way to do this than doing it by hand… but I don’t know what it is. That’s why I say, if you have the time. Maybe you’re real busy. I don’t know. Maybe too busy to deal with this whole SI problem. If you are too busy, pick ten (10) of each, or some other number you can manage with.

For every page that’s in the SI, pick a page that’s in the main index to be it’s "linking buddy." So if Page A is in the main index and page B is supplemental, maybe Page A can be Page B’s linking buddy. Now, go put a link on Page A that points to Page B. It helps, of course, if these pages are logically related in some way. Do this for all of your linking buddies.

Wash, Rinse, Repeat…

Now go away for a couple of weeks. Take a trip to the islands or stay at home and get some work done – maybe do some work to promote your web site so that it has more PageRank to begin with (start reading at page 63). Then repeat the steps above, observing which pages have just gone from the supplemental index to the main index, and giving those pages a little time before you assign them to be another page’s linking buddy.

If you discover that the number of supplemental pages far outnumbers the pages that are actually in the main index, so that you can’t find a linking buddy for everyone, you may want to look at your site structure (Chapter 4, don’t make me look up the page #), and of course, you could always help yourself out by building more links.

This process can be time consuming, but as long as you’re doing a reasonable job of building links to your site, you shouldn’t have to keep doing this stuff forever. Moving pages into the main index means that those pages can now participate in building your site’s PageRank, and that they can pass link reputation to your other pages.

It’s a good thing. Trust me. Supplemental results aren’t the end of the world (thanks for letting us know, Matt Cutts) but they aren’t a delicious chocolate sundae either. I prefer sundaes. That’s why I need to go to the gym.

For large sites (over 1000 pages), see you next week…

If your site has more than 1000 pages, you should be able to find a few supplemental pages by following the steps above, but what you really want to do is use a tool like GSite Crawler to spider your site and map out all the URLs, then check Google to see which ones are supplemental.

I’ll be back next week with details on how to crawl your way out of the supplemental index with a large site.

For anyone who found that this moved a little too fast for you, please read SEO Fast Start (it’s free) and get up to speed. Thanks!

39 thoughts on “How To Get Pages Out Of Google's Supplemental Index (SI)

  1. I am amazed at the quality and quantity of instruction you offer.
    As I am still new to creating my own site, I had not enlisted your full professional services yet.

    But it is time.
    Not only do I feel quite confident it will be a great investment, but you deserve it!

    I have forwarded your site to a web developer who does
    in-house “listing” for my brother’s land development corporation.
    He has told them to consult with you or he’s leaving with his account.
    (I only wish you had an affiliate program!!)

    Thank You again for the terrific information!
    -Ed

  2. Hey Dan,

    Great stuff, as usual. I checked my site and of the 600 pages shown, only about 100 are supplemental and they are all dynamic pages generated from my CMS. Will pages like these get moved from the SI if I follow your suggested process?

    This stuff is truly fascinating.

    Kelley

  3. Hey Dan,

    Thanks for the tip; however, my website is already set up like that (i.e. pages that are not in the supplemental have links to pages that are in the supplemental). I know it helps in general, but would it make a big difference if I made the links on the keywords? In other words, I have a sectional page that is not in the supplemental. It has several products on that page, but the links to those product pages are embedded in the image and in the word “Details” instead of the name of the product being a link. Please let me know how important that would be to change that so the product name is a link (note: the product name is also the Title of the detail page). Thanks for the help!

  4. Hi Dan,
    I followed your instructions on finding pages on the site in the supplemental index and was shocked to see that nearly all my pages were listed there. I don’t understand how that could happen since most of the pages are unique with different content.

    Can you take`a quick look and explain what you think has happened and the fastest way to recover.

    Thanks.

    Ray

  5. There are a few things that we know can put a page into the SI:

    1) The #1 cause is low PageRank, and simply pointing more links at the page as described is usually all it takes. Sometimes you need to bring more into the site overall or the site structure is totally messed up, but 9 times out of 10, you can bring a lot more pages into the index just by making a few small structural adjustments.

    2) Duplicate content appears to be another cause – empty or near-duplicate pages can wind up in the SI, even when sibling pages that should have equal PageRank are in the main index.

    3) Pages that have gone, 404 errors, stuff you aren’t linking to any more, can linger in the SI for a long long time. Naturally, we don’t need to put an effort into bringing non-existent pages into the main index, but it can be worthwhile to redirect (301) these pages if the content has simply been moved or replaced.

    Speaking of redirects…
    It will probably take a few weeks to get to this topic, but I’ll have to do a piece about link recovery using the Google Webmaster Tools console and 301 redirects. I catch a half dozen mistyped inbound links every month, and redirecting them to the right page is a free link. :D

  6. Dan, I love your writing style. I had a good laugh, because as you know from the posts I’ve been leaving…yeah, i have supplemental pages!!! The cadence of your writing mirrored the roller coaster ride this can be sometimes.

    Thanks for the step by step. Great article.
    I’m going to get my sister to read it as she is my buddy in crime. It’s HER site we are banging around and she’s a real trooper. Me, I’m going to be doing this stuff for many of my clients, eventually. And you are helping me along the way. I really appreciate it…

    claudia

  7. Probably the best explanation I have seen on how to find supplemental pages. For the longest time, I have never been able to accurately identify them. A quick search of my site reveals that I have about ten. Not good..

    Even better is your suggestion on how to get out of limbo. A simple A to B linking method with those in that are indexed – is not all that hard to understand..

    Thanks for this..

    Robert C.

  8. My trick for SI is put content of at the header of website. I’m using php on my websites so I just put little code at the top to show exactly the same text as title, but under the In each case it helped. Now every new website I’m designing with space for this one line of text.

  9. Hi Dan,
    I have been reading your analyses for about two years now and I enjoy the plain language that you use. I really appreciate the analysis you have offered me, and I read with interest all of your articles.
    I work in the French-English market, so I don’t know if my question is an issue of Google France being underdeveloped, or something I am doing fundamentally wrong. Basically supplemental results don’t appear in the listings, and two thirds of the pages are missing completely (the site has been up for 3 months in it’s current form, and launched 2 years before). I like the idea/logic of buddy-linking pages but I wonder whether the fact that I am dynamically (PHP) generating the link menu causes problems (i.e. creating same ‘title’ link tags descriptors). Is that wasting a SEO opportunity, making everything banal and unremarkable to Google? Anyway, just thinking out loud here… no need to reply. Thanks again.

    Paul.

  10. Thanks all for your support!

    Kevin, Webmaster Tools does have links to the site: search under Statistics / Index Stats, but they don’t have an SI report as far as I know. They’d really rather we all forgot about the SI altogether, and there’s a better-than-zero chance that they’ll stop displaying "Supplemental Result" on SERPs at some point.

    We can (and will…) use Webmaster Tools to help us find SI pages, though. If you look at your Links / Internal Links report, you can see the pages that have very few links. So, yeah… it’s a tool we can use.

    Paul, your content management system should give you the ability to put a static link onto a URL – by editing the article or something. You don’t have to have all of your internal links in those dynamic menus.

  11. Hi Dan
    Really appreciated your help in this. I’ve read you advice SN forum and it looks like I’m finally climbing out of SI Hell.
    Thanks

  12. Hi Shimon – We also have php sites and I couldn’t understand what you were talking about when you said:

    …put content of at the header of website… I just put little code at the top to show exactly the same text as title, but under the In each case it helped.

    I think there are some words missing from your post. Can you please rephrase it to make it a little more clear?
    Thanks!

  13. I just did a search on my clients name to see what articles were showing up ‘out there’ and i’m finding that many of the SERPS her articles show up in, are supplemental! If i’m thinking of this correctly, those articles probably won’t show up for any keywords (except her name, since it is so unique). Is that right? This means these articles are doing her no good??? Don’t think i really want to hear the answer to that, but what can i do to make the articles worthwhile?

    Thanks!

  14. Claudia..

    I, too, have articles that are in the supplemental index (these article are on my website). I also have one article that is not in the SI.

    I noticed that some people have linked to that one article in particular, so that could be the reason why some pages – and article pages fall into the SI results.

    I am no expert here, and just found out yesterday where they were,(supplementals) thanks to Dan’s post. But, since then, I went looking for more information on the subject.

    I am paraphrasing here, but there was an article by – I think – Jill Whalen that mentioned this tid bit of information.

    1) SI results happen because the page has not changed, or Google thinks that it is not that important.

    2) Just like your main page, you need other sites to link to that particular article – or page – in order for it to grind it’s way back to importance.

    Or, as Dan suggests…

    “For every page that’s in the SI, pick a page that’s in the main index to be it’s “linking buddy.” So if Page A is in the main index and page B is supplemental, maybe Page A can be Page B’s linking buddy.

    Now, go put a link on Page A that points to Page B. It helps, of course, if these pages are logically related in some way. Do this for all of your linking buddies.”

    Great advice and I am going to be doing just that – as well as trying to get more people to link to those article pages that are currently in the SI..

    Hope my un-professional opinion helps..

    Robert C.

  15. Hey Dan,

    Thanks for the great step-by-step info on getting pages out of Google HELL (SI). You never fail to deliver practical ready to use information and this clearly puts you head and shoulders over other SEO’s that I follow.

    On my website I was observing that the links that I had placed in the footer navigation were in the main index but many of the individual credit card and content pages that weren’t placed in the footer navigation were in the SI.

    I’m testing a strategy that’s a bit different to your buddy system whereby I’m adding pages to my footer navigation in an attempt to pass more page rank to the SI pages as well as get them out of Google HELL (SI) and into the main index.

    I’d be grateful if you could share your thoughts in this strategy.

  16. Hey Dan

    Great article – simple and concise. I still see allot of people saying that pages are in the supplemental index because of duplicate content (off site dup content). Of course much of this comes from people selling article spinners – etc. It is finally getting into my head (after 3 years)that the meaning of life is links, links, and more links.

    Thanks for this site.

    Valerie DuVall (Eric Shannon’s favorite Stomper)

  17. Dan, Thank you for responding, and you are right. Every product page on my site can be accessed as .com/page or .com/page.html – But there is only one page.

    I guess that there are two url’s that can be used to address that page.

    I have a Yahoo store and I use Dreamweaver to build the pages.

    When I build a page, Dreamweaver automatically assigns a file name with an extension of .html. Then it is loaded to the file site.

    This page can be accessed by using domain.com/page or domain.com/page.html – but it is only one page.I don’t understand what you mean that these are two pages.

    Can you please explain how to avoid any problems that this dual address of the same page causes. For example, when I put a “return to” link, I point to the page name and it automatically fills in the address with the extension of .html –

    Since this is only one page accessed by two different addresses, I don’t understand how this is a duplicate content issue.

    What do I need to do???

    Thanks,
    Ray

  18. Dan, I think this article is your best one yet! I think it is telling from the record number of posts in the last few days.

    Now let me give you some ‘Creamy Sundae’ advice as I have been known to have quite a sweet tooth myself. I figured it would be okay to go a little off topic since you started it. And it’s good content!

    It’s okay to have a sundae… But…
    1) Try to eat it early in the day
    2) Drink plenty of water prior to
    3) Size does matter…so go easy on your portion
    4) Try to eat it on a training day
    5:) What’s for desert after the live event?

    Jeff (:

  19. I am glad to report that no page appeared in supplemental list for two sites I am associated with.

    Thanks again.

    By the way when Dan said :Take a trip to the islands…” he was referring to the Caribbean, so please take his advice.

    Cheers ;)

  20. Another way of looking for supplemental results is:
    In the google search box, enter: site:www.yoursite.com *** -sjpked

    This will give you a list of only the supplemental results. Go to the end and click on the “repeat search” link so it gives up all of the supplementals.

    Since my site serves up dynamic (product.asp?codeXXXX) and html (/product_XXXX.html) versions of my pages I’ve got a ton of pages in the SI. I repeated the search to only show .html pages to get a result of the approx 170 pages that are supplemental. A lot of those pages will be improved with some more copy added to them but they’ll get some more link love also.

    Question – would it be wise to go ahead and “kill” the ASP pages using the robots file to disallow the asp pages?

    Cheers,
    Craig

  21. Ray, the narrowest definition of “duplicate content” is:
    “The same content found at more than one URL.”
    That’s exactly what you have.

    Craig, I recommend picking one version for your URLs, and keep the other version from getting indexed – use robots.txt or redirect (301 permanent) the other version, depending on the technical specifics.

    I will try to get a deeper look at duplicate content into the portal pretty soon.

    Thanks for posting that search hack, BTW, Craig. Didn’t know that was still working!

  22. Dan – I uploaded the robots.txt file earlier today. I’ll be tracking how that does over the next few weeks.

    Now it’s off to tweak the buddy system to get my internal page structures working better. They are not anywhere near efficient enough right now.

    Craig

  23. Hi Dan

    I have a quick question for you. I have just read your original comment and you say that you should add a link from page A (main index page) to Page B (Sl page).

    Can I also add a link back to the main index page from the Sl page?

    Look forward to your response.
    Rachel

  24. Dan,

    You should realize that all of the pages that are supp probably already have at least one link from a non-supp page. If you go 2 levels deep, and find pages that are linked to only from supp pages, chances are that the bottom level pages won’t be supp… they won’t be indexed at all.

    The problem with supp pages isn’t that they don’t have any non-supp pages linking to them, it is that they don’t have enough PageRank flowing in to them. True, with the rinse and repeat, eventually you might get the link structure balanced… but honestly, that’s just redistributing the PageRank you already have flowing in to the site. If you have more supp than non-supp pages, and already have a decent link structure, then you might just not have enough PageRank coming in to support your entire site, period.

    Yes, internal linking structure does play a huge part in it, but all on it’s own it probably won’t solve most people’s problems in the long run.

    Also, just so you know, the duplicate content in and of itself doesn’t contribute to supps, except that if you have multiple versions of the page then you are dividing PageRank efforts needlessly. Wouldn’t matter though if the pages were identical or substantially different.

  25. Dan, I do not know how to fix this problem. I have to make the pages using the .html but customers can access the page without entering the .html. I did not know that that is considered to be 2 pages.

    Dreamweaver uses relative links that result in all internal links being domain/page.html I don’t know of a way to change that except by manually changing each link to a fully resolved link without the .html and it would appear that you would lose some of the advantages of dreamweaver that updates links when you rename pages.

    You and Craig mentioned using robot.txt files and 301 redirects to solve the problem. Can you give an example of robots txt file that would exclude domain.com/page urls but still index domain.com/page.html urls

    Could you specify exceptions? For instance I have changed all my internal links to my home page to be domain.com versus domain.com/index.html External links have the same URL.

    This was done not with the the idea of duplicate content but because either you or Leslie Rhodes pointed out that you would be dividing link popularity,page reputation,etc among two urls instead of concentrating on one.

    Since I am currently stuck with the problem of most of my product pages and some of my section pages apparently viewed as duplicate by google I would appreciate any advice on the quickest way out.

    What do I do?

  26. What luck! I came here to post a question in the “Chapter 4″ section and it’s already been addressed here.

    Dan, I’ve read your book and I like it better than the SeoBook. Thank you for the “fat free” version to SEO and real steps to follow. Not that I’m ragging on Aaron’s book, I loved it too, but your book has given me the broad overview I’ve needed. I’m a website designer, I don’t want to be an SEO expert/consultant. Your book has given me the introductory tools needed to get started. So, thanks!

    Hope to get more involved in the community as I begin taking action!

    Paul

  27. Dan – are menu navigation links counting or do you mean text links written into the body copy? I have an ecommerce site but it’s only three months old so I guess I should just be patient.

    JON

  28. Michael, thanks for driving by… hopefully this won’t be your only visit. Nobody’s going to argue with you that we all need to promote our sites and get more inbound links. I mention this in the original post, and there’s a whole chapter on it in the book. :D

    However… I have to say again, that one way to give a page more PageRank is to give it some from inside of your own site. This is easy to do, and getting more pages indexed actually increases the amount you have to “redistribute” within the site.

    As far as your opinion on the SI and dupe content, I can’t buy what you’re selling. We’ve seen it too often. BTW, the Google rep at SEMNE (http://www.highrankings.com/advisor/getting-into-google/) recently mentioned another type of candidate for the SI: “pages that don’t change very often.” Got one of those myself…

  29. Jon, I know your addressing Dan here but… Dan previously mentioned in a video that he does not consider links in the menu nav as an effort. Body copy is the way to go.

    Jeff

  30. Ray, I don’t see a robots.txt solution for your situation. I’d use redirects.

    Thanks for your kind words, Paul. I hope you’ll have some time to participate in this community. I’ve read the SEO Book, and I have nothing bad to say about it at all… it just serves a different purpose. I knew Aaron before he started writing it and it’s gotten better with every update.

    Jon, if you’ve got a page in your navigation menus, and it’s sitting in the SI, you may be dealing with one of the other causes. In general, when I talk about adding a link from one page to another, I’m talking about using text in the main body of the page.

  31. You can either redirect like Dan has said (the best way because it also redirects any link power) or you can use the robots.txt to block the page you are talking about.

    Example (works only for Google and Yahoo)

    User-agent: *
    Disallow: /*.html

  32. Dan,

    Thank you for the welcome, and I will be browsing by here from time to time. :)

    I would love to hear verbatim what the G rep at SEMNE actually said, I have never even heard of a page being stale as a reason for for supps. I regularly see pages that haven’t changed since 1997 out there, never noticed a problem.

    As to the duplicate content issue, Google themselves actually confirmed it not too long ago:

    http://googlewebmastercentral.blogspot.com/2007/06/duplicate-content-summit-at-smx.html

    I also have contrived of a way to demonstrate the point, but as of yet have not had time to finish setting it up. Hopefully soon.

  33. Yes your info is very good, i hope this way will work well for me, as im suffering from supplemental results for a longer period now. I have one doubt, you spoke about link buddy,it should be only one, or more to a page?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>