Zazzle not allowing search engines to index products or collections

[AtypicalPuzzles]

This question is mostly directed to the Zazzle staff, but I'm really curious to know what the logic is behind Zazzle blocking search engines from indexing products and collections as shown by the following entries in robots.txt

Disallow: /*/collections
Disallow: /*/products

I would have though that it was in Zazzles' best interest to make as many of the creators products discoverable as possible.

I'm also interesting in what fellow creators think of this.

[JimCarnicelli · Accepted solution]

There's been a lot of discussion about this in the forum here. And in the Snuggle Hamster Designs Discord server. Zazzle did officially respond once that I've seen:

I think what's going on is that Zazzle's website is so huge and grows so rapidly that Google can't keep up with indexing it. Finding new pages, let alone noticing changes over time to those pages. I think they are trying to trim down some of what Google pays attention to in order to improve the apparent quality of the site. However. That does mean they are throwing a lot of products under the proverbial bus.

[Cat]

Interesting. Assuming I understand both Mark's reply and how robots.txt works, what I get from all of this is that they are noindexing pages with the words "products" or "collections" in the path - or at the end of the path, I can't exactly remember how those wildcard characters work. So robots.txt tells search engines not to index collections and the aggregate product pages, but not the individual product pages. To take some examples from the OP's store, the main product page is not indexed as it includes the word "products" in the path:

https://www.zazzle.com/store/atypical_puzzles/products
However, department specific product pages would be indexed as they do not contain the word "products" in the path:
https://www.zazzle.com/store/atypical_puzzles/crafts+party
https://www.zazzle.com/store/atypical_puzzles/art
Under this rule, individual product pages would also be indexed as they do not contain the word "products" in the path either:
https://www.zazzle.com/puzzle_6_animal_groups_chaotic_poster-256489388362216957
https://www.zazzle.com/puzzle_master_t_shirt-256934248135928747

Similarly, the page that lists all of the collections in a store would not be indexed:
https://www.zazzle.com/store/atypical_puzzles/collections
I'm not sure about individual collections. Their path does contain the word "collections" but if I understand the wildcard characters correctly, the word "collections" would need to appear at the end of the line in order to fit the criteria, right?
https://www.zazzle.com/collections/pi_day-119122072834691241

Anyhow, I do know that I have at least a few products that have been indexed through Google so however it works exactly, they're not preventing search engines from accessing the individual product pages. Assuming I've got this all correct, I don't think it's something to worry about.

ETA: I can't remember exactly how robots.txt works. If a page is noindexed, does that mean that Google can't see/crawl any links on that page? If that's the way it works, then it would make it harder to get your products and collections indexed at least through your store - it could still get there through the marketplace. Maybe that's a good reason to link directly to them from your own website? 

[Anne]

I'm not sure if I understand all this correctly, but if a collection is not being indexed by Google, would it be indexed when you place a link from your own website? Or would it "downgrade" your own website because of being a "blocked" link?

[Cat]

I'm not sure. I can't remember exactly how this all works so I'm not sure if collections themselves are being noindexed, or if it's just the page in your store that lists all of the collections (https://www.zazzle.com/store/your--store-name/collections). I do know that individual product pages are not being blocked by robots.txt, it's just the crawl path for getting there - and I can't remember exactly how that works either! If I have time today I'll do some research and post back with what I find. 

[JimCarnicelli]

Thanks Cat. Some thoughts.

To take some examples from the OP's store, the main product page is not indexed as it includes the word "products" in the path: https://www.zazzle.com/store/atypical_puzzles/products

Here's the specific rule Cat is referring to:

It would be more accurate to say that it tells webcrawlers to ignore URLs that end in "/products". Cat gave a valid example of an ignored page. I'm guessing that Zazzle considers the store product lists to be extraneous. Frankly I think that's a mistake because Google likes seeing cross-linking pages within a website rather than isolated pages like products with at best single links to them. But to be sure, this does not mean that if your product has the word "products" in it that it'll get ignored. Products have URLs like this: "https://www.zazzle.com/funny_cat_christmas_tree_medium_gift_bag-256508500093231016". They always end with the product ID, so "/products" could never be at the end.

I can't remember exactly how robots.txt works. If a page is noindexed, does that mean that Google can't see/crawl any links on that page? If that's the way it works, then it would make it harder to get your products and collections indexed at least through your store - it could still get there through the marketplace. Maybe that's a good reason to link directly to them from your own website?

In order to see a noindex meta tag on your page the webcrawler has to fetch the page. Odds are good it will record the fact that this URL does not want to be indexed and just avoid visiting it in the future. Although it may eventually come back around to check to see if that's still true.

Zazzle does provide several ways for Google and other webcrawlers to find our individual product pages. One way is through their sitemap. If you look to the end of the robots.txt file you'll see a bunch of "Sitemap:" lines:

This is a huge reservoir of specific URLs Zazzle is telling Google about. Google will generally regard the sitemap as the most valid and valuable source of links. I also read in a large portion of this for Snuggle Hamster Designs's own Niche Explorer and Niche Analysis tools. The sitemap is one way to point Google's webcrawler directly to product pages.

There are some practical limits to how many URLs you can point to. Zazzle's list of products is the largest portion of their sitemap. The last time I checked I think I found about 1.5M product links listed there. There's no way that this is the total number of products in Zazzle's catalog. That should be in the tens or possibly hundreds of millions by now. Consider that Wikipedia has nearly 62M pages as a comparison. My bet is that at most 1 in 10 products is in Zazzle's sitemap. And it's probably closer to 1 in 100.

It gets worse too. Part of Zazzle's sitemap is devoted to a list of all store home pages. But that list is not a complete list of stores. If memory serves me correctly there were something like 32k stores listed. For a long time we found our own stores were not in the sitemap. So I speculated that perhaps it was only a list of Pro members' stores. But after we went Pro I still did not see us listed. And I did see quite a few stores that are not apparently by Pro Zazzlers listed. So now I have no idea how they decide which stores to list.

So there is a variety of ways for Google's and others' webcrawlers to ultimately find your stores and products. Unfortunately there are holes all over the place. So odds seem very good that some products will not be found by crawling Zazzle's site and its sitemap directly. That does leave external links into Zazzle as a bridge. But one crucial problem remains. Once Google does eventually get to your product page, what if it contains a noindex meta tag? End of the line. Google respects that and decides not to include that page in its searchable index.

I do want to take an opportunity here to plug Snuggle Hamster Designs. Importing your stores to SHD is like throwing a lifeline to your stores and their products. All of our members' product pages are available to be indexed by Google. All of them are in our sitemap. Google regularly spiders our site. And all product pages link to their Zazzle pages where people can buy your products. We are in effect reducing the burden on Zazzle's website while increasing Zazzle's visibility and probably sales. Although we could bring in any part of Zazzle if we wanted, we only import stores that our members explicitly ask us to.

We aren't the only way you can get more exposure of course. You can take advantage of Pinterest and other social media sites. We offer our Premium members the ability to create SM posts using our beautiful templates and pipeline visitors through SHD on their way to Zazzle. But there's also the popular NiftyToolZ suite of social media management tools. There's your own website or blog site if you choose to make and manage one. And there's a variety of seasoned Zazzlers offering templates, tools, and mentoring in marketing. Like LeahG and Kate Thompson, to name only two. You can learn more about this and other topics via YouTube, especially on the Happy Zazzling channel. In any case it does seem as though now more than ever the burden of letting people and even Google know that your products exist is falling to you.

[Cat]

I think that in order for Disallow: /*/products to return true, the url path doesn't have to end in "products" as I originally thought, it only has to contain the word "products." Somewhere in all of the Google documentation that I read yesterday it said that those statements are only evaluating the beginning of the URL so no wildcard after the target text is needed for it to return true no matter what follows it. In this case that's a bit of a moot point since as far as I can tell the only place "products" appears in a URL is in the main products list where it is at the end of the URL.

Interesting point about stores in the sitemap. But I'm not sure how valuable it actually is to have your store indexed by Google since the crawler wouldn't be able to get beyond the store main page in terms of indexing products or collections. I mean, I guess people could search for your store, but I'm not sure how many people would do that - seems like they're much more likely to search for a product than a store. 

[JimCarnicelli]

I was going to knee-jerk agree that having your Store > Products page no-indexed makes having stores in the sitemap index moot. But it looks like @AtypicalPuzzles is correct. Google apparently will still follow links from a no-index page. Would it do the same for a page matching a Disallow rule in robots.txt? Looks like no. It shouldn't even visit your Store > Products page. Or anyone else's for that matter. So it would never even see those links. So that's Zazzle knee-capping the most direct way to discover our products outside of the sitemap. *sigh*

[Cat]

I think you're correct. What I take from this is that they want the crawlers to focus on crawling their marketplace rather than individual stores. Does bring up interesting questions in terms of the best way to go about promoting! I've just been linking to my store, or in some cases to my collections - but now I'm thinking that at least in terms of SEO and getting things indexed on Google, perhaps it's better to find a way to link to individual products... gonna have to figure out how to automate it in some way though since I've got well over 14K products!

[JimCarnicelli]

Not to beat a dead horse. But consider bringing ZingerBug and any other stores to Snuggle Hamster Designs. We can fetch all your inventory automatically for you. It may take a couple hours but it's automatic. And we'll keep it up to date mostly automatically too. (Some changes you make like adding cover images later won't be detected automatically.) You'll have a mirror of your store(s) outside Zazzle that is fully visible to Google and organized and presented in a way that we think is more pleasant and accessible to your visitors. We have loads of other goodies, some you can pay for, but this part is free. So if you're wondering how to automate SEO for Zazzle, we've got one easy answer for ya.

[AtypicalPuzzles]

---

ETA: I can't remember exactly how robots.txt works. If a page is noindexed, does that mean that Google can't see/crawl any links on that page? 

---

No index should just be for the content of the page itself, it should follow links as long as there is no "nofollow" tag.

[AtypicalPuzzles]

Yeah, search volume management makes sense given that there are multiple ways to get to products.

[Cat]

OK - I did some research and I'm pretty sure that collections cannot be indexed - even if you link to them from an outside website. There are several rules in the robots.txt file that reference collections, but the kicker is that there's also a meta tag in the html for the collection page that reads: <meta name="robots" content="noindex,follow" /> which disallows all search engine indexing of those pages, no matter who links to them.

You can still link to a collection from an outside site, but that won't put it into Google's index, so it can't be searched for.

In terms of the question @Anne posed, about whether linking to collections will negatively impact your site's SEO - the answer appears to be no - at least not directly. Here's what Google AI said about it:

No, there is not a direct SEO penalty for linking to a "noindex" page on another site, as the "noindex" tag tells search engines not to index that page, meaning it won't be considered when evaluating your site's ranking; however, linking to a large number of low-quality noindexed pages could indirectly harm your SEO by sending mixed signals about the relevance of your content to search engines. 

So, if your web page contains little content and mostly has links to Zazzle collections, that could negatively impact SEO. It also seems that having a lot of internal links to noindex pages on your own site can negatively impact SEO - maybe that's why Zazzle hasn't made the new collections pod clickable? (That's a wild guess on my part.)

ANYHOW, I think the main thing to remember here is that Zazzle isn't trying to hurt designers or shoot themselves in the foot - they're trying to optimize site performance. And having Google (and other bots) crawl pages that aren't "important" can definitely overwhelm the servers and slow down the site. Whether collections are "important" in terms of Google indexing or not is, I suppose, an open question.

So, if you're looking to try to get your products indexed through Google (and other search engines) I think one strategy is to link to individual products (as opposed to collections) directly from your website. How much that will help in terms of getting your products seen is also an open question. 

As always, this is just my interpretation. I'm by no means an expert on any of this stuff, and I certainly don't have an inside track on what Zazzle is doing or why, so I may have some of this wrong.

[Anne]

Thank you for looking into this and explaining, @Cat 
I always post a link to a collection under products on my own website. Apparently what I am doing is not good for SEO. From a marketing point it seems to me that offering something like "You may also like" might be a good idea, but if that is what brings down the number of visitors to the site ... ouch. So maybe I should replicate the collections on my website? (don't get me started ...)

[JimCarnicelli]

(deleted duplicate comment)