Moz Isn’t Finding Your Links

In this guide we’ll walk through some troubleshooting tips for investigating why links to your site may not be appearing in the Moz Link Index or Link Explorer.

Quick Links

• Troubleshooting links not appearing in Moz's index
• The link is very new
• We haven’t yet found the page with the link to your site
• Ensure our crawler isn’t blocked by the robots.txt file
• Check that the link to your site is an html link
• Check the page’s canonicalization
• Ensure the page is not marked as noindex
• Check that our crawler isn’t being blocked by the server
• What’s next?

The Moz Link index data found in Link Explorer, and the Links section of your Moz Pro campaign, is gathered by crawling and indexing links using Moz’s crawler, Dotbot. It is possible that you may see a backlink pointing to your site that doesn’t appear to be indexed by Moz.

Lots of factors can impact our ability to find and report links pointing to your site. These include things like whether our crawler is able to access the page in order to crawl it, and index it, if the links are discoverable and how recently they were published.

We understand that it’s important for website owners and agencies to keep a close eye on links that are pointing to the site you’re working on. This guide will take you through seven things to check so you can identify why we haven’t crawled and indexed an expected backlink, and help us to queue the page to be crawled.

• Learn more about how we index the web
• Learn more about Dotbot
• Learn more about Link Explorer

It’s important to note that if a link has been recently added to a page, it can take some time for our crawler to find it and add it to the index. After being found, newly discovered links have the ability to be populated into our index in about 3 days however the discovery date is not necessarily indicative of when a link was created. Our link index is always growing so links that are older, newer, and everything in between may be discovered along the way.

The web is a vast and ever growing jungle of pages and links. The good news is that you can help us find links to your site.

We recommend adding the URL containing the link to your site to a Link Tracking List to help monitor the discovery of your backlinks. If the page is accessible and we’re able to parse the page for your link, it should be added to our index within about 3 days of discovery.

• Learn more about Link Tracking Lists

If you’re still not seeing your expected links in Link Explorer, it’s a good idea to check to make sure that our crawler isn’t being blocked by the robots.txt file for the site.

To check that we’re not being blocked from the page your link is on, head to the robots.txt file (typically it would be the root domain of the site followed by /robots.txt. For example: https://moz.proxybugs.com/robots.txt). If there is a disallow directive for ‘user-agent: *’ or ‘user-agent: dotbot’ this means our crawler is being blocked and won’t be able to access the site to crawl and index.

Disallow directives can also be specific to a type of URL parameter. If the link you’re expecting to see is on a page blocked by this type of disallow directive, we won’t be able to access and crawl that page.

• Learn more about robots.txt files
• Learn more about Dotbot
  

Our crawlers are designed to find and follow links coded as html links within the source code. For example:

<a href="https://moz.proxybugs.com/blog">Blog

If the link to your site is coded in javascript or is only able to be viewed by human visitors in a browser, our crawler won’t be able to find and parse the link. Content that is rendered in a browser may not be able to be seen by our crawler.

You can check how a link is coded by looking at the source code of the page. To access the source code, right click in the browser window and click View Source. An HTML link will be coded as href= as seen below.

If the link you’re expecting to see is not coded as an html link, it will not be indexed and added to your backlink profile in Moz.

In order for our index crawler to crawl a page, it must be able to find a canonical URL that is self-referential. This means that if your link is on a page that is canonicalized to a different page, our crawler will not crawl for your link. These directives are also case-sensitive and protocol-sensitive which means that the canonicalized URL must be a character for character match of the page we’re trying to crawl.

You can check the canonical URL for a page in the source code of the page. To access the source code, right click in the browser window and click View Source.

If the page your link is found on is not canonicalized to itself, character for character, our crawler will not crawl the page and your link will not be added to our index.

• Learn more about canonicalization

If the page containing a backlink to your site is marked with a meta robots tag of noindex, nofollow or noindex, follow our crawler will not be able to index the links on that page. This means any links on that page will not be added to the index.

You can check the page’s meta index directives in the source code. To access the source code, right click in the browser window and click View Source.

• Learn more about meta index directives

If you’ve gone through and checked all the above information and you’re still not seeing your link in Link Explorer, this means our crawler is being blocked from accessing the page with your link.

If, at the time we attempt to crawl the page your link is on, the server returned an error or blocked us we won’t be able to index your link.

Status codes which can inhibit our ability to crawl are 4xx and 5xx status codes. These come directly from the server of the site and tell our crawler it’s unable to access that page.

• Learn more about status codes
• Learn more about DotBot
  

If you find that any of these issues apply to your link or the page your link is on, they would need to be updated and changed on the site that’s linking to you.

Unfortunately, Moz does not have the ability to make any changes on other sites and would not be able to resolve this issue on your behalf. This includes our crawler being blocked by the server (as mentioned in step seven). The response we are seeing in this case comes directly from the server of the site and would need to be explored with the site admin or hosting provider to resolve.