Crawler: IgnoreNoFollowTo

About this parameter

Whether the Crawler should follow links with the rel=”nofollow” tag and extract links from a page whose robots meta tag contains nofollow or none.

When true, the Crawler will consider links as if they weren’t marked by a nofollow directive.

This setting applies to both:

• links which would be ignored because the robots meta tag contains nofollow
• links whose rel attribute contains the nofollow directive.

The crawler always ignores links that don’t match the patterns of your configuration.

While ignoreNoFollowTo only accepts a boolean, it might eventually accept an array of patterns, similar the ones in exclusionPatterns or ignoreCanonicalTo. Setting ignoreNoFollowTo to an array of patterns would ignore the noFollow directives for only links matching those patterns. Please reach out to your Customer Success Manager or the support team if you need such fine-grained control.

Examples

1
2
3
{
  ignoreNoFollowTo: true,
}