[FIX] website_blog: prevent crawler on multiple blog tags page

Before this commit:
If a blog had multiple tags on the page, the crawler would click on every one
of them one after another until every tags combination got crawled.
With a lot of tags, this could result in million of URLs.

Now, we prevent URLs with more than one tag to be crawled.

Eg: With tags 'A', 'B', C':
/A
/A/B
/A/C
/A/B/C
/A/C/B

/B
/B/A
/B/C
/B/A/C
/B/C/A

/C
/C/A
/C/B
/C/A/B
/C/B/A
This commit is contained in:
Romain Derie
2018-04-05 16:55:58 +02:00
parent eaf54656fc
commit d39856dbe8
@@ -103,6 +103,7 @@
<t t-call="website_blog.index">
<t t-set="head">
<link t-att-href="'/blog/%s/feed' % (blog.id)" type="application/atom+xml" rel="alternate" title="Atom Feed"/>
<meta t-if="len(active_tag_ids) > 1" name="robots" content="noindex, nofollow"/>
</t>
<div class="container">
<t t-call="website.pager" >