[FIX] website_blog: prevent crawler on multiple blog tags page
Before this commit: If a blog had multiple tags on the page, the crawler would click on every one of them one after another until every tags combination got crawled. With a lot of tags, this could result in million of URLs. Now, we prevent URLs with more than one tag to be crawled. Eg: With tags 'A', 'B', C': /A /A/B /A/C /A/B/C /A/C/B /B /B/A /B/C /B/A/C /B/C/A /C /C/A /C/B /C/A/B /C/B/A
This commit is contained in:
@@ -103,6 +103,7 @@
|
||||
<t t-call="website_blog.index">
|
||||
<t t-set="head">
|
||||
<link t-att-href="'/blog/%s/feed' % (blog.id)" type="application/atom+xml" rel="alternate" title="Atom Feed"/>
|
||||
<meta t-if="len(active_tag_ids) > 1" name="robots" content="noindex, nofollow"/>
|
||||
</t>
|
||||
<div class="container">
|
||||
<t t-call="website.pager" >
|
||||
|
||||
Reference in New Issue
Block a user