[IMP] website_forum: add lastmod in sitemap for forum post URLs

When a crawler (eg Googlebot) come to visit your website, it grants you
a limited amount of time and ressources, it's called "Crawler budget".
If you have millions of URLs, it won't go through each one of them in a
single go.

The best you can help those crawler, the better. The sitemap `lastmod`
attribute, despite not being fully respected and trusted by crawlers, is
one of the way you can still try to help them.

For website.pages, it's already done. But for controllers, it's not an
easy thing to do as we have no way to automatically figure what are the
relevant records/fields to look at to know the last update date.
For instance, on the event pages, some pages content are mostly stored
inside an `ir.ui.view`, but the title, hours etc are part of the event
itself.
We can't just say "we take the last write_date of the record", it's
wrong in 2 ways:
- The first one I just explained where we wouldn't be able to easily get
  all the elements part of the page rendering and would miss a possible
  element write_date, leaving an outdated date in `lastmod`.
- Then, there is another issue (which is more problematic in stable):
  the `write_date` is often updated for non website related purposes.
  For instance, on /partners/<partner>, we wouldn't be able to use the
  write date on odoo.com as the partners shown there (having a grade)
  are update every weeks in average, because of many fields, for
  instance: commission_plan_id, partner_weight, grade_id, ...

Still, there is a quick win possible in stable about forum posts which
are not impacted by the 2 issues explained above:
- The `write_date` doesn't seem to be updated too frequently for
  (sitemap) irrelevant reasons. We can ensure to show a date which is
  only modified when the forum.post page really gets a modification.
- All the forum.post information displayed on the page are stored inside
  the forum.post itself.

This commit is thus adding the `lastmod` on forum.post URLs in the
sitemap in hope of not making Google waste time on (very) old posts.

Note: we don't use the `last_activity_date` as it wouldn't be updated in
      case of the post `content` or `name` being modified for instance,
      which would be wrong.
      Some metrics: On odoo.com, out of the 83499 forum posts having a
      `last_activity_date < 2023-10-01`, only 2335 have a `write_date`
      after `2023-10-01`. It means that those two fields are actually
      giving almost the same results.
      Using `write_date` is thus the best choice: not impacted by the
      issues mentioned above and cover all relevant record change, as
      opposed to `last_activity_date`.

Note: the `lastmod` has to be trustworthy and correct, if you set wrong
      or outdated info inside it, Google won't trust you/it anymore.

closes odoo/odoo#155197

Signed-off-by: Jérémy Kersten <jke@odoo.com>

addons/website_forum/controllers/website_forum.py

@@ -283,8 +283,19 @@ class WebsiteForum(WebsiteProfile):
         # Compatibility pre-v14
         return request.redirect(_build_url_w_params("/forum/%s/%s" % (slug(forum), slug(question)), request.params), code=301)
 
-    @http.route(['''/forum/<model("forum.forum"):forum>/<model("forum.post", "[('forum_id','=',forum.id),('parent_id','=',False),('can_view', '=', True)]"):question>'''],
-                type='http', auth="public", website=True, sitemap=True)
+    def sitemap_forum_post(env, rule, qs):
+        ForumPost = env['forum.post']
+        dom = expression.AND([
+            env['website'].get_current_website().website_domain(),
+            [('parent_id', '=', False), ('can_view', '=', True)],
+        ])
+        for forum_post in ForumPost.search(dom):
+            loc = '/forum/%s/%s' % (slug(forum_post.forum_id), slug(forum_post))
+            if not qs or qs.lower() in loc:
+                yield {'loc': loc, 'lastmod': forum_post.write_date.date()}
+
+    @http.route(['''/forum/<model("forum.forum"):forum>/<model("forum.post"):question>'''],
+                type='http', auth="public", website=True, sitemap=sitemap_forum_post)
     def question(self, forum, question, **post):
         if not forum.active:
             return request.render("website_forum.header", {'forum': forum})

addons/website_forum/tests/__init__.py

@@ -6,4 +6,5 @@ from . import test_forum_internals
 from . import test_forum_karma_access
 from . import test_forum_tag
 from . import test_forum_tours
+from . import test_sitemap
 from . import test_web_editor

addons/website_forum/tests/test_sitemap.py

@@ -0,0 +1,31 @@
+# Part of Odoo. See LICENSE file for full copyright and licensing details.
+
+from freezegun import freeze_time
+from unittest.mock import patch
+
+from odoo.addons.website_forum.tests.common import TestForumCommon
+from odoo.tests import tagged
+
+
+@tagged('post_install', '-at_install')
+class TestWebsiteControllers(TestForumCommon):
+
+    def test_01_forum_sitemap(self):
+        website = self.env['website'].browse(1)
+
+        # Simulate post from 2023-05-31
+        datetime = '2023-05-31'
+        with freeze_time(datetime), patch.object(self.env.cr, 'now', lambda: datetime):
+            self.post.name = "RenameIt"  # update write_date
+            self.post._update_last_activity()  # update last_activity_date
+
+        locs = website._enumerate_pages(query_string='/forum/%s' % self.forum.name)
+        self.assertEqual(list(locs)[1]['lastmod'].strftime("%Y-%m-%d"), datetime)
+
+        # Edit post content the 2024-01-01
+        datetime = '2024-01-01'
+        with freeze_time(datetime), patch.object(self.env.cr, 'now', lambda: datetime):
+            self.post.content = "I am a bird"  # update write_date
+
+        locs = website._enumerate_pages(query_string='/forum/%s' % self.forum.name)
+        self.assertEqual(list(locs)[1]['lastmod'].strftime("%Y-%m-%d"), datetime)