Allow an archived site to be indexed by search engines, see #96.

ukwa · Jun 30, 2022 · e39facb · e39facb
1 parent 0705a3c
commit e39facb
Showing 1 changed file with 4 additions and 0 deletions.
diff --git a/access/website/config/static/robots.txt b/access/website/config/static/robots.txt
@@ -15,3 +15,7 @@ Allow: /wayback/
 User-agent: facebookexternalhit
 Allow: /wayback/
 
+# Allow search engines to index specific sites:
+User-agent: *
+# As requested in https://github.com/ukwa/ukwa-services/issues/96
+Allow: /wayback/archive/*/http://www.europeandialogue.org/