fix(app-root): allow search engines in production

[eatyourgreens]

• remove /public/robots.txt.
• add /app/robots.js which builds a dynamic robots.txt file, depending on APP_ENV. Allow access to / in production, but block the Zooniverse site otherwise.

Please request review from @zooniverse/frontend team or an individual member of that team.

Package

• app-root

Linked Issue and/or Talk Post

• closes Entire Zooniverse web site blocked from search engines #6331.

How to Review

Builds should build /robots.txt dynamically now.

yarn build && yarn start
# /robots.txt blocks all URLs for all user agents.
# User-Agent: *
# Disallow: /

APP_ENV=production yarn build && yarn start
# /robots.txt allows all URLs for all user agents.
# User-Agent: *
# Allow: /

Checklist

PR Creator - Please cater the checklist to fit the review needed for your code changes.
PR Reviewer - Use the checklist during your review. Each point should be checkmarked or discussed before PR approval.

General

• Tests are passing locally and on Github
• Documentation is up to date and changelog has been updated if appropriate
• You can yarn panic && yarn bootstrap or docker-compose up --build and FEM works as expected
• FEM works in all major desktop browsers: Firefox, Chrome, Edge, Safari (Use Browserstack account as needed)
• FEM works in a mobile browser

General UX

Example Staging Project: i-fancy-cats

• All pages of a FEM project load: Home Page, Classify Page, and About Pages
• Can submit a classification
• Can sign-in and sign-out
• The component is accessible
  • Can be used with a screen reader BBC guide to VoiceOver
  • Can be used from the keyboard WebAIM guide to keyboard testing
  • It is passing accessibility checks in the storybook

Bug Fix

• The PR creator has listed user actions to use when testing if bug is fixed
• The bug is fixed
• Unit tests are added or updated

[coveralls]

coverage: 78.54% (+0.03%) from 78.515%
when pulling e0afcd4 on eatyourgreens:dynamic-robots-txt
into 79c80fa on zooniverse:master.

[eatyourgreens]

Does Next.js pass the host name to route handlers? If so, this could be changed to check if the host is www.zooniverse.org, which might be easier to follow.

[goplayoutside3]

Copying this comment from the Issue:

Via discussion with Zach we decided to use the following strategy:

• fem apps all serve the restrictive version of /robots.txt
• host the permissive one in blob storage
• proxy explicit requests to www.zooniverse.org/robots.txt to the permissive version

I'm going to close this PR in favor of zooniverse/static#380 and #6340