wiki-migration-test

Repo for testing migration of wiki (and maybe other stuff)

Plan

• This is a test-bed for conversion of the trac wiki to a github wiki

Stage 1: Dump the pages from Trac wiki to files

• Main difficulty was getting access to the server, but now sorted
• Also trac-admin only seems to work when run from inside the webapps/trac/bin directory
• Commands to dump the wiki files and tar them up

  cd webapps/trac/bin
  ./trac-admin ../trac wiki dump ~/will-trac-wiki-dump
  cd ~
  tar czf will-trac-wiki-dump.tar.gz will-trac-wiki-dump
      

Stage 2: Import wiki pages into github repo

There are two options here:

1. Leave files as in current format
   • It appears that trac wiki format is similar to moinmoin and mediawiki, but not quite the same as either
   • Github offers the following formats
     • AsciiDoc
     • Creole
     • Markdown
       • This is the only one with full support from the online editor
       • This is default, and probably most commonly used on github
     • MediaWiki
       • Zero support from online editor
     • OrgMode
       • 90% support from online editor
       • Does have some features that Markdown lacks, such as tables
       • Potential clash between org-mode link syntax and wikilinks
     • Pod
     • RDoc
     • Textile
     • RestructuredText
   • I will try copy a file verbatim and adding a mediawiki extension
     • [X] Try with CompilingCloudyC17
       • This works partially
       • Good
         • Auto-generates TOC
         • Headers, bold font, and external links work fine
       • Bad
         • All the verbatim code, like {{{whatever}}}, is just dropped. Not surprising since this is not actually mediawiki syntax
2. Convert to markdown
   • There are various scripts that claim to do this
     • pandoc

       pandoc --from mediawiki --to markdown -o WikiStart.md WikiStart
                   

       • This was my first attempt, using WikiStart
         • Good
           • Headers (but see below) and links and bold seems OK
         • Bad
           • Plain text CamelCaseLinks are not honored
           • Explicit links like [wiki:NewC17] are not converted either
           • Strange target-ids appear after headers, like {#getting_started_with_cloudy}
     • https://gitlab.xiph.org/ePirat/trac-to-gitlab has a Trac2Down.py
     • Tracboat also has a trac2down.py
       • They both seem to be descended from the same code (Eric van der Vlist, 2013), but this one is more full-featured
       • There is a function trac2down.convert(TEXT, BASEPATH, **kwds)
         • TEXT is text of wiki or ticket (or other … ?)
         • BASEPATH is a required argument and seems to be related to the source code tree, which I will ignore for now (set to “.”)
       • Make a simple wrapper test-track2down.py to convert all the dumped wiki files:

         import sys
         import re
         from pathlib import Path
         sys.path.append("../../../Source/tracboat/src/tracboat")
         import trac2down
         
         # Paths for reading input and saving output
         try:
             input_path = Path(sys.argv[1])
             save_path = Path(sys.argv[2])
         except (IndexError, ValueError, TypeError):
             sys.exit(f"Usage: {sys.argv[0]} INPUT_PATH SAVE_PATH")
         
         # Get the names so that we can convert links to gollum format
         # "CamelCaseName" -> "[[CamelCaseName]]"
         wikinames = [p.name for p in input_path.glob("*")]
         
         
         def fixup_wikilinks(text, targets):
             """Add double [[...]] around wikilinks (given as `targets`)
         
             Most prosaic method possible - simply loop over the explicit list
             of targets.  Tries to be careful in avoiding code blocks, but
             will still be fooled by inline code
             """
             lines = []
             is_code_block = False
             for line in text.split('\n'):
                 # not blockquote?
                 if not line.startswith('    '):
                     if line.startswith("````"):
                         is_code_block = not is_code_block
                     if not is_code_block:
                         for target in targets:
                             line = line.replace(target, f"[[{target}]]")
                     # line = re.sub(r'\!(([A-Z][a-z0-9]+){2,})', r'[[\1]]', line)
                 lines.append(line)
             return "\n".join(lines)
         
         
         # Process all the files in the input folder
         for p in input_path.glob("*"):
             # Open file from the dump of the trac database and convert it
             with open(p) as f:
                 text = trac2down.convert(f.read(), ".")
                 text = fixup_wikilinks(text, wikinames)
             # Save the converted file with a ".md" extension to output folder
             # The 3rd, 4th, 5th arguments to this function are unused
             trac2down.save_file(text, p.name, None, None, None, save_path)
                         

       • Try it out

         python test-trac2down.py ../will-trac-wiki-dump ../wiki-migration-test.wiki
                         

     • Maybe modify one or write my own
       • Dealing with the CamelCaseLinks
         • We can get a list of all the current pages names, and then replace all mentions of these with a suitably formatted link: [[WikiLink]]
         • This is now what I am doing, but need to improve it a bit
         • I noticed that there are false positives that are inside URLs
         • Could be fixed by not allowing any letters either side - so need to go back to regexps

Other useful-looking stuff

• [ ] Should find a better home for this section

Documentation of gollum - the github wiki rendering engine

• https://github.com/gollum/gollum/wiki
• Can also be run standalone
• It has its own wiki tag syntax, [[STUFF]] which supports
  • internal wiki links
  • external links
  • images
  • include of other files but I think this is not supported in github wikis
• Also has macros - but is that really supported on github?
• Quote from https://help.github.com/en/articles/editing-wiki-content

  For security and performance reasons, some syntaxes are unsupported.
  
  Transclusion
  Definition lists
  Indentation
  Table of contents
      

Allowing math equations

• Github doesn’t support embedded javascript, so no mathjax
• Seem to be two main approaches
  1. Call out to some web service to render equation on-the-fly
     • Seems very clunky
  2. Replace equations with image or SVG
     • would be fine if efficiently automated
• Best bet (method 2) seems to be github-texify
  • Main repo https://github.com/agurodriguez/github-texify
  • Githup app install https://github.com/apps/texify
  • Adds a commit hook that transforms all files with names FOO.tex.md to FOO.md with all latex math replaced with inline SVG images (only supports dollar signs, it seems)
  • There are complaints it does not support wikis
    • See agurod42/github-texify#10
    • However, I think maybe the complainants aren’t recognizing that the wiki is itself a repo - so possibly it could still work

Continuous integration, etc with Github Actions

• Currently this is a public beta that you have to ask to join
• https://github.com/features/actions
• It looks very powerful - workflows can run on windows, linux, or macos hosts, triggered by pushes, releases, etc.
• Pricing is free for public repos, 3000 free minutes/month for private repos after beta period ends