Preserving the original file type for ingested tabular files is broken as of 4.6.2

[landreev]

Tabular ingest has been modified in 4.6.2 to better handle error conditions, and prevent leaving the files in "half-ingested" state. To achieve this we had to change the order in which we modify the content of the file and the database entries. One unfortunate side effect of that change ia that we now have this line:

dataFile.setContentType(FileUtil.MIME_TYPE_TAB);

before this one:

tabDataIngest.getDataTable().setOriginalFileFormat(dataFile.getContentType());

meaning, the type is already overwritten by the time we try to preserve it.
The fix is to swap the order of the 2 lines (trivial).
But also provide a script, or an API call that would properly reset these original format entries for all the files that were ingested in 4.6.2; fairly straightforward too.

[pdurbin]

@landreev good catch. While we're in that part of the code I wonder if we could also address #2734 or at least come up with a game plan.

[pdurbin]

Let's also at least do a little investigation of #2822

[landreev]

A direct effect of this for the user is that downloading the original results in getting it without any extension. They still get a correct file, and if they figure out what to open it with - Stata, SPSS, etc. - it can be used... but definitely a usability issue.

[landreev]

(the above has been confirmed)

[landreev]

Checked in a quick fix, in branch 3952-fix-original-type.
This is just a fix for the bug itself; with the plan to (potentially) release it in 4.7. The fix for the affected files already in the database will require a little bit more coding; and will be addressed in 4.7.1

[landreev]

@pdurbin #2822 is pretty straightforward. When a file is ingested, its type becomes "tabular". But the original type is preserved in DataTable.OriginalFileFormat.
It can be searched on in the database; but it appears that we are not indexing it; and that's what this user asked for.
Would be trivial to address; we somehow just missed the request and never addressed it.

[landreev]

pull request: #3954