Add bulk tiddler upload

[elliott5]

See requirement in issue: #1

Use the standard TiddlyWiki "ExportAll", in one of the formats, then import it as a single file.

[dioclecio]

Dear Prof. Elliot,

I deployed my tiddlywiki to my personal Google Engine server. And later, I imported my 2300 tiddlers divided into 10 small files of 230 tiddlers.

Some of imported tiddlers presented the following message:

And of course, these specific tiddlers wasnt imported.

I think is something like timeout. I really dont know.

Yours,

Dino

[elliott5]

Well done Dino for persisting with using this software!

The sync messages are really not very helpful, to find out where the problem is.

Please forgive my ignorance, but what did you do in TW to import each of the 10 small files?

[elliott5]

The errors reported are 502 errors.
The code in this repository does not generate 502 errors (just 500 errors).
You can find out more about 502 errors here.
The mystery deepens...

[elliott5]

I just went to tiddlywiki.com did 'export all' to JSON, then imported that JSON file.
All 914 tiddlers seem to have been imported OK, and I saw no errors.
Should I try a bigger JSON import file?

[elliott5]

I wonder if the 502 errors are from Google rate-limiting you because you do not have billing enabled (which I do).

Also, you may be able to see the errors in the app engine console, let me know if you find anything of interest.

[dioclecio]

Dear Elliott,

thank you for your prompt action.

I will test it and return as soon as possible.

Thank you again.

Dino

[dioclecio]

Hi Mr. Elliot,

During this week, I made 3 tests. One test was made with the Couchdb adaptor, the second, with your Go server with your later considerations, and the third with this Heroku nodejs.

The first test took me hours to implement and to deploy. Both Tiddlywiki webpage and tiddlers were stored inside the Cloudant database. And for me, this is a bad solution.

It was quite fast to import locally, and to replicate the database into the external database webserver.

Your solution (Go server in Google Engine) took me some minutes to change the original Tiddlywiki with some plugins. And It took me some few minutes to deploy into the Google App Engine.

I exported and imported my 2300 tiddlers, and I took at least 20 minutes to save all tiddlers into the datastore. So, after activating your suggested billing function, I had to wait the saving icon from red to grey. But after this, it was quite fast to access the tiddlers.

The third solution was useless. Took some hours to implement and deploy into the Heroku servers. It doesn't separate the data of tiddlers from the nodejs files. And after some hours, all saved tiddlers is lost and the Tiddlywiki is restored to its original state.

Your solution works perfectly for a personal hosting and all connected devices, like Mobile phone, iPad, notebook and desktop. It takes 6,7 seconds to load all frontpage and menu of the main Tiddlywiki.

This is the screenshot of my page with 2300 tiddlers stored.

Unfortunately, none of these solutions someone can search inside of the tiddlers. The search works only on the titles. Do you know why?

Thank you for your great work.

Yours,

Dino

PS: Frankly, I don't fit into this USB/Dropbox hosting Tiddlywiki solution. I prefer to use some online solution. That's why I tried so many times to deploy one of few self hosting online solutions.

[elliott5]

Thank you Dino, I am very pleased that this particular TiddlyWeb server software works so well for you!

The 20 minute upload time is a rather a long. I think it is because the tiddlers are saved individually, rather than as a batch. In future, it would be possible to write a batch upload endpoint (url) which should speed things up - but I have no plans to do this work at present.

The reason that all the tiddlers' contents cannot be searched is that they are not downloaded from the server to the browser. This makes the start-up time much faster, as you have reported above.

Again, it would be possible to write a server-side search function, but I have no plans to do this work either.

All the best,

Elliott Stoneham

[elliott5]

I tried a hack...

If you did an 'export all tiddlers' operation, which would get all of the tiddlers into the browser, you should find that you could then do a search on the text of the tiddlers too.

... which did not work :( ... it just exports the empty tiddlers.

[elliott5]

Working from this thread I hacked together an 'OpenAllTiddlers' tiddler containing a single button:

<div class="tc-menu-list-item">
<$link to={{!!title}}>
<$view field="title"/>
</$link> <$button class="tc-btn-invisible2">→
<$list filter='[all[tiddlers]]'>

<$action-navigate $to=<<currentTiddler>> $scroll=no/>
</$list>
</$button>
</div>

When I pushed the button, nothing happened for a very long time ... then the tiddlers appeared.

It 'worked' in the sense of opening all the tiddlers and allowing me to search their contents. But with all the tiddlers open, it is very slow - so you need to 'close all tiddlers' to get acceptable performance.

Someone with more knowledge of TiddlyWiki than me would know how to do it better I'm sure!

[elliott5]

Dino,

I've created a very simple server search endpoint, which looks for an exact match.

https://the-name-of-your-project.appspot.com/search/the-term-you-want-to-find

I think this is what you are after, as it is very quick.

The next step would be to integrate calling this back-end functionality into TiddlyWiki.

I could also easily add pattern matching that allows any practical text search pattern to be described, rather than the current exact match.