Mail search over multiple folders

[rcubetrac]

Reported by tillkruess on 28 Jul 2008 15:36 UTC as Trac ticket #1485234

Roundcube needs as every good mail client does a search feature.

The user should be able to search in user-specified (sub)folders.

The request can take a few secounds, but if you have more than 1 folder, it is annoying to select each folder one by one and re-type your search string every time.

Keywords: search
Migrated-From: http://trac.roundcube.net/ticket/1485234

[rcubetrac]

Owner changed by @alecpl on 16 Jun 2009 18:27 UTC

=> none

[rcubetrac]

Milestone changed by @alecpl on 16 Jun 2009 18:27 UTC

0.3-stable => later

[rcubetrac]

Comment by @alecpl on 9 Oct 2009 10:31 UTC

http://code.google.com/p/roundcubeplugin-allfoldersearch/

[rcubetrac]

Comment by serpico7456 on 2 Jul 2010 14:28 UTC

This shouldn't really be a plugin, as it is part of the core functionality of most email clients, and frequently requested by our users. Additionally roundcubeplugin-allfoldersearch plugin has some pretty big issues:

1. If you drag a message from the search results to a folder it moves the wrong message . That's a pretty big bug.
2. It requires a bunch of hacks and requires you edit at least 5 files in the RC source code (which sort of defeats the purpose of a plugin), which will make upgrading painful since you need to track where you modified the code.

[rcubetrac]

Keywords changed by serpico7456 on 2 Jul 2010 14:28 UTC

search

[rcubetrac]

Summary changed by @alecpl on 3 Jun 2011 11:05 UTC

Extended Search

Extended Mail Search

[rcubetrac]

Comment by swix on 14 Jul 2012 16:01 UTC

It would be great if this could be added to a later release ! Some people migrating from Squirrelmail are missing this possibility, and the plugin has not changed since 2009...

[rcubetrac]

Comment by tirsales on 17 Dec 2012 16:17 UTC

This would be a really really important feature .. the plugin doesn't work (at least not for me and with the current revision of RoundCube), so what would be needed in order to implement it?
(Actually a 'search in all folders'-options would be sufficient for the start)

[rcubetrac]

Comment by moltedopablo on 5 Feb 2013 14:15 UTC

A lot of users are asking this feature that was in Squirrelmail. Unfortunately the plugin, although it's works, is really invasive to the core and doesn't allow to do any actions over the search results. Besides the search results doesn't show in what folder the mail is. We really think that this require to be added to the core functionality.

[rcubetrac]

Comment by Sucram on 8 Apr 2013 22:03 UTC

Is this seriously not part of the core functionality yet. 5 years later? The plugin is not practical at all as it requires core file editing.

[rcubetrac]

Milestone changed by @alecpl on 8 Nov 2013 12:09 UTC

later => 1.1-beta

[rcubetrac]

Comment by @thomascube on 5 Mar 2014 12:58 UTC

Replying to Sucram:

Is this seriously not part of the core functionality yet. 5 years later?

Yes, obviously people prefer to complain about the missing feature rather than start contributing it. This is free software folks, so act accordingly and start hacking if you miss something.

But for this particular features, we're actually working on it now. First results can be seen in the dev-multi-folder-search branch in our github repository. It'll soon be merged back into master but if somebody is willing to give it a test run, please report your findings here.

[rcubetrac]

Comment by kezarjg on 6 Mar 2014 17:10 UTC

Replying to thomasb:

...if somebody is willing to give it a test run, please report your findings here.

I installed the development branch on my server, and tested it out a bit. I'm new to Roundcube, so please forgive my ignorance on any of these comments.

1. The feature only appeared with the 'Larry' skin.
2. When performing a search 'This and subfolders' I received an error the first time. The error message was "Request timed out". When performing the search again, the results appeared. This was a mixed bag of working or not working.
3. The result list seemed to sort OK. Some messages previewed OK, but some did not. The error message was "Could not load message from server". This may be caused by nested folders. I couldn't seem to load messages from INBOX.Archive.2013...
4. The gear icon to display which columns to list does not contain the new column 'Folder'.
5. When I click on 'Subject' to sort by subject, I received and error. The error message was 'Server Error! (Internal Server Error)'. This happens for the 'From' and 'Date' fields as well. I was able to sort OK if I used the gear icon and select the sort field from there. I think this error only occurs after selecting the gear icon. From If username_domain has only one domain, don't allow users to specify the... #3 above, sorting worked before I clicked the gear icon.
6. I was unable to sort by the 'Folder' column.
7. I was unable to change the width of the 'Folder' column.
8. If I reordered the fields in the search results, when I returned to the inbox, the fields headings are correct, but the content is incorrect. Whatever location I moved 'Folder' to now displayed the folder name in that column. Clicking on the gear icon makes it correct again.
9. When switching folders from the search results view, the search box test stays the same, but the radio button defaults to 'Current Folder'.

That's all for now.

Thank you for working on this enhancement. It will be useful to me.

[rcubetrac]

Comment by smurf on 7 Mar 2014 20:05 UTC

Firstly let me say thankyou so much for roundcube!

You've done things really beautifully ( I'm talking about the code as much as the functionality ). People should stop to appreciate the huge effort that it takes to make an application as good as this.. and then stop to pinch yourself that someone has shared this wonderful thing freely with everybody - without regard to race, nationality, sex or species. Thomas,Alec and everybody else who has worked to polish this diamond for the benefit of the world - I salute you!

Secondly. I'm really interested in making multi-folder search + multi-folder-thread view ( i.e. show all messages related to thread , regardless of folder ) work.

PHP doesn't help by straight-jacketing the underlying IMAP interface. I would use python here in an ideal world , but needs must.

BTW: I have just cloned the dev-multi-folder-search.

I need to search for an e-mail address and then bring up all the correspondence ( in threaded views ) either To or From that address regardless of what folder it has been moved to.

I might be some time ... ;0)

Grateful for any tips!

[rcubetrac]

Comment by @alecpl on 23 Mar 2014 10:01 UTC

Another one:
10) check_recent.inc file need to be modified to work properly in multi-folder (no current folder) mode. $is_current doesn't make sense. Also e.g. the code in lines 119-135 does not work correctly.

[rcubetrac]

Comment by @alecpl on 23 Mar 2014 11:00 UTC

11. compose.inc wasn't modified, so when replying to a message in multi-folder search result you can observe empty message body (and invalid IMAP FETCH command). I suppose similar issues will be with forwarding. Another thing will be handling of messages from Drafts folder.
12. is this intentional that switching a folder does not reset a search? If so, that's fine, but there's one issue, page title is not updated.
13. in general we hide IMAP hierarchy separator from the user. Here, in Folder column the delimiter is not replaced with »
14. Using comma as multi-folder-UID separator might be a weak solution. I can imagine folders with comma in name. We should use some special character or use arrays in POST/GET.
15. I saw somewhere in the diff, that you use empty($folder). This is not correct check as folder can be named "0" or "0000", which in PHP is empty. So, strlen() is better.

[rcubetrac]

Comment by @thomascube on 7 Apr 2014 10:52 UTC

Replying to kezarjg:

Replying to thomasb:

...if somebody is willing to give it a test run, please report your findings here.

I installed the development branch on my server, and tested it out a bit. I'm new to Roundcube, so please forgive my ignorance on any of these comments.

1. The feature only appeared with the 'Larry' skin.

True. It'll be added to the classic skin once the feature is complete.

2. When performing a search 'This and subfolders' I received an error the first time. The error message was "Request timed out". When performing the search again, the results appeared. This was a mixed bag of working or not working.

The request timeout is set to 3 minutes. Maybe we'll have to extend that for multi-folder searches. But preferably the IMAP backend should be "improved" to perform better on searches.

3. The result list seemed to sort OK. Some messages previewed OK, but some did not. The error message was "Could not load message from server". This may be caused by nested folders. I couldn't seem to load messages from INBOX.Archive.2013...

Works for me.

4. The gear icon to display which columns to list does not contain the new column 'Folder'.

Right. The "Folder" column is only displayed in certain cases and thus isn't listed as a "regular" col. I think we should keep it this way.

5. When I click on 'Subject' to sort by subject, I received and error. The error message was 'Server Error! (Internal Server Error)'. This happens for the 'From' and 'Date' fields as well. I was able to sort OK if I used the gear icon and select the sort field from there. I think this error only occurs after selecting the gear icon. From If username_domain has only one domain, don't allow users to specify the... #3 above, sorting worked before I clicked the gear icon.

Confirmed and fixed.

6. I was unable to sort by the 'Folder' column.

Yes, that's not yet implemented. Problem is 4). What sort order to fall back when the column is hidden again?

7. I was unable to change the width of the 'Folder' column.

Columns generally cannot be resized in the Roundcube UI.

8. If I reordered the fields in the search results, when I returned to the inbox, the fields headings are correct, but the content is incorrect. Whatever location I moved 'Folder' to now displayed the folder name in that column. Clicking on the gear icon makes it correct again.

Confirmed and fixed.

9. When switching folders from the search results view, the search box test stays the same, but the radio button defaults to 'Current Folder'.

That's intended behavior. This way you can "refine" the search in a specific folder.

Thanks a lot for testing and reporting!

Please note that the feature has now been merged into git master. Thus for further testing, just use the master branch.

[rcubetrac]

Comment by @thomascube on 7 Apr 2014 14:27 UTC

Replying to alec:

Another one:
10) check_recent.inc file need to be modified to work properly in multi-folder (no current folder) mode. $is_current doesn't make sense.

Should be fixed in e8cb516.

Also e.g. the code in lines 119-135 does not work correctly.

We just skip these in search mode. I consider the listing of search results a temporary state where this sort of updates can easily be skipped.

[rcubetrac]

Comment by @thomascube on 7 Apr 2014 14:32 UTC

Replying to alec:

11. compose.inc wasn't modified, so when replying to a message in multi-folder search result you can observe empty message body (and invalid IMAP FETCH command). I suppose similar issues will be with forwarding. Another thing will be handling of messages from Drafts folder.

Should be fixed now.

12. is this intentional that switching a folder does not reset a search? If so, that's fine, but there's one issue, page title is not updated.

Yes, it is. I observed other clients and most do preserve the search query when switching folders. Also this can be used to refine the results with results from only one folder. Will look into the title issue, though.

13. in general we hide IMAP hierarchy separator from the user. Here, in Folder column the delimiter is not replaced with »

<space>&raquo;<space> eats up a bit too much space in the listing... but I get the point.

14. Using comma as multi-folder-UID separator might be a weak solution. I can imagine folders with comma in name. We should use some special character or use arrays in POST/GET.

Will investigate this.

15. I saw somewhere in the diff, that you use empty($folder). This is not correct check as folder can be named "0" or "0000", which in PHP is empty. So, strlen() is better.

Fixed.

[rcubetrac]

Comment by @alecpl on 7 Apr 2014 17:31 UTC

Here's a next set of problems from my recent tests:
16) special case when _uid=* is not handled correctly. E.g. do multi-folder search, choose "All" from "Select" menu to select all messages, now do Delete. (I suppose Move/Copy/Archive/Forward too). You'll see that does not work.
17) shouldn't we unselect selected folder (on folders list) when executing search for all folders?
18) when I move a message from multi-folder result to the same folder from which this message comes from, error is displayed. I think we could handle this nicer, no?
19) I see search result is re-sorted to display messages from many folders in correct order, right? however, I think that should be skipped if I choose no sorting (sorting by UID). Also, fetching all headers (default set of headers) might be too expensive, why not fetch only headers used for sorting?

[rcubetrac]

Comment by @alecpl on 8 Apr 2014 06:22 UTC

20. steps: enable threads, search in one folder, see that you can't disable threads now.

[rcubetrac]

Comment by @alecpl on 8 Apr 2014 06:57 UTC

20a) I noticed that 20. happens when I use only search filter (no search text) for searching.
21) I can't also change a folder after searching with filter only.

[rcubetrac]

Comment by @thomascube on 8 Apr 2014 08:58 UTC

Replying to alec:

Here's a next set of problems from my recent tests:
16) special case when _uid=* is not handled correctly. E.g. do multi-folder search, choose "All" from "Select" menu to select all messages, now do Delete. (I suppose Move/Copy/Archive/Forward too). You'll see that does not work.

Fixed.

17. shouldn't we unselect selected folder (on folders list) when executing search for all folders?

Works for me.

18. when I move a message from multi-folder result to the same folder from which this message comes from, error is displayed. I think we could handle this nicer, no?

Fixed.

19. I see search result is re-sorted to display messages from many folders in correct order, right? however, I think that should be skipped if I choose no sorting (sorting by UID).

Fixed.

Also, fetching all headers (default set of headers) might be too expensive, why not fetch only headers used for sorting?

This is done when we want to display message headers in the list and thus we need all headers. I don't think we'd win much when we first fetch one header per message for sorting and then fetch all headers (again) for the set to be displayed. I assume the average search resultset be < 100 messages but maybe I'm wrong about that.

[rcubetrac]

Comment by @thomascube on 8 Apr 2014 09:01 UTC

Replying to alec:

20a) I noticed that 20. happens when I use only search filter (no search text) for searching.

Works for me.

21. I can't also change a folder after searching with filter only.

Fixed.

[rcubetrac]

Comment by @alecpl on 8 Apr 2014 09:37 UTC

Great, I have some more:
22) search in current folder, switch folder, see that scope is set to "This and subfolders" even if folder has no subfolders. Looks like not a big problem but this disables list mode change.
23) scope change does not invoke search request if search box is empty, but search filter is not "All".
24) scope is not used when using only filter, ie. I can't list just unread messages in all folders.
25) search in all folders, move a message to any folder, result: the message is removed from the list which is wrong, because it still matches search criteria, but is in another folder. Another (more case is copying such messages.
26) handling of "*" is still broken. My search found 4 messages (in two folders). When I tried to Forward them (using select all), I got "Allowed memory size of 67108864 bytes exhausted". It looks that it tried to fetch more than 4 messages.
27) sorting order is ignored when sorting by UID.

[rcubetrac]

Comment by @thomascube on 8 Apr 2014 13:48 UTC

Replying to alec:

Great, I have some more:
22) search in current folder, switch folder, see that scope is set to "This and subfolders" even if folder has no subfolders. Looks like not a big problem but this disables list mode change.

True. But we don't have the information if a folder has childs available on the client.

23. scope change does not invoke search request if search box is empty, but search filter is not "All".

True. But the search options drop-down clearly is part of the search box. I don't think one would expect it to interact with the filter selection. Moving it elsewhere would probably be required.

24. scope is not used when using only filter, ie. I can't list just unread messages in all folders.

Mainly a UI issues. See 23) above.

25. search in all folders, move a message to any folder, result: the message is removed from the list which is wrong, because it still matches search criteria, but is in another folder. Another (more case is copying such messages.

You're too fast! Just fixed that in my last commit.

26. handling of "*" is still broken. My search found 4 messages (in two folders). When I tried to Forward them (using select all), I got "Allowed memory size of 67108864 bytes exhausted". It looks that it tried to fetch more than 4 messages.

I successfully tested with moving and flagging but not with forwarding.[[BR]]
Forwarding probably didn't work properly with '*' before because the search request ID was not sent to the server.

Now fixed.

27. sorting order is ignored when sorting by UID.

Fixed.

[rcubetrac]

Comment by @thomascube on 8 Apr 2014 13:51 UTC

Renaming this ticket to represent the feature of searching over multiple folders.

Let's track the other "advanced" searching aspects to other tickets: #1489488, #1488852

[rcubetrac]

Summary changed by @thomascube on 8 Apr 2014 13:51 UTC

Extended Mail Search

Mail search over multiple folders

[rcubetrac]

Comment by @thomascube on 8 Apr 2014 14:27 UTC

Replying to alec:

22. search in current folder, switch folder, see that scope is set to "This and subfolders" even if folder has no subfolders. Looks like not a big problem but this disables list mode change.

Fixed.

[rcubetrac]

Comment by @alecpl on 14 Apr 2014 18:04 UTC

I'm not sure it's related to multi-folder search changes, but these are regressions:
28) in classic skin thread tree icons aren't displayed
29) there's issue in messages list options popup - columns aren't selected.

[rcubetrac]

Comment by @alecpl on 14 Apr 2014 18:12 UTC

Ah, sorry, I forgot classic skin changes are to be done... Anyway maybe it's time to do this.

[rcubetrac]

Comment by @alecpl on 16 Apr 2014 07:22 UTC

30. when in multi-folder mode (no folder is selected) some options in folder actions menu should be disabled, i.e. Compact, Import messages,
31. "Download folder" (zipdownload) option name in multi-folder context doesn't look good. I think we should remove this option as "Download" option in "More actions" menu can be used to do the same. See also #1486069,
32. if multi-folder search returns no messages current folder is not unselected,
33. don't forget about 23-24.

[rcubetrac]

Comment by @thomascube on 18 Apr 2014 14:18 UTC

Replying to alec:

I'm not sure it's related to multi-folder search changes, but these are regressions:
29) there's issue in messages list options popup - columns aren't selected.

Unable to reproduce.

[rcubetrac]

Comment by @thomascube on 18 Apr 2014 14:39 UTC

Replying to alec:

30. when in multi-folder mode (no folder is selected) some options in folder actions menu should be disabled, i.e. Compact, Import messages,

Fixed.

32. if multi-folder search returns no messages current folder is not unselected,
33. don't forget about 23-24.

Both fixed in 26b5200.

[rcubetrac]

Comment by @thomascube on 18 Apr 2014 14:53 UTC

Updated classic skin in 9429087.

Closing this ticket as the basic functions are now available. Please open individual tickets for additional issues related to searching.

[rcubetrac]

Status changed by @thomascube on 18 Apr 2014 14:53 UTC

new => closed