Personal Blogs

It's a long time since I wrote a blog post but I thought I should make an effort to talk about the global search enhancements that I worked on for the Moodle 3.4 release.

Introduction

At the Open University, which has a large Moodle installation with many custom plugins, we are now beginning to use the Moodle global search facility (replacing internal alternatives). This is happening very gradually and we intend for students not to notice! (If you're an OU student or staff member, the first place you'll see it is on subject websites, probably fairly early in 2018. But it will look almost the same as our current search.)

This is good news for other Moodle users because the current global search doesn't seem to have been all that widely used at scale yet so people might be a bit hesitant about adopting it. We've found and fixed (or helped fix) several problems with it - in terms of how easy it is to manage, and how well it works. These bugfixes are included in Moodle 3.4, and although I'm not suggesting it's perfect now, if you are another institution thinking about using global search, Moodle 3.4 might be a good time. It's a pretty neat system!

The changes

Here are some specifics about the core changes I implemented which are relevant to institutions implementing Moodle global search. 

As usual, thanks go to the many HQ developers, and others, who have assisted significantly with these changes! There are also other search fixes in 3.4 (for example Matt Porritt at Catalyst has improved indexing to include more files that weren't previously indexed), which I'm not listing.

MDL-59039 Global search: Allow partial indexing (in scheduled task)

If you have a significant amount of content in your Moodle system, the initial index is going to take a while. The scheduled task now automatically divides the work into chunks (by default 10 minutes at a time) so that it can run normally in the background - you don't need to run any CLI scripts on your server, there are no long-running tasks that might crash halfway through, and students don't need to see the search interface before you finish indexing. All you need to do to set up search now is:

1. Install and configure search as before (install Solr somewhere, configure Moodle to point to it, etc.) but do not enable global search yet.
2. On Manage global search page, enable the new option to index in the background.
3. If necessary, configure the time allowed per task run (on Manage global search page) and the frequency of the indexing task (on the Scheduled tasks page).
4. After the task has run, look at the Search areas page. You'll see Not yet fully indexed against areas which haven't finished indexing, and by looking at the Newest document indexed column you can get an idea of how long it will take to index. (I.e. if the newest document indexed is back in 2010, you may be waiting a while for it to catch up.)
5. Once indexing of all areas is complete (this could be days or weeks later), enable global search as before.
   

MDL-58957 Global search: Make it possible to search blocks

Any content in the HTML block is now searchable (provided the block is added to a course page). Also, developers can add support to search their own custom blocks - useful if there are cases where lots of content is provided in a custom block.

MDL-55356 Add restored content support to global search API

When you backup and restore a course, any content in the restored course probably has modified dates in the past. As a result, it was not previously indexed, because the search system only looked for content newer than its index. So if you needed to back up and restore courses (for example when creating a new copy of the course for each cohort) then the new ones wouldn't return any search results! We've added a rather complicated new mechanism so that everything in a restored site is indexed regardless of date. (This also means developers now have a way to re-index a specific context if necessary.)

MDL-59913 Global search: Allow search of non-enrolled courses

Unless you are a system administrator, the search system used to search only courses you are enrolled in. Now it also works on courses you can access for other reasons: because it has been configured to be open to everyone without login, or because your role has been overridden to have the moodle/course:view capability. This was really important for our systems as we have many courses that are open to all students. 

This is an option, which you need to turn on using the the Manage global search screen, because in some unusual cases it might make search results worse for your users.

MDL-60357 core_search: Future modified times cause serious problems

There are a few situations where content in Moodle might get a time in the future This content was previously indexed, but that would prevent any other content added between now and that future time from being indexed. We've changed it so that content dated in the future is not indexed until that date arrives.

MDL-60346 core_search: Solr connection ignores proxy settings

The Solr connection previously ignored proxy settings; if your servers are behind a firewall and need to use a proxy to access the Internet, and if the Solr server is hosted on somebody else's computer, outside your network, then it didn't work. This now works. 

However, you should be aware that during indexing, there will be a very large number of requests to the Solr server - it might still be better for your infrastructure if you can configure it to allow those requests through the firewall directly, and add your search server to the proxy bypass option.

What's next?

This project isn't over. We have plans to add more enhancements, some of which will make the interface better for students. With thanks to everyone who helped so far, I hope we can get some more changes in for Moodle 3.5.