User:Claus chr/Searching
Searching
The problems
There are (at least) three distinct problems with searching UserBase; they are (in order of increasing obscurity to regular readers and utility to writers and administrators):
- Using the search box gives unhelpful and possibly incomplete results
- Using What links here doesn't find Special:myLanguage links
- Using DPL yields incomplete results
The search box
The problem: The query searches both page names and page content (by default only the main namespace). This means you will often get hits for pages in many different languages, and the search results are presented in no particular order, so frequently the results are if not useless at least unreasonably difficult to use.
I'm not even sure we find all relevant hits. Last time I did a search for "Amarok/Man" I got two pages worth of results (about 40 pages in all) most of them translated pages. The search ought to have found all the Amarok Manual pages (and all their translations). In fact, only two English manual pages were found: Amarok/Manual/Various/FAQ and Amarok/Manual/Various/FAQ/en! (As an interesting side note, the search box query did find some pages containing a Special:myLanguage link!) (See following Note box.)
Note
Update: The problem described above was caused by my searching on an incomplete page name. It seems, that is this case only the page contents is searched. Searching Amaroc/Manual/ gives a lot of hits in many languages - including English!
Going to the advanced search page is no help at all; that just gives us the option to cast an even wider net, searching more namespaces. (Btw. translated pages all live in the main namespace. The namespace Translation holds individual translated units.)
What we should wish for in a general search:
- Only results in the readers language by default
- Some sort of Google-like prioritizing of results (?)
- The option to specify desired language(s) as well as namespaces on the advanced page.
- Search results should be comprehensive (fx, all Amarok manual pages should be found)
What Links Here
In its current form What Links Here is simply useless - it doesn't know about Translate links (Special:myLanguage), so it doesn't pick up most pages. Only lingering old style links are found (mostly on translated pages). We need a Translate-aware What Links Here replacement.
DPL
In theory DPL should be able to overcome the limitations of both of the other two options. Sadly that doesn't hold in the real world. For some queries it doesn't find all matching pages. Experiments shown on User:Claus chr/DPL/Test seems to indicate that some fixed capacity is exceeded. Sometimes performing a broad search on a pattern gives fewer results than searching for the same pattern in a narrower range of pages (fx searching only User namespace as opposed to searching both User and main namespace in one query). The results are very reproducible; always the same hits (and misses).
Note
Update
We have an answer from the DPL forum: There is a limit to the number of pages considered when searching with 'includematch' (as we must). By default that limit is 500 pages. We can limit the number of pages considered by fx specifying a namespace, which is why we sometimes get more results from a narrower search. However, Main namespace alone contains 5.5k pages (and growing! - about 1000 translatable pages + translations + catrgory pages, etc.). If we try to use 'titlematch' and 'notitlematch' to narrow the search we only make things worse, because in that case namespace filtering is turned off, which means that we get all the units in the translate system included (each unit is a separate page in Translation namespace); that would account for about 150k pages by my best guesstimate. Filtering those out would be possible if messy, and we would almost certainly filter out something that really should be included.
Our options seems be these:
- Increase the limit to how many pages are searched. There is a config setting for that.
- Can the server handle that? We would need to raise the number from 500 to at least 6000, or better to 10000 - we need to guard against banging our heads against the limit again; no warning is given when that happens, and the page count increases daily thanks to many new translators.
- What will that mean for search times? Increased search times may be acceptable for administrative work, but probably not for user searches.
- Try to split each search into many partial searches with clever use of 'titlematch' and 'notitlematch' to filter the number of pages considered.
- It is not clear what criteria based on simple wild card matching we can use for this.
- We'd need to keep an eye on the number of pages in each group. Not something I'm looking forward to maintain
- Find a different extension
- I have already searched wikimedia extensions many times without coming up with anything that looks remotely promising.
- Otoh, we might be able to make it ourselves. It seems, that fundamentally what we need is a fairly simple query on the underlying database. I think that in a way the main problem with DPL is that it is much more powerful that what we need. A simpler tool might fill our needs without putting as much strain on the system. The only problem is, that this simpler tool doesn't seem to exist. - yet
In my opinion the first option is by far the best (unless it brings our server to a crawl). It would be very simple to implement and require very little effort. If we can't use that I think we should seriously consider the third option. I don't feel, that the second option is worth considering as anything but a temporary stop-gap. Added: Even though the original problem with titlematch was all in my head the second option is still very unatractive. We'd have to split our pages (including the translated ones) into 15+ groups of ca. 400 pages each (to allow for addition of new pages) based on simple wildcard matching on page titles. And we'd have to monitor the actual size of these groups regularly, since we get no warning when there are too many pages in a group - the extra pages are silently ignored!