User blog:The Fizzbuzzler/Logging searches

Ever wondered what your readers are searching for? Been frustrated that you can't tailor your wiki's content to what your consumers want? Now you can, using sitemeter to log the results and a Perl script to put them onto a text file.

With this information, you know exactly what needs work. New pages can be made based on the readers' preferences, redirects can be made where necessary, and new disambiguations can be made.

Using the following method, you can log the searches of your entire userbase with very little effort, and very low technical prowess.

The site that will actually be logging your results is sitemeter. To begin the process, click here and enter your wiki's main page in the URL, and its name (they don't really matter that much for this project.) For the codename, pick something that will identify your wiki; for example, on w:c:runescape, I chose rswiki. This is your login username. Check the box, and click enter. Fill out the next page, and you'll get a confirmation e-mail including your password. Authenticate yourself, and you will have a sitemeter account! Log onto your sitemeter account, and go to the manager tab, and click HTML code on the sidebar (here. Copy the top code (JavaScript HTML), removing the commented out part if you please. Create a new page in the MediaWiki namespace (sysop permissions required); the name does not matter too much. On the RuneScape Wiki, I used w:c:runescape:MediaWiki:Search sitemeter. Save it onto the page.
 * Step 1: Sign up for sitemeter
 * Step 2: Install the script on your wiki

To put this code on the search page, you need to transclude it onto the search page using a different MediaWiki page. Go to MediaWiki:Showingresultsheader, and, after the default information, put the following: Search sitemeter This will finally put the sitemeter on your search page, and begin logging results. Go to the statistics page on your sitemeter account; the number of visitors and page views should start to go up. Then go to recent visitors by entry pages, and you should see the URLs under Special:Search that show search keywords. Sitemeter is really great, except it only shows the 100 latest results and it can't be exported. That's where the Perl script comes in. (Disclosure: I did not create this script.) Go to here and download the script, replacing instances of with your sitemeter username, with your sitemeter password, and with the full directory location of where you want it to output the results (this is preferably in the same directory as searchlog.pl).
 * Step 3: Pulling keywords from sitemeter

Go to command prompt, navigate to the folder where searchlog.pl is, and type  (replacing   with wherever you're outputting the logs). If all works well, you should then see a .txt file in your folder with some recent searches, including the IP or hostname (note that we do not use that extra information for anything; it is only used to make sure we do not duplicate the results when we pull results from sitemeter). If you use Windows, go to Task Scheduler and create a new task: have it run whether or not you are logged in, and under the Actions tab, select "Start a program" from the dropdown. Under the program/script, browse to the location of searchlog.pl. Under arguments, put .txt file of the location that you are logging ( or something). Under "triggers", have it run indefinitely, and have it repeat the task every five minutes (it can run less often, depending on the size of your wiki and number of searches.) This will let it log to the .txt file whenever you have an internet connection on the computer. With the results, go to this page and enter all of the logs into the textbot. Run it, and it will order the keywords based on how many times they have been used. This information can be used to see what your wiki is missing, whether it's pages or redirects or something else.
 * Step 4: Automating the logging
 * Step 5: Doing something useful with this information

I think this information is really valuable and a must-have for any wiki who wants to improve the experience for its readers. On the RuneScape Wiki, I spent about three hours adding new redirects, saving an estimated 900,000 unnecessary searches by readers over the course of a year. I hope you all can use this new information to grow your communities and make them more accessible.