User blog comment:BertH/Introducing WAM/@comment-452-20130417235543/@comment-452-20130418220002

One thing I've noticed while looking at some of the early dates is that wikis which were once in the top 5000, but aren't today, aren't shown when searching for today's date.

So if you want to know if your wiki was ever in the top 5000, you're currently out of luck unless you want to check every date.

It would be beneficial if the search also searched the past dates automatically.

I assume (because it's the simplest implementation) that you're just generating a list of 5000 wikis, and storing that information daily, and so have around 2.5 million entries in the WAM database already, but when you search, it's only searching the 5000 entries of the specified date. (Which makes sense. However simple or complicated your database is, that last part is surely true.)

For search purposes, it might be worth adding a boolean "lastseen" field to each entry, set as "true" for the entry when that wiki was last in the top 5000, and unset when there is a more recent entry. (So, each wiki which has ever been in the top 5000 has exactly one entry in the database with "lastseen=true".)

Then you can restrict the search to "lastseen=true & before X date". That way, someone looking for brieflypopularin2012wiki.wikia.com could find info about the last date they were ranked, without having to hunt through every day of the previous 16 months to find it.

(I'd assume that the top 5000 is actually pretty stable, or at least the top 2000 at any rate, so the number of entries with "lastseen=true" may not be terribly high. How many wikis have ever been in the top 5000? - I'd love real stats on this, if you have them.)

edit: typos, some rewording for clarity.