Forum:Limited edit rate for a bot

Hello, I've asked a question on the talk page of bots, I'm not sure it's gonna have any visibility at all there, so I'm adding a link here, I hope it's permitted. Hunter789 18:07, June 14, 2011 (UTC)


 * Tagging Staff needed. Wikia should be clear about what a "limited editing rate" means for acceptable bots. -- Fandyllic (talk &middot; contr) 14 Jun 2011 11:22 AM Pacific


 * My first inclination is to say they're talking about an actual edit request, i.e. the equivalent of clicking the save button, or  on  . I know from personal experience that the server is more than capable of dishing out dozens of tokens per second, and most other non-action requests can be completed quickly and efficiently. Edit requests are not, however, nor are moves, deletes, etc etc.

As you said on that talk page, your edit rate of 1/sec is a good baseline to start. If its running late night/offpeak times, you could probably let it run faster without too much problems. It also depends on what you are doing. Provided your not hammering us non-stop with 100s of requests a second for solid blocks of minutes, you probably wont even show up on Ops' logs/radar (though the chain reaction of internal mediawiki actions from updating categories/redlinks/etc might, but only if you are doing something very wrong, or very right ;P). I dont think you grasp the amount of edits/actions/pageviews does every second already, a few more from a bot on one wiki wont matter in the large scale of things for the brief period of time.

Most bots I run I generally dont put a throttle in, and let it just go at the pace that the server can handle the actions. Generally, most edit actions I've noticed take about 1sec from open connection, send, response, close. When I do that, I usually put some sort of loop logic in, so that every X things, it does a sleep for Y seconds (usually something 100/1 or 1000/2), just to give a little pause/break for my connection and the server. (I also tend to use that window to save some sort of continue data, so that if it does crash, I can resume from a known point, not at the top of the list.) Remember that doing an edit is fairly "cheap" to do, provided theres not dozens of templates that have to update, but moving/deleting pages may be a little more "expensive" and take longer to process, so when doing these, may want to look at adding delays/throttles, just to give a possible chance for things to update.

Something to look at, is look in the api docs, for the "maxlag" param stuff. You can actually set up your bot to detect when our servers are reporting they are loaded up, and to slow down for a while. This rarely happens on Wikia (and usually not that often on wikipedia anymore either), because of how our databases are now spread across multiple clusters, and have lots caches infront of the actual servers. You should also try to detect when your saves/actions are failing, and if you fail a whole bunch, or get a bunch of "cant resolve host" type errors, to stop, because it usually means the servers are having a fit, and are not available. I'd say about 80% of the code you are going to write to make a good bot will be error handling. There is only one "correct" condition, but there are dozens of ways it can fail.

Something else to remember is that bandwidth is cheap and large, but connections to server are finite. Once your flagged a bot, your account will be able to grab more data in a single request (500's turn to 5000's as an example), so try to do more per request when you are getting data.

--uberfuzzy, lover of bots and api's 08:51, June 15, 2011 (UTC)


 * All right, that gives me a better idea now! I'll add a link there to here, or should the text be copied over there because this one might get removed? Hunter789 18:45, June 15, 2011 (UTC)


 * This forum post should stay around. You'd want to link here anyway, since it really should be in a set of policies hosted here. -- Fandyllic (talk &middot; contr) 16 Jun 2011 2:37 PM Pacific