Forum:Is there an upper limit to how many bytes of code can be generated from template calls on the one page?

The symptom of this problem is that after some threshold is exceeded subsequent template calls do not expanded on the page. Instead the   wiki code is displayed. If the last template to expand calls other templates then the expansion may be only partial with the above symptom showing up part way through the expansion of that last template call.

After a little debug work with this demo template and this sandbox I can demonstrate that the number of ASCII characters in the template affects how many calls to that template are possible on the one page.
 * This is the case even if the ASCII characters are in commented-out blocks of code within an otherwise very small template.
 * It is also the case even if the characters are wrapped in noinclude../noinclude tags.


 * 1) Has anybody else noticed that templates cease to expand after too many have been used on a given page?
 * 2) What are the per page limits on template use? (I am guessing this is measured in kilobytes)
 * 3) How do I measure a page's current usage against this limit?

I may have figured out a a kind of "divide and conquer" workaround involving the use of template sub-pages to push any template usage documentation as well as frequently used modules of code. However, this workaround involves passing many more parameters to sub-pages than feels right so it looks like making the template very difficult to interpret and maintain.

Notwithstanding the above mentioned potential workaround, I'd first like to understand the fundamental limit that I am running into.


 * If you go snooping around the demo template link mentioned above then please review the edit history for my notes on which versions are problematic. The more recent ones are using the workaround so may not manifest the above symptom. Any version earlier than or including this one (Revision as of 00:56, 2008 October 31) will demonstrate the limitation.

najevi 01:01, 31 October 2008 (UTC)


 * There is a report of usage vs. limitations in the generated page source of each article page. For example, I got this from the source of http://rappelz.wikia.com/wiki/User:Najevi/sandbox5


 * Off-topic: As an aside, you can also find out whether or not the wiki you're on uses the new preprocessor as the report has different text based on which preprocessor is in use. - Adan Aileron (talk) 02:10, 31 October 2008 (UTC)


 * Thank you. I have now learned that the Rappelz Wiki is using the old pre-processor (this forum uses the new pre-processor). I was able to use the tool Special:ParserDiffTest on the sandbox demo page and found that the entire page was rendered successfully!


 * FYI: This article, Migration to the new preprocessor makes it very clear what that ParserDiffTest tool is to be used for. It also explains how to test which pre-processor your wiki is currently using.


 * najevi 06:09, 31 October 2008 (UTC)

Can the pre-expand size limit be increased from 2MB to say, 20MB ?
Thank you. I now see explicit comments like this one when I use, (Ctrl+U) View.Page Source, on the incomplete page:  < !-- WARNING: template omitted, pre-expand include size too large -- >  Sometimes you just need to be reminded where to look for clues! I expect I will remember that source in future.


 * So, 2MB seems to be the threshold. See also: wikipedia:Wikipedia:Template_limits although I have not yet found a wikia or wikimedia site-specific help/info page on this subject.

I can at least work on paring down my existing templates to squeeze to that space. I can easily remove code comments from my templates and relocate "noinclude"-ed Usage documentation to a sub-page. However, I don't think that is going to be enough to make a significant difference.

The beauty of conditional parser functions inside a template is that the post-expand size can be so small in a one-template-suits-many type of application. In the above case you set the post expand size is only 1% of the pre-expand size.


 * Is there any chance of having the pre-expand size increased?

It doesn't make sense for "comment"-ed blocks and "noinclude"-ed blocks to be counted in the pre-expand size. Am I alone in thinking this way? Mmmm, OK ... after reading this section even I can understand why it is this way.

najevi 05:01, 31 October 2008 (UTC)

Use onlyinclude and maybe the new parser
I've tested this here on central (see User:M.mendel/template, User:M.mendel/template1, User:M.mendel/template2), and using the onlyinclude tag, the "Preprocessor node count" and "Post-expand include size" are exactly equal even though the second template contains extra text. On rappelz, with the old preprocessor, the extra text and tags get counted for the pre-expand size.

With the old preprocessor, it makes lots of sense to shove the template documentation off to because that directly reduces the Pre-expand include size; with the new preprocessor, it still makes sense because then you can protect the template if need be and leave the docs editable; a doc edit will also not invalidate the pages that use the template (I hope).

I've always wanted to research that, thanks for giving me the poke I needed to finally do so. --◄mendel► 09:10, 31 October 2008 (UTC)


 * At some point in my debug process I did try both includeonly blocks and the noinclude blocks but that did not change the fact that, under the old pre-processor, blocks of text tagged as "noinclude" still get counted toward the pre-expand size. I just reran that test on my demo template and confirmed this.
 * I had never used onlyinclude blocks. (Even after reading about them today I still don't understand how such blocks of code are any different than general blocks of wikicode in a template.) I ran a test just now (also under the old pre-processor) and found that it made no difference.
 * Running that same test here on wikia central (where the new pre-processor is used) rendered all 400 template calls of the demo template.
 * The new pre-processor definitely makes a difference! I had previously experimented using the Special:ParserDiffTest tool at the rappelz wiki and found that I can successfully expand about 3 times as many templates as before.  Beyond that the tool failed to complete processing the wikicode and my Firefox browser stopped rendering and responding. So that suggests to me that (i) my templates are still exceeding some pre-expand size limit and also (ii) that the new pre-processor is  more intelligent about what wiki code gets counted toward that pre-expand size.
 * I did examine the page source for the output of the ParserDiffTest tool but there were none of the same helpful comments to be found like are found in a regular page's source view.


 * I suspect that after I (i) strip out comments from my templates and (ii) delete redundant arguments and (iii) figure out how to shift my usage documentation to a sub page ... then I will be well over half way toward my goal. I just need some time to work on that.


 * In the mean time I plan to ask that the Rappelz Wiki be migrated to the new pre-processor.
 * Are there any caveats or preparatory steps required before migration to the new pre-processor?


 * najevi 12:32, 31 October 2008 (UTC)


 * Yeah, onlyinclude makes no difference with the old preprocessor - that's what I found out. Onlyinclude makes all text outside of it behave as if it was inside noinclude tags, and I often find that to be more readable.
 * Migration:
 * Read Migration to the new preprocessor.
 * DiffTest your important templates.
 * We also put up a sitenotice advertising the changeover and asking for problem reports (there weren't any). --◄mendel► 23:07, 31 October 2008 (UTC)


 * With thanks to Emil we now have a URL method for previewing pages with the new parser preprocessor. It is actually the same method as described at the mediawiki article link above but it is now enabled for use at wikia as well. viz.
 * Append ?timtest=newpp to most 'clean' URLs (e.g. see below) or append &timtest=newpp to those URLs that involve a php? style query.
 * Using this method made my testing of various templates in use at all types of pages at the rappelz wiki much more time efficient. The Special:ParserDiffTest tool is still good for the HTML element level of subtle differences but I would probably only use that again after first noticing an error using the URL method.
 * More important to this topic however, using this method meant that for the first time I could see these complex pages render successfully. (http://rappelz.wikia.com/wiki/User:Najevi/More_or_less?timtest=newpp) So compared to the old-pp, the new-pp is certainly better at handling the expansion of templates that heavily use conditional tests for creating complex (expanding & contracting) tables of information. najevi 10:28, 4 November 2008 (UTC)