I would say that there's no efficient tool in MediaWiki for handling large amounts of data (other tha the built-in data like page content, categories, etc.) . Sure Lua can handle that better than basic Wikitext but it's inherently limited, too.

As for tips - arrange the data modules in a way that would group together items which are used on the same page (to minimize the number of modules loaded on a single page) and use mw.loadData to cache it for the entire page.

There's no built-in tool to collect the data for the data module you create, but you can probably use dpl queries for retriveing the data and then compile it to a Lua table.

If you are still interested in using Lua and still need a place to start, I have an export of a database module I was helping to develop for another wiki before it was merged with its Gamepedia counterpart. Since this was early in my Lua experience and it was not quite finished, we would probably need to make some additional modifications but it would at least be a place to start.

@HumansCanWinElves Thank you so much for the detailed reply and the advice! It didn't occur to me to try the idea of using dpl together with Lua to compile, but I'll definitely look more into that and try to come up with something!

@Andrewds1021 Only if you don't mind...! Thank you so much for the offer!

I still don't have much experience in Lua unfortunately, so I would definitely love to see some usage examples to study if that's alright!

Okay, I have made a modified version of what I previously helped work on. You should create the page Module:Database and paste the following as the initial version.

local getArgs = require("Dev:Arguments")["getArgs"]

local function getDatabase(key)
    local _, result = pcall(mw.loadData, "Module:Database/" .. key)
    return type(result) == "table" and result or {}
end

local function getItem(table, key)
    local item = table[key]
    return type(item) == "table" and item or {}
end

local function getValue(table, key)
    local value = table[key]
    if type(value) == "number" or type(value) == "string"
      or type(value) == "boolean" then
        return value
    end
    return nil
end

local function getList(table, key, value)
    local list = {}
    for k, v in pairs(table) do
        if type(v) == "table" and (value and value == tostring(v[key])
          or not value and v[key]) then
            table.insert(list, k)
        end
    end
    table.sort(list)
    return list
end

local p = {}

function p.get(frame)
    local args = getArgs(frame)
    if not args[3] then
        error("Database: Please specify a property to retrieve.", 0)
    end
    args[2] = args[2] or frame:getTitle()
    return getValue(getItem(getDatabase(args[1]), args[2]), args[3])
end

function p.call(frame)
    local args = getArgs(frame)
    if not args[2] then
        error("Database: Please specify a template to call.", 0)
    end
    args[3] = args[3] or frame:getTitle()
    return frame:expandTemplate{
        title = args[2],
        args = getItem(getDatabase(args[1]), args[3])
    }
end

function p.list(frame)
    local args = getArgs(frame)
    if not args[2] then
        error("Database: Please specify a property to check.", 0)
    end
    local list = getList(getDatabase(args[1]), args[2], args[3])
    if type(frame.getParent) == "function" then
        list = table.concat(list, (args["item-stop"] or "") ..
          (args["delimiter"] or "") .. (args["item-start"] or ""))
        if list ~= "" then
            list = (args["item-start"] or "") .. list .. (args["item-stop"] or "")
        end
        list = (args["list-start"] or "") .. list .. (args["list-stop"] or "")
    end
    return list
end

function p.callList(frame)
    local args = getArgs(frame)
    if not args[2] then
        error("Database: Please specify a template to call.", 0)
    end
    if not args[3] then
        error("Database: Please specify a property to check", 0)
    end
    local database = getDatabase(args[1])
    local list = getList(database, args[3], args[4])
    for i, v in ipairs(list) do
        list[i] = frame:expandTemplate{
            title = args[2],
            args = getItem(database, v)
        }
    end
    if type(frame.getParent) == "function" then
        list = table.concat(list, (args["item-stop"] or "") ..
          (args["delimiter"] or "") .. (args["item-start"] or ""))
        if list ~= "" then
            list = (args["item-start"] or "") .. list .. (args["item-stop"] or "")
        end
        list = (args["list-start"] or "") .. list .. (args["list-stop"] or "")
    end
    return list
end

return p

As it is right now, the module assumes each database is a subpage of the module. There are currently 4 function:

• get - Takes in a database name, an item name, and an attribute name. Returns the value of that attribute for the specified item. If item name is not provided, uses the name of the current page.

• call - Takes in a database name, a template name, and an item name. Calls the template passing in the attributes of the specified item as the template parameters. If item name is not provided, uses the name of the current page. Returns the result of the template call.

• list - Takes in a database name, an attribute name, an attribute value, and optional delimiters/prefixes/suffixes (specified by named arguments as described later). Searches through the specified database for all items where the specified attribute has the specified value and compiles a list of the item names. If no value is specified, it searches for the attribute to have a truthy value. If called from another Lua module, returns the list as an object. If called from a page using #invoke (which is usually the case), concatenates the list into a string with the optional delimiters, prefixes, and suffixes. The optional delimiters, prefixes, and suffixes are as follows. "list-start" prepended to the list. "list-stop" appended to the list. "item-start" inserted before each item. "item-stop" inserted after each item. "delimiter" inserted between items.

• callList - Takes in a database name, a template name, an attribute name, an attribute value, and optional delimiters/prefixes/suffixes. Similar to list but, instead of returning simply the item names, it returns the result of calling the specified template for each of the items in the list. Basically like combining the functions list and call together into one function.

FYI, here is a recent thread where HumansCanWinElves helps a different user use Dynamic Page List (DPL) to achieve something similar to callList. If you are unfamiliar with DPL, it is used to collect data from multiple pages into a single place. Basically the opposite of a database which is use to display data from a single place on multiple pages.