util.prosodyctl.check: Correct modern replacement for 'disallow_s2s' The code would have suggested adding to modules_enabled instead of modules_disabled

mod_s2s: Fix reporting of DANE mismatch Thought it was a case mismatch at first, fixed that, but it changed nothing because the error was in the leaf part of the errors, not the chain part.

core.portmanager: Join strings broken into multiple lines Improves readability. Reduces line count. What's not to like? The code style and luacheck rules allows longer lines, and these strings aren't long enough to need breaking into multiple lines like this.

core.portmanager: Hint at HTTP servers for conflicts over port 443 Since 443 is just as much a web port as port 80 these days, if not more. What's with port 81 here?

net.websocket.frames: Remove completed TODO The XOR is done in C since 4e5a2af9dd19

mod_http: Generate URL from configuration in prosodyctl This removes the need to configure e.g. http_external_url or similar settings in order to get correct URLs out of prosodyctl, as the API depends on portmanager to know the actual ports that are used.

Merge 0.12->trunk

util.array: Fix new() library function Backport of ffe4adbd2af9 since new was added in the 0.12 branch

mod_tokenauth: Fix revoking a single token without revoking whole grant This appears to have been a copy-paste of the grant revocation function, or maybe the other way around. Either way, it deleted the whole grant instead of the individual token as might be expected.

util.array: Fix new() library function

tests: Add hack to test only a single storage driver Fixes that LuaDBI being unavailable makes these produce nothing but endless stack overflows in luarocks.

tests: Update storagemanager tests for prosody.* namespace change Part of an attempt to make these tests work again. Previously they would just explode in a million luarocks stack overflows

mod_storage_sql: Spell out missing dependencies Using util.dependencies appeared to cause problems with running tests in Busted, so this also removes that and uses pcall directly.

mod_storage_sql: Pass variables as arguments instead of upvalues Probably a workaround for the lack of argument passing when using xpcall in Lua 5.1, no longer relevant.

util.sql: Remove unused String() and Integer() functions According to MattJ, leftovers from an earlier vision for util.sql

util.sqlite3: Clean up unused variables Many leftovers from the earlier version of util.sql this was based on and cleanup applied there since then.

util.datamanager: Always reset index after list shift Shifting the index does not work reliably yet, better to rebuild it from scratch. Since there is minimal parsing involved in that, it should be more efficient anyway.

core.moduleapi: Parse period min/max arguments Allows specifying them the same way as the default and in the config, for consistency

util.datamanager: Add way to close indexed list store

util.datamanager: Close file handle when done using it It gets closed eventually but at high load they could potentially lead to reaching FD limits faster.

util.datamanager: Disable blockwise removal In desperate need of tests

mod_muc: Use enum config API for 'restrict_room_creation' This communicates the accepted values in case the config diverges from them. Note that older documentation used an "admin" value behaving like an alias to true, but this is no longer handled. Should it?

plugins: Use get_option_array for some list shaped options Passing something from module:get_option() to ipairs() suggests that the option is a list of some sort.

util.datamanager: Disable block alignment Until we have more test coverage. Somehow the index becomes incorrect after inserting padding, unclear why.

plugins: Handle how get_option_period returns "never"

plugins: Use boolean config method in some places Because it makes sense and improves feedback via logging

mod_storage_internal: Use integer option method for cache size Missed this one in previous sweep

mod_muc_mam: Use period option method

mod_muc: Use enum option method for 'muc_room_default_presence_broadcast'

mod_storage_xep0227: Use enum option method

mod_auth_ldap: Use enum option method

mod_storage_sql: Use integer config option for cache size Missed this one, was probably only looking for get_option_number

mod_storage_sql: Use config enum for 'sqlite_tune'

mod_storage_sql: Allow higher precision timestamps in SQLite3 Since it doesn't actually do strict typing :)

Merge 0.12->trunk

util.prosodyctl.check: Hint about the 'external_addresses' config option

Merge 0.12->trunk

util.prosodyctl.check: Validate format of module list options Should detect things like misplaced settings inside modules_enabled

util.prosodyctl.check: Get some config options via minimal moduleapi #896 The module API has certain coercion features that are useful. Fixes traceback reported in #1812 and other duplicates

core.moduleapi: Fix min/maxinteger fallback for Lua 5.2 Maybe these should live in util.mathcompat?

mod_http: Fix passing minimum limits in wrong argument position

plugins: Use integer config API with interval specification where sensible Many of these fall into a few categories: - util.cache size, must be >= 1 - byte or item counts that logically can't be negative - port numbers that should be in 1..0xffff

core.moduleapi: Add min/max range support to :get_option_period To match :get_option_number etc, specifying the allowed interval. Default is essentially (0, inf].

moduleapi: Add :get_option_integer() Many options in Prosody that are treated as numbers don't make sense as floats, e.g. sizes and limits measured in bytes. Simplified implementation based on an earlier attempt dating back to 2020

util.human.io: Fix stray 'stty' error by only querying width of real ttys This adds a dependency on a binary and *nix-specific module but then stty is probably *nix-specific anyway so maybe that's fine.

plugins: Switch to :get_option_period() for time range options Improves readability ("1 day" vs 86400) and centralizes validation.

core.moduleapi: Accept boolean false to disable period setting

core.moduleapi: Log error for unexpected types (booleans?) set as periods

core.moduleapi: Turn negative periods or "never" into infinity As a way to signal that the periodic thing should be disabled, matching existing mod_mam usage

core.moduleapi: Improve handling of different types in :get_option_period Pass positive numbers trough unharmed, parse strings as periods, discard anything else.

core.moduleapi: Add :get_option_period for parsing time intervals E.g. for use in mod_mam and others that take an amount of time before some (usually cleanup) action is taken.

core.moduleapi: Allow specifying an acceptable range for number options

plugins: Use get_option_enum where appropriate

moduleapi: Add enum config option method For when a setting has a few fixed values it can take

tools: Fix file ending of mod2spec.sh (thanks buildbot) Accidentally .lua ?

util.human.io: Fix pattern in parse_duration() to cover all used letters Notably 'h' was missing. Awkwardly, 'hour' would result in 'ho' which was missing from table.

util.human.io: Add tests for parse_duration() (some failing)

util.human.io: Include relevant arguments in test messages This way the relevant arguments are shown in case a test case fails

util.human.io: Use tail call in test to get correct line numbers This is probably not guaranteed to work and might vary with Lua version, but it's good enough for me to get accurate line numbers out of Busted that don't all point to the test() function.

tools: Add mod2spec.sh, turns util.example into spec/util_example_spec.lua Useful for opening a module and its tests at the same, can be awkward to auto-complete sometimes. sensible-editor util/example.lua $(./tools/mod2spec.sh util.example)

editorconfig: Include the command used to normalize *.xml The double asterisk seems unnecessary.

editorconfig: Specify max line length to match luacheck settings

editorconfig: Explicitly specify preferred charset (UTF-8) Really should be the default everywhere by now, but doesn't hurt to be extra explicit

editorconfig: Add link to format description

mod_storage_sql: Remove completed TODO (testing UPSERT on PostgreSQL)

CHANGES: Move line about LuaSQLite3 to Storage section

CHANGES: Mention performance improvements for internal archives Specifically the index and more efficient delete. These are however still in need of testing.

mod_storage_internal: Implement efficient deletion of oldest archive items Using the new shift function in datamanager, either the oldest items are removed or all the later items are moved into a new file that replaces the old. Hidden behind a feature flag for now.

util.datamanager: Pad list writes to avoid crossing block boundaries By padding items so that they do not cross block boundaries, it becomes eaiser to delete whole blocks with fallocate() without cutting items in half, improving efficiency of such operations. Since list stores are used for message archives, where the most common deletion operation would be of the oldest entires, at the top of the file. With this, all blocks that contain items to be removed could be deleted without needing to read, delete and write out the whole file.

util.datamanager: Efficiently remove whole blocks to shift lists Using the new pposix.remove_blocks() it should be very performant to delete whole sections of a file, given a supporting file system.

util.pposix: Add remove_blocks() for deleting parts of files Allows implementing e.g. a FIFO Will probably only work on some Linux file systems like ext4.

util.datamanager: Add way to efficiently remove first items in a list Copying data without parsing it should be more performant than parsing it serializing back.

util.datamanager: Fix indexing first item if not at the very start If the first item does not start at position 0 then the index function produces a phantom first entry covering position zero until where the real first item starts. When using the index, this would make it either appear as the first item was missing or cause an off-by-one issue with remaining items.

util.datamanager: Reduce log level of left over debug messages to debug (thanks Trung) These were mostly 'warn' to make them stand out from the debug noise

util.datamanager: Fix missing separator in log line

Merge 0.12->trunk

core.certmanager: Update Mozilla TLS config to version 5.7 Ref https://github.com/mozilla/server-side-tls/issues/285

mod_pubsub: Send correct jid attribute in disco#items Fixes use in PEP where the JID does not equal the bare domain.

mod_http_file_share: Put 'expires' back, thought it was unused Removed in 536055476912 because it was not used anywhere else in the file, but per the documentation it is meant to inform external upload services of the expiry time of the upload itself.

util.cache: Pass cache itself to eviction callback Simplifies access to the cache without moving code around a lot given the currently common pattern of local some_cache = cache.new(size, function(k,v) end)

util.cache: Keep eviction candidate if callback resized to make room Previously either the old or the new values would be rejected, even if the cache was resized to allow more items.

util.serialization: Teach Teal about the new "pretty" preset

util.jsonpointer: Change function prototype to allow anything But anything that's not a table can't be resolved into, which could happen in the middle, so eh.

util.jsonpointer: Silence Teal warning It seems to think 'table' never has array items, but we don't know that.

core, plugins: Split prosody:user role into prosody:{guest,registered,member} This gives us more granular control over different types of user account. Accounts registered by IBR get assigned prosody:registered by default, while accounts provisioned by an admin (e.g. via prosodyctl shell) will receive prosody:member by default.

usermanager: Add create_user_with_role() method to atomically set initial role

mod_http_file_share: Set slot token TTL so util.jwt validates expiry Overrides the util.jwt default of 1h with the intended TTL of 10 minutes. Because util.jwt now has its own expiry checks, so the 'expiry' field is no longer used and can thus be removed.

prosodyctl: Add experimental way to reload specific modules directly Mostly thinking out loud about how various actions may use the shell This enables the following sequence of commands: prosodyctl install mod_example prosodyctl reload mod_example which is simpler than prosodyctl shell module reload example

net.server: Handle loading from outside Prosody (e.g. Verse) server_select only depending on LuaSocket generally makes it more portable, so fall back to that if util.poll can't be found.

renamening: Fix newly added imports to use the new namespace

util.jsonschema: Remove wrapper function This was to silence some Teal warning that seems to have gone away.

util.jsonschema: Silence Teal warnings about utf8 library Teal worries that we redefine the global. Also that the fallback was missing type information.

util.jsonschema: Silence Teal warnings about counting items in tables Teal thinks that these are key-value maps which are always of length zero, but that is not the case.

mod_invites: Refactor argument handling using util.argparse This makes it so that --admin and --role are no longer mutually exclusive, they the former is simply treated as another --role. This was likely a leftover from when only a single role was possible. It does however become unclear which should be the primary, since the order is not preserved by argparse. Bonus: Loading of modules is avoided with only the --help is shown.

util.argparse: Add support for repeatable parameters These are gathered into arrays

mod_storage_sql: Fix column name in index check for PostgreSQL Forgot to change the column name in 9a7523ea45cb

mod_storage_sql: Adjust indentation to align with surrounding code

mod_storage_sql: Only remove old index if it exists Avoids an error if the upgrade is performed twice..

mod_storage_sql: Be more specific when checking for old index in SQLite3 Prevents false positives in the odd case where something other than an index with this name might exist.

mod_storage_sql: Improve check for old table index on PostgreSQL The "pg_indexes" view is much simpler to inspect than "pg_class"

mod_storage_sql: Enable UPSERT with PostgreSQL Tested. Works.

doap: Fix typo in attribute name

util.pposix: Use Lua enum API for resource limit name argument Because diffstat.

mod_storage_sql: Add some TODO comments for future UPSERT work

mod_storage_sql: Do not keep track of quota when no quota is set No point in doing this expensive O(n) query if the result is not used for anything. Will still cache the total item count if an explicit query for this is performed, then try to keep it updated with new items added. Will likely forget eventually tho.

mod_storage_sql: Add setting to tune SQLite3 performance vs safety Notably the default journal_mode of DELETE is somewhat slow, some users might want to catch up to the amazing performance of internal storage.

mod_storage_sql: Record all SQLite3 compile options for potential use Knowing what features are available could be useful for future experiments. For example, with the JSON module or full text search.

util.sqlite3: Don't cache prepared statements for one-off queries The :execute method is mainly used for one-off queries such as creating tables and indices. There is no need to cache this prepared statement, as those queries are only done on startup. Further, prepared statements can't be reused without being reset, so this was likely broken anyway.

util.sqlite3: Deduplicate query methods There were 3 very similar methods: - :execute() - :execute_query() - :execute_update() The first one returns the prepared statement and is mainly used internally in the library for CREATE statements. The later two only really differ in how the results are returned. Those two are one main method and one small one that only picks out the iterator.

util.sqlite3: Fix indentation

mod_storage_sql: Compose a keyval+ store out of keyval and map store methods Removes the need for the shim in storagemanager. The methods only really access the 'store' property of the first (self) argument, so this is safe.

mod_storage_sql: Add UPSERT support Currently limited to SQLite3 for lack of testing on other databases. Adds a migration to replace the non-UNIQUE prosody_index, renaming it prosody_unique_index since ALTER INDEX does not seem to be portable.

tools: Update imports to use new prosody.* namespace

Merge 0.12->trunk

mod_http: Simplify conversion of Set to Array Avoids the _items semi-private value, that is used everywhere for some reason.

mod_http: Fix error if 'access_control_allow_origins' is set Because it changes the type of the 'opt_origins' variable from util.set to the internal _items table so next time an http app is added an error "attempt to call a nil value (method 'empty')" is triggered. The value is not used anywhere else. Noticed when reviewing uses of the '_items' set property. Not reported by any users, implying this setting is rarely used.

util.array: Expose new() on module table For consistency with other utils. Consistency is good.

util.datamanager: Halve size of list index Instead of storing (start, length) tuples, store the offset to the end of items and derive length using the previous entry.

mod_storage_internal: Use a binary search for time based ranges Iterating over an entire archive to find a few items in the far end from where iteration started is expensive, and probably more expensive with the lazy-loading of items added in the previous commit. Since we can now efficiently read items in random order, we can now use a binary search to find a better starting point for iteration.

mod_storage_internal: Lazy-load archive items while iterating Very large list files previously ran into limits of the Lua parser, or just caused Prosody to freeze while parsing. Using the new index we can parse individual items one at a time. This probably won't reduce overall CPU usage, probably the opposite, but it will reduce the number of items in memory at once and allow collection of items after we iterated past them.