MUC: Don't default component admins to being room owners This change has various technical and social benefits. If ownership of a MUC is really needed, it can be gained using the 'Set affiliation' ad-hoc command or prosodyctl shell. Example client incompatibility with the old behaviour: - https://github.com/monal-im/Monal/issues/1085

MUC: Add per-room PM restriction functionality (thanks Wirlaburla) Based on mod_muc_restrict_pm in prosody-modules d82c0383106a

Merge 0.12->trunk

scansion: Add roster groups setting to pubsub form in tests

scansion: Enable blocklist compat during tests to fix CI

util.prosodyctl: Use notify socket to wait for Prosody to be ready Previously, prosodyctl only waits for the pidfile to appear, which does not necessarily mean that Prosody is fully ready to receive traffic. By waiting until Prosody says it's ready via the systemd notify socket we know for sure that Prosody is really ready. Notably this should ensure that when running `make integration-test` Prosody is really ready when Scansion starts running tests. Not sure if this timeout handling is optimal.

net.server_epoll: Add support for systemd socket activation Allows creating listening sockets and accepting client connections before Prosody starts. This is unlike normal Prosody dynamic resource management, where ports may added and removed at any time, and the ports defined by the config. Weird things happen if these are closed (e.g. due to reload) so here we prevent closing and ensure sockets are reused when opened again.

Merge 0.12->trunk

mod_admin_shell: Allow matching on host or bare JID in c2s:show Only supporting exact match on full JID isn't helpful if you want to list sessions per host or user. Backport of 430333198e4c Fixes #1857

mod_blocklist: Drop blocked messages without error, option to restore compliant behavior From XEP-0191: > For message stanzas, the server SHOULD return an error, which SHOULD > be <service-unavailable/>. Following this may leak to a blocked JID that they have been blocked, which seems contrary to the goal of pretending to be perpetually offline.

mod_pep: Implement 'roster' (group) access_model Allows e.g. restricting your vcard4 to only family or similar. Notes: This does not include roster groups in the configuration form, so the client will have to get them from the actual roster.

mod_announce: Suppress luacheck warnings

mod_announce: Add shell commands and APIs for sending to all/online/roles

prosodyctl shell: Fix invocation with 3+ command arguments The code correctly inserted the ',' when there was already a "%q" in the format string, but then the next argument would fail to match because it inserted ", %q" instead of "%q". The code now matches both, ensuring the generated code will not produce a syntax error with multiple arguments.

mod_blocklist: Fix fix signal for letting stanzas pass Returning nothing/nil lets stanzas pass, returning anything else blocks

mod_blocklist: Check JID of mediated MUC invite sender against blocklist This ensures that someone on your blocklist is unable to invite you to MUC rooms.

mod_saslauth: Log when tls-exporter is NOT supported, as well as when it is

net.unbound: Show canonical name in textual format (e.g. in shell) libunbound does not tell us the whole chain of CNAMEs, only the final canonical name. This is to aid in debugging since it will only be shown in the shell.

mod_http_file_share: Fix expiry disabled check for new config API Similar to 26c30844cac6

util.startup: Fix notifying config-reload to systemd Does this event name seem backwards to anyone else?

mod_version: Fix uname result style (thanks riau) `result[, err]`, not `ok, err|result`, must have confused it with pcall

mod_server_contact_info: Sort form fields to please scansion The unstable hash table order caused the tests to fail and I don't know how to tell scansion to ignore the order.

mod_version: Handle access denied from uname() Discovered while experimenting with a stricter SystemCallFilter setting See man:systemd.exec(5)

mod_admin_shell: Add connection created time This adds an output format option to show the time that the connection was created. Ref #1852

Merge 0.12->trunk

util.startup: Support systemd Type=notify service type This lets Prosody report its lifecycle status to systemd, so it knows when Prosody has completed its startup, when it's reloading and shutting down. Both Type=notify and Type=notify-reload is supported Example systemd .service configuration snippet: [Service] Type=notify

mod_invites_adhoc: Fix result form type (thanks betarays)

MUC: Fix legacy hats (thanks nicoco) Why do we not have tests for this?

MUC: Switch to official XEP-0317 namespace for Hats (including compat) (thanks nicoco)

util.startup: Fix exiting on pidfile trouble prosody.shutdown() relies on prosody.main_thread, which has not been set yet at this point. Doing a clean shutdown might actually be harmful in case it tears down things set up by the conflicting Prosody, such as the very pidfile we were looking at. Thanks again SigmaTel71 for noticing

Merge 0.12->trunk

prosodyctl check: Warn about invalid domain names in the config file This ensures that domain names of virtual hosts and components are valid in XMPP, and that they are encoded correctly.

util.startup: Abort before initialization of logging when started as root Prevents creation of log files owned by the root user which could be inaccessible once started correctly.

util.startup: Don't use not yet existent shutdown procedure when started as root (thanks SigmaTel71)

util.startup: Check root after detecting platform and reading config (thanks SigmaTel71) Ensures that startup.detect_platform() runs so know whether to use the POSIX method of checking the current user or something else. Also after reading the config so we know whether the root override setting is set.

mod_posix: Move everything to util.startup This allows greater control over the order of events. Notably, the internal ordering between daemonization, initialization of libunbound and setup of signal handling is sensitive. libunbound starts a separate thread for processing DNS requests. If this thread is started before signal handling has been set up, it will not inherit the signal handlers and instead behave as it would have before signal handlers were set up, i.e. cause the whole process to immediately exit. libunbound is usually initialized on the first DNS request, usually triggered by an outgoing s2s connection attempt. If daemonization happens before signals have been set up, signals may not be processed at all.

mod_bosh: Set base_type on session This fixes a traceback with mod_saslauth. Ideally we move this to util.session at some point, though.

util.startup: Back out 598df17b8ebb Broke signal handling again, such that an early s2s connection results in libunbound catching signals and getting Prosody killed on e.g. SIGHUP This returns to the situation where prosody --daemonize does not respond to signals.

util.startup: Hook signals after daemonization signalfds stop working with epoll after forking hooking signals later should not affect anything

mod_pubsub: Ignore shadowed variable [luacheck]

mod_pubsub: Add shell commands to create and list nodes

core.features: Advertise that events are fired for SIGUSR1/2 Moved here from mod_posix since these events no longer originate there

util.startup: Fix firing of USR1/2 events

net.server: Restore epoll signalfd handling Reverts 4a9a69659727

mod_posix: Move POSIX signal handling into util.startup to avoid race When libunbound is initialized, it spawns a thread to work in. In case a module initializes libunbound, e.g. by triggering a s2s connection, Prosody would not handle signals, instead immediately quit on e.g. the reload (SIGHUP) signal. Likely because the libunbound thread would not have inherited the signal mask from the main Prosody thread. Thanks Menel, riau and franck-x for reporting and help narrowing down

net.server: Disable epoll signalfd handling by default until problems resolved

net.server_epoll: Log creation of signalfd handles at noise level To aid in tracking down signalfd-related problems

util.bit53: Add bnot() method

util.signal: Fail signalfd() if unable to change signal mask By aborting early, the failure should be brought to the attention somehow.

net.server_epoll: Log failure to hook signals To make any such failures noticeable

Merge 0.12->trunk

net.http.files: Validate argument to setup function Fixes error in #1765 by throwing an error earlier

mod_s2s: Comment on why we avoid hostnames in stanza bounce messages

mod_cron: Fix log format to account for float that was integer before

mod_cron: Sync Teal source with 92301fa7a673 Yeah, it's weird to have two versions. Needing more build dependencies is also something we want to avoid, so here we are.

util.signal: Wrap signalfd in an userdatum for gc handling etc

net.server_epoll: Support hooking signals via signalfd Handling signal events the same way as all other events makes sense and seems safer than the signal handling just jumping around in C and messing with Lua states.

util.signal: Add support for signalfd(2) on Linux signalfd allows handling signal events using the same method as sockets, via file descriptors. Thus all signal dispatch can go through the same main event loop as everything else, removing need for thread-scary signal handling where execution would just jump to the signal handler regardless of the state of Lua, and needing to keep track of Lua states/threads.

features: Add mod_server_info

mod_server_contact_info: Update to publish fields via new mod_server_info

mod_server_info: New module to manage the serverinfo disco extension form This allows multiple modules to populate the form dynamically. Currently the form is "owned" by mod_server_contact_info, which prevents other modules from contributing to it. A further commit will port mod_server_contact_info to use this module.

util.strbitop: Add common_prefix_bits() to Teal interface description

util.hashes: Add missing entries to Teal interface description Also sorted to match C source

util.crypto: Update Teal interface description to match C sources Was missing some entries. Rearranged to match order of entries in the C source Reg table.

tools/test_mutants.sh: Load loader helper when running busted

util.ip: Remove ip.bits and related code, switch to more efficient strbitop 100,000 iterations of match() on my laptop from 3.5s -> 0.5s.

util.strbitop: Remove unused import in tests

util.strbitop: Add common_prefix_bits() method This returns the number of bits that two strings have in common. It is significantly more efficient than similar calculations in Lua.

util.ip: Add another test case for match() and commonPrefixLength()

util.strbitop: Rename spec file to correct name so tests actually run

util.rfc6724: Remove, unused since introduction of Happy Eyeballs It was mainly used to determine whether to try IPv6 or IPv4 first, following the rules for this in the RFC. Now we always try IPv6 and IPv4 at roughly the same time, thus there no need to carry these rules.

features: Add module-ready (for commit e20949a10118)

util.startup: Expose core.features.available as prosody.features for convenience.

mod_s2s_auth_certs: Handle potential string error conn:ssl_peerverification() can now return a single error in case the connection has been closed for whatever reason

net.server_epoll: Prevent traceback when checking TLS after connection gone Unclear why this would be done, but an error is not great.

mod_cron: Allow configuring various "internal" delay parameters Notably, it is now possible to add a randomized spread factor to the check interval.

mod_c2s: Fix error on role change on Components (thanks Menel)

mod_smacks: Adjust buckets for resumption age statistic Given that there are recommendations floating around recommending 24 hours session lifetime, having buckets up to 10 minutes wouldn't be useful in that case. Would be nice if we had some way to automatically assign suitable number series for buckets, scaled to what the configuration might be.

mod_storage_internal: Fix off-by-one when searching archive for Fixes a test case provided by MattJ where the very first item matched by a 'start' timestamp was not returned.

mod_s2s_auth_dane_in: Try single TLSA lookup per draft-ietf-dance-client-auth Moves some complexity from the implementation into DNS operations.

mod_s2s_auth_dane_in: Simplify result processing Fewer loops

MUC: Record reason for affiliation changes and return in list (fixes #1227)

MUC: Test that <subject/> + <thread/> is not handled as subject change Ref #667 Ref #1838

mod_invites: Fix argument handling Not sure what the next() was supposed to do. Reject unknown --options perhaps?

mod_invites: Show short help instead of traceback on missing hostname

mod_invites: Show help if --help passed instead of hostname Because I couldn't guess the right way to get the help message without reading the source twice.

mod_invites: Allow specifying invite ttl on command line Was missing a way to pass TTL via command or shell.

Merge 0.12->trunk

mod_disco: Advertise disco#info and #items on bare JIDs to fix #1664 Having to add these in *there* places seems less than ideal. I would also think that advertising disco#info is a bit redundant, since it is a requirement for everything in XMPP and if it was missing you would get an error back.

scansion: Use new style for accessing Lua globals

scansion: Use new prosody namespace in import

scansion: Use captures or wildcards instead of mocking time > Mockery is one of the things I hold dear! And he's making a mockery of it!! -- Belkar Bitterleaf

util.xtemplate: Test the each template function It iterates over childtags, so a template like {foo|each{...}} would be equivalent to root:childtags("foo"), while a deeper query needs the bit that becomes arguments to :childtags as an argument to each, e.g. {foo/bar|each(baz)} would behave like root:get_child("foo"):get_child("bar"):childtags("baz")

tools: Fix selection of container engine Seems command -v in sh only checks and returns one argument, unlike bash.

mod_storage_internal, tests: Fix before/after combined with the 'reverse' flag

util.http: Silence strict luacheck warning in tests

core.moduleapi: Silence strict luacheck warnings in tests

util.throttle: Silence some strict luacheck warnings

util.prosodyctl.shell: Fix lint [luacheck]

CHANGES: Mention new prosodyctl shell method behavior

util.prosodyctl.shell: Add :method syntax to make e.g. MUC commands easier e.g. prosodyctl shell muc room room@muc.example.com :set_name "This Room"

util.xtemplate: Add some initial tests Strict typing does not magically make code correct

util.xtemplate: Adopt {-path-} syntax to strip preceding and/or trailing whitespace Seen in some other template languages

mod_http_errors: Simplify CSS via built-in dark mode

CHANGES: Document some of the recent changes and features in trunk

configmanager: Fix linter issues

configmanager: Support for appending to existing config options ...and some other useful operations

configmanager: Make _G accessible via `Lua` variable, deprecate direct access

configmanager: Allow referencing previously-set options in the config file

mod_user_account_management: Clear pending deletion if account re-enabled

mod_saslauth: Fire event per SASL step This matches the behaviour of the newer mod_sasl2 implementation. It allows plugins to observe (and potentially, with caution, modify) the SASL exchange.

util.jsonschema: Return basic structured validation response

mod_c2s: Make c2s_timeout timer reachable to allow access from other modules E.g. the timeout could be extended under certain conditions.

tools: Add a tool for comparing DOAP to the latest XEP versions Needs wget, awk, sed and xml2

mod_s2s: Close connection on smacks timeout This merges the mod_s2s_smacks_timeout behavior from prosody-modules This event is fired by mod_smacks when the connection has not responded to an ack-request for a period of time defaulting to 30 seconds, indicating that the connection has become stuck or non-responsive. Closing it prevents routing further messages via this connection and frees resources. A stuck connection may otherwise remain until for a time determined by the OS TCP subsystem, which can be quite long.

mod_saslauth: Fire event at start of authentication attempt As extension point for rate limiting and similar checks, so they can hook a single event instead of <{sasl1}auth> or stream features, which might not be fired in case of SASL2 or e.g. HTTP based login.

net.http.server: Fix whitespace-ignoring syntax

Merge 0.12->trunk

net.http.parser: Reject overlarge header section earlier This case would eventually be rejected by the buffer size limit.

lua-format: Let simple things be one line This doesn't really handle nesting all that nicely tho.

lua-format: Further tweaks Keeping things a single line makes very deeply nested things "pyramids" a single line, which makes them hard to read.

lua-format: Add new settings

lua-format: Tweaks attempting to fit our code style

lua-format: Check in defaults By starting with the built-in defaults, we get a nice history of differences from those as we figure out what settings suit us Sorted make comparisons easier.

mod_user_account_management: Fire events with a fake (not destroyed) session Previously these events fired after the session had been destroyed, which removes many of the useful properties. The ones I chose to preserve here are the ones used by the community module mod_audit, which seems like a good baseline.

mod_cron: Rebuild with new LuaFormatter settings (tabs!)

util.datamanager: Fix missing pack format when reading first index entry Thanks MattJ

mod_user_account_management: Add support for soft-deletion of accounts via IBR When registration_delete_grace_period is set, accounts will be disabled for the specified grace period before they are fully deleted. During the grace period, accounts can be restored with the user:restore() shell command. The primary purpose is to prevent accidental or malicious deletion of a user's account, which is traditionally very easy for any XMPP client to do with a single stanza.

moduleapi: Log error message when ambiguous period spec is found in config

util.human.io: Don't accept ambiguous durations by default The new method parse_duration_lax() exports the old behaviour, mainly for compatibility purposes.

mod_cron: Update Teal source and rebuild

mod_cron: Rename variable to fix shadowing (#luacheck)

mod_cron: Add shell command to list registered cron tasks with status

mod_saslauth: Allow plugins to override return SASL condition/text

luacheckrc, mod_http_file_share: Update for module API change (once->on_ready)

modulemanager: Allow modules to expose module.ready - to be called after init This is a shortcut for module:on_ready() which exposes the functionality in an idiomatic way consistent with module.load, module.unload, etc. module.ready runs when the module is loaded and the server has finished starting up.

moduleapi: Rename :once() to :on_ready() for clarity 'Once' is ambiguous - once per what? on_ready() executes its parameter when the module is loaded *and* the server has finished starting.

usermanager, mod_auth_internal_hashed: Support metadata when disabling a user This allows us to store a time, actor, comment and/or reason why an account was disabled, which seems a generally useful thing to support.

util.datamanager: Load first item into index earlier Should get rid of fseek() call

mod_mam: Use for loop in metadata query Some storage drivers will perform cleanup after the last iteration, but if only one step is taken this might be delayed until the garbage collector gets to it.

mod_tokenauth: Ignore invalid grants in storage that have no id

mod_invites: Fix linter issues

mod_admin_shell: Fix linter issues

mod_invites: Use new shell-command API

mod_admin_shell: Remove verbose logging

mod_admin_shell: Remove timer:info() (it's been debug:timers() for some time)

mod_admin_shell: Support for 'shell-command' items (global and per-host) This should simplify adding shell commands from other modules, which will reduce the growth of mod_admin_shell and make it easier for community modules to expose commands too.

mod_admin_shell: Refactor help to data structures for extensibility This makes it easier for commands added by other modules to add to the help output, for example.

migrator: Add an escape hatch to allow arbitrary config options Previously only SQL settings and the 'path' for internal storage could be set, and only for SQL and internal storage. input { type = "whatever"; config = { whatever_foobar = "something" } }

util.poll: Rename things to clarify poll(2) limits With epoll(7), MAX_EVENTS controls how many events can be retrieved in one epoll_wait call, while with poll(2) this MAX_WATCHED controls how many sockets or other FDs can be watched at once.

mod_storage_internal: Clear archive item count cache after experimental trim The previous count would be invalid at this point. Should be possible to math out how many items are left, but this is left as future work.

mod_blocklist: Remove weak cache (and increase default LRU cache size) Weak tables are said to have suboptimal performance, so we might as well get replace it with an increased default LRU cache size. Sorry about the 'and'

mod_storage_internal: Close lazy-loading list abstraction after trim Should be done here too.

mod_storage_internal: Only close lazy-loading list store abstractions Since datamanager can fall back to the old method of loading the whole list, which wouldn't come with a :close method.

util.startup: Use prosody. module namespace Maybe we need some sort of lint for this?

net.http.server: Complete async waiter for non-persistent connections Otherwise requests with Connection: close would be stuck in the async wait that starts after the handle_request() call. Together with the new async debugging, this makes the async thread stay in the set of waiting runners forever, where previously it would simply be garbage collected.

mod_storage_internal: Close archive list after completion of iteration This closes the two FDs that the random access list abstraction uses, otherwise they are left to the garbage collector.

mod_pubsub: Provide some node properties in summary template #1809 Gives some access to node details which are otherwise hard to determine if you only see the plain text summary, since it is shared based on the pubsub#type setting (or payload xmlns).

mod_admin_shell: Fix lint [luacheck]

util.async: Clip long line [luacheck]

net.server_epoll: Avoid call to update socket watch flags when nothing changed Should skip a syscall for each write when using epoll.

util.poll: Quadruple number of events retrieved at once from epoll Better performance under load maybe? See b890ceb1c24f for previous increase

mod_admin_shell: Add debug:async() command to show blocked async runners

util.async: Expose default runner function This is purely for informational purposes, so it's possible to determine externally whether a runner is using the default runner function (which executes functions as work items) or a custom runner function.

util.async: Record current work item in the runner object Mostly expected to be useful for debugging purposes.

util.async: Improve debug logging in a few places Knowing the state of the coroutine as well as the runner state can be helpful.

util.async: Export a table of currently-waiting runners This can be used for debugging and introspection.

util.poll: Return early if given zero timeout and no pending events Should have been part of f33887f925e1 to ensure it won't skip processing timers at all when very busy.

moduleapi: Update Teal spec Updates for 65fb0d7a2312::59c3d775c7fa

util.prosodyctl.check: Disable https cert check if http_external_url set This would indicate that a reverse proxy is used, which gets to be responsible for that since it probably holds the actual cert.

util.prosodyctl.check: Check cert for HTTPS if http module enabled

util.prosodyctl.check: Update conditions for s2s cert checks The 'anonymous_login' setting is deprecated and prosodyctl check config will tell you to change it to 'authentication = "internal_hashed"', so we shouldn't need to care about here anymore.

util.prosodyctl.check: Simplify conditions for c2s and s2s cert checks This code is hard to follow and in need of some refactoring.

tools/build-env: Tools for building and testing in a container ./tools/build-env/build.sh Creates a container image based on Debian or Ubuntu ./tools/build-env/here.sh Starts a container and mounts in the current working directory, from where one can ./configure; make; make test etc

mod_s2s_auth_dane_in: Bail out on explicit service denial

mod_tokenauth: Include more details in debug logs Had a hard time following what was happening when it did not specify which grant or token was being removed.

net.http: Set Connection header based on connection pool usage Connection: keep-alive is implicit in HTTP/1.1 but explicit > implicit

net.http: Add simple connection pooling This should speed up repeated requests to the same site by keeping their connections around and sending more requests on them. Sending multiple requests at the same time is not supported, instead a request started while another to the same authority is in progress would open a new one and the first one to complete would go back in the pool. This could be investigated in the future. Some http servers limit the number of requests per connection and this is not tested and could cause one request to fail, but hopefully it will close the connection and prevent it from being reused.

mod_storage_sql: Use UUIDv7 as keys Potentially allows sorting on those directly as they will be in increasing order.

util.uuid: Add UUIDv7 Allows sorting by id as a substitute for sorting by timestamp since it has the timestamp in the encoded in the first part, and only things that happen extremely close together may get out of order by such a sort, which might not matter. From draft-ietf-uuidrev-rfc4122bis formerly draft-peabody-dispatch-new-uuid-format

util.prosodyctl.check: Try to clarify check for misplaced k=v in modules_enabled (thanks aab and Menel)

doap: Update XEP-0359 version, no protocol changes Security considerations added, no protocol changes.

doap: Update XEP-0353 version, no change affecting server handling

doap: Update XEP-0313 version, only change align with current mod_mam behavior

doap: Update XEP-0045 version, only minor changes

util.startup: Attempt to bring some order to startup/shutdown with util.fsm

.luacheckrc: Add module:could()

moduleapi: may(): Support explicit actor_jid in context object

mod_muc: Switch to module:could() for some implicit access control checks

mod_muc: Allow guest users to list rooms by default

mod_muc: Add :list-rooms permission

mod_tokenauth: Fix saving grants after clearing expired tokens Previously the whole grant was deleted if it found one expired toke, which was not indented.

mod_s2s_auth_certs: Remove LuaSec compat that moved to net.server

core.certmanager: Handle dane context setting same way on reload as on initialization

util.prosodyctl.check: Print DANE TLSA records for certificates Not the prosodyctl check dane I wanted to make but a start.

util.prosodyctl.check: Wrap each check in a function One small refactor but one huge step in the right direction Mostly because adding another check would make the line checking for a valid check exceed the column limit.

muc.register: Clarify what's going on when enforcing nicknames Does this make it clearer what is going on?

util.datamanager: Clean up list index files on purge (i.e. user deletion)

mod_s2s: Automagically enable DANE for s2sin if 'use_dane' is enabled Simplifies configuration, only one already existing boolean to flip.

mod_s2s_auth_dane_in: DANE support for s2sin Complements the DANE support for outgoing connections included in net.connect

migrator: Add mod_http_file_share example to config template

migrator: Update default config template with new stores * mod_authz_internal adds account_roles * mod_cron has its state * mod_smacks also has some non-critical state

core.certmanager: Tweak log level of message about SNI being required Everything supports SNI today, so this is not useful information.

mod_bosh: Include stream attributes in stream-features event This matches what mod_c2s does, and fixes a traceback in mod_sasl2_fast when used with BOSH (that module tries to use event.stream.from).

Merge 0.12->trunk

core.certmanager: Validate that 'tls_profile' is one of the valid values A typo should not result in ending up with "legacy"

mod_saslauth: Clear 'auto' from endpoint hash var, it's not a real hash (thanks tmolitor)

mod_saslauth, mod_c2s: Disable tls-server-end-point channel binding by default This channel binding method is now enabled when a hash is manually set in the config, or it attempts to discover the hash automatically if the value is the special string "auto". A related change to mod_c2s prevents complicated certificate lookups in the client connection hot path - this work now happens only when this channel binding method is used. I'm not aware of anything else that uses ssl_cfg (vs ssl_ctx). Rationale for disabling by default: - Minor performance impact in automatic cert detection - This method is weak against a leaked/stolen private key (other methods such as 'tls-exporter' would not be compromised in such a case) Rationale for keeping the implementation: - For some deployments, this may be the only method available (e.g. due to TLS offloading in another process/server).

mod_saslauth: Fix traceback in tls-server-end-point channel binding

mod_admin_shell: Make 'Role' column dynamically sized Some of the new roles don't quite fit nicely into 4 characters (excluding ellipsis). Given the ability to dynamically add additional roles from the config and possibly from modules, it seems better to just make it a relative size since we can't know how long they will be.

mod_saslauth: Actively close cert file after reading Explicit > implicit

mod_saslauth: Fix read format string (thanks tmolitor)

mod_cron: Make task frequencies configurable in overly generic manner Requested feature for many modules, notably MAM and file sharing.

mod_cron: Fix missing restore method in Teal record definition

CHANGES: Mention 'tls-server-end-point'

mod_saslauth: Get correct 'tls-server-end-point' with new LuaSec API MattJ contributed new APIs for retrieving the actually used certificate and chain to LuaSec, which are not in a release at the time of this commit.

mod_c2s: Add session.ssl_cfg/ssl_ctx for direct TLS connections

portmanager: Expose API to get at SSL/TLS config for a given interface/port

mod_saslauth: Derive hash from certificate per tls-server-end-point This originally used a WIP implementation of cert:sigalg(), a method to retrieve certificate signature algorithm, but it was never submitted upstream. https://github.com/Zash/luasec/tree/zash/sigalg cert:getsignaturename() was merged in https://github.com/brunoos/luasec/commit/de393417b7c7566caf1e0a0ad54132942ac4f049 XEP-0440 v0.3.0 made implementing tls-server-end-point a MUST

mod_saslauth: Support tls-server-end-point via manually specified hash Since this channel binding method is said to enable TLS offloading then you need tell Prosody the hash (or the full cert), so this seems like a good start. Support is RECOMMENDED in XEP-0440 version 0.2

mod_tokenauth: Set name/description on cleanup job

mod_tokenauth: Save grant after removing expired tokens Ensures the periodic cleanup really does remove expired tokens.

mod_tokenauth: Periodically clear out expired tokens and grants This should ensure expired grants eventually disappear.

mod_tokenauth: Delete grants without tokens after period Generally it is expected that a grant would have at least one token as long as the grant is in active use. Refresh tokens issued by mod_http_oauth2 have a lifetime of one week by default, so the idea here is that if that refresh token expired and another week goes by without the grant being used, then the whole grant can be removed.

mod_tokenauth: Clear expired tokens on grant retrieval

mod_tokenauth: Delete grants in the wrong formats on retrieval

lint: Teach luacheck about module:once Silence warning for using this introduced in 9c62ffbdf2ae

mod_cron: Remove unused import [luacheck] Use of datetime was removed in 6ac5ad578565

Merge 0.12->trunk

mod_muc_mam: Improve wording of enable setting Suggested by jstein in the chat This option label is used by XMPP clients to explain what the option does. a) The user should know where the data is archived. b) The user needs a statement that can be enabled/disabled by the variable. A question would have the wrong logic here.

mod_http_file_share: Retrieve stored total in async-friendly way Does this run in a thread?

mod_cron: Load last task run time inside task runner to fix async This ensures that all interactions with storage happen inside an async thread, allowing async waiting to be performed in storage drivers.

mod_cron: Revert bbd3ac65640d Maybe it is better to run daily and weekly tasks 'now' on the theory that people set these things up during times that are appropriate for maintenance already, so the same time next day or next week might be fine for periodic cleanup.

mod_storage_internal: Don't report error when attempting to trim empty archive Fixes "Could not delete messages for room 'x': (nil)"

mod_storage_internal: Fix fast trimming of archive with exactly one item This method would previously never delete the first (and only) item since it works out which item should become the first item after the trim operation, which doesn't make sense when all should be removed. This also works as an optimization for when all the last item should be trimmed, thus items should be removed.

net.http.server: Fix typo in previous commit

net.http.server: Support setting Content-Type of uncaught HTTP errors mod_http_errors normally sets the Content-Type header via the response object, which isn't available when handling these uncaught errors. Without a Content-Type header the browser is forced to guess, which may or may not result in something sensible.

mod_http_file_share: Switch to the new authz API (BC) Behavior change: It becomes up to the authorization module whether to allow requests. The default, mod_authz_internal, will allow users on the *parent* host only, breaking use by some components. Remaining question is whether to deprecate the `http_file_share_access` setting or leave as a way to complement/bypass access control?

core.storagemanager: Remove 0.10 era sql2 driver COMPAT Unlikely that anyone has had sql2 in their configs for a long time, so this serves little purpose. Leaving the indirection function in case some similar compat code is needed in the future.

Added tag 0.12.4 for changeset a2ba3f06dcf4

Merge 0.12->trunk

tools: Reflect bash-ness of mod2spec in shebang Non-portable substitution syntax? Oh well.

util.prosodyctl.check: Correct modern replacement for 'disallow_s2s' The code would have suggested adding to modules_enabled instead of modules_disabled

mod_s2s: Fix reporting of DANE mismatch Thought it was a case mismatch at first, fixed that, but it changed nothing because the error was in the leaf part of the errors, not the chain part.

core.portmanager: Join strings broken into multiple lines Improves readability. Reduces line count. What's not to like? The code style and luacheck rules allows longer lines, and these strings aren't long enough to need breaking into multiple lines like this.

core.portmanager: Hint at HTTP servers for conflicts over port 443 Since 443 is just as much a web port as port 80 these days, if not more. What's with port 81 here?

net.websocket.frames: Remove completed TODO The XOR is done in C since 4e5a2af9dd19

mod_http: Generate URL from configuration in prosodyctl This removes the need to configure e.g. http_external_url or similar settings in order to get correct URLs out of prosodyctl, as the API depends on portmanager to know the actual ports that are used.

Merge 0.12->trunk

util.array: Fix new() library function Backport of ffe4adbd2af9 since new was added in the 0.12 branch

mod_tokenauth: Fix revoking a single token without revoking whole grant This appears to have been a copy-paste of the grant revocation function, or maybe the other way around. Either way, it deleted the whole grant instead of the individual token as might be expected.

util.array: Fix new() library function

tests: Add hack to test only a single storage driver Fixes that LuaDBI being unavailable makes these produce nothing but endless stack overflows in luarocks.

tests: Update storagemanager tests for prosody.* namespace change Part of an attempt to make these tests work again. Previously they would just explode in a million luarocks stack overflows

mod_storage_sql: Spell out missing dependencies Using util.dependencies appeared to cause problems with running tests in Busted, so this also removes that and uses pcall directly.

mod_storage_sql: Pass variables as arguments instead of upvalues Probably a workaround for the lack of argument passing when using xpcall in Lua 5.1, no longer relevant.

util.sql: Remove unused String() and Integer() functions According to MattJ, leftovers from an earlier vision for util.sql

util.sqlite3: Clean up unused variables Many leftovers from the earlier version of util.sql this was based on and cleanup applied there since then.

util.datamanager: Always reset index after list shift Shifting the index does not work reliably yet, better to rebuild it from scratch. Since there is minimal parsing involved in that, it should be more efficient anyway.

core.moduleapi: Parse period min/max arguments Allows specifying them the same way as the default and in the config, for consistency

util.datamanager: Add way to close indexed list store

util.datamanager: Close file handle when done using it It gets closed eventually but at high load they could potentially lead to reaching FD limits faster.

util.datamanager: Disable blockwise removal In desperate need of tests

mod_muc: Use enum config API for 'restrict_room_creation' This communicates the accepted values in case the config diverges from them. Note that older documentation used an "admin" value behaving like an alias to true, but this is no longer handled. Should it?

plugins: Use get_option_array for some list shaped options Passing something from module:get_option() to ipairs() suggests that the option is a list of some sort.

util.datamanager: Disable block alignment Until we have more test coverage. Somehow the index becomes incorrect after inserting padding, unclear why.

plugins: Handle how get_option_period returns "never"

plugins: Use boolean config method in some places Because it makes sense and improves feedback via logging

mod_storage_internal: Use integer option method for cache size Missed this one in previous sweep

mod_muc_mam: Use period option method

mod_muc: Use enum option method for 'muc_room_default_presence_broadcast'

mod_storage_xep0227: Use enum option method

mod_auth_ldap: Use enum option method

mod_storage_sql: Use integer config option for cache size Missed this one, was probably only looking for get_option_number

mod_storage_sql: Use config enum for 'sqlite_tune'

mod_storage_sql: Allow higher precision timestamps in SQLite3 Since it doesn't actually do strict typing :)

Merge 0.12->trunk

util.prosodyctl.check: Hint about the 'external_addresses' config option

Merge 0.12->trunk

util.prosodyctl.check: Validate format of module list options Should detect things like misplaced settings inside modules_enabled

util.prosodyctl.check: Get some config options via minimal moduleapi #896 The module API has certain coercion features that are useful. Fixes traceback reported in #1812 and other duplicates

core.moduleapi: Fix min/maxinteger fallback for Lua 5.2 Maybe these should live in util.mathcompat?

mod_http: Fix passing minimum limits in wrong argument position

plugins: Use integer config API with interval specification where sensible Many of these fall into a few categories: - util.cache size, must be >= 1 - byte or item counts that logically can't be negative - port numbers that should be in 1..0xffff

core.moduleapi: Add min/max range support to :get_option_period To match :get_option_number etc, specifying the allowed interval. Default is essentially (0, inf].

moduleapi: Add :get_option_integer() Many options in Prosody that are treated as numbers don't make sense as floats, e.g. sizes and limits measured in bytes. Simplified implementation based on an earlier attempt dating back to 2020

util.human.io: Fix stray 'stty' error by only querying width of real ttys This adds a dependency on a binary and *nix-specific module but then stty is probably *nix-specific anyway so maybe that's fine.

plugins: Switch to :get_option_period() for time range options Improves readability ("1 day" vs 86400) and centralizes validation.

core.moduleapi: Accept boolean false to disable period setting

core.moduleapi: Log error for unexpected types (booleans?) set as periods

core.moduleapi: Turn negative periods or "never" into infinity As a way to signal that the periodic thing should be disabled, matching existing mod_mam usage

core.moduleapi: Improve handling of different types in :get_option_period Pass positive numbers trough unharmed, parse strings as periods, discard anything else.

core.moduleapi: Add :get_option_period for parsing time intervals E.g. for use in mod_mam and others that take an amount of time before some (usually cleanup) action is taken.

core.moduleapi: Allow specifying an acceptable range for number options

plugins: Use get_option_enum where appropriate

moduleapi: Add enum config option method For when a setting has a few fixed values it can take

tools: Fix file ending of mod2spec.sh (thanks buildbot) Accidentally .lua ?

util.human.io: Fix pattern in parse_duration() to cover all used letters Notably 'h' was missing. Awkwardly, 'hour' would result in 'ho' which was missing from table.

util.human.io: Add tests for parse_duration() (some failing)

util.human.io: Include relevant arguments in test messages This way the relevant arguments are shown in case a test case fails

util.human.io: Use tail call in test to get correct line numbers This is probably not guaranteed to work and might vary with Lua version, but it's good enough for me to get accurate line numbers out of Busted that don't all point to the test() function.

tools: Add mod2spec.sh, turns util.example into spec/util_example_spec.lua Useful for opening a module and its tests at the same, can be awkward to auto-complete sometimes. sensible-editor util/example.lua $(./tools/mod2spec.sh util.example)

editorconfig: Include the command used to normalize *.xml The double asterisk seems unnecessary.

editorconfig: Specify max line length to match luacheck settings

editorconfig: Explicitly specify preferred charset (UTF-8) Really should be the default everywhere by now, but doesn't hurt to be extra explicit

editorconfig: Add link to format description

mod_storage_sql: Remove completed TODO (testing UPSERT on PostgreSQL)

CHANGES: Move line about LuaSQLite3 to Storage section

CHANGES: Mention performance improvements for internal archives Specifically the index and more efficient delete. These are however still in need of testing.

mod_storage_internal: Implement efficient deletion of oldest archive items Using the new shift function in datamanager, either the oldest items are removed or all the later items are moved into a new file that replaces the old. Hidden behind a feature flag for now.

util.datamanager: Pad list writes to avoid crossing block boundaries By padding items so that they do not cross block boundaries, it becomes eaiser to delete whole blocks with fallocate() without cutting items in half, improving efficiency of such operations. Since list stores are used for message archives, where the most common deletion operation would be of the oldest entires, at the top of the file. With this, all blocks that contain items to be removed could be deleted without needing to read, delete and write out the whole file.

util.datamanager: Efficiently remove whole blocks to shift lists Using the new pposix.remove_blocks() it should be very performant to delete whole sections of a file, given a supporting file system.

util.pposix: Add remove_blocks() for deleting parts of files Allows implementing e.g. a FIFO Will probably only work on some Linux file systems like ext4.

util.datamanager: Add way to efficiently remove first items in a list Copying data without parsing it should be more performant than parsing it serializing back.

util.datamanager: Fix indexing first item if not at the very start If the first item does not start at position 0 then the index function produces a phantom first entry covering position zero until where the real first item starts. When using the index, this would make it either appear as the first item was missing or cause an off-by-one issue with remaining items.

util.datamanager: Reduce log level of left over debug messages to debug (thanks Trung) These were mostly 'warn' to make them stand out from the debug noise

util.datamanager: Fix missing separator in log line

Merge 0.12->trunk

core.certmanager: Update Mozilla TLS config to version 5.7 Ref https://github.com/mozilla/server-side-tls/issues/285

mod_pubsub: Send correct jid attribute in disco#items Fixes use in PEP where the JID does not equal the bare domain.

mod_http_file_share: Put 'expires' back, thought it was unused Removed in 536055476912 because it was not used anywhere else in the file, but per the documentation it is meant to inform external upload services of the expiry time of the upload itself.

util.cache: Pass cache itself to eviction callback Simplifies access to the cache without moving code around a lot given the currently common pattern of local some_cache = cache.new(size, function(k,v) end)

util.cache: Keep eviction candidate if callback resized to make room Previously either the old or the new values would be rejected, even if the cache was resized to allow more items.

util.serialization: Teach Teal about the new "pretty" preset

util.jsonpointer: Change function prototype to allow anything But anything that's not a table can't be resolved into, which could happen in the middle, so eh.

util.jsonpointer: Silence Teal warning It seems to think 'table' never has array items, but we don't know that.

core, plugins: Split prosody:user role into prosody:{guest,registered,member} This gives us more granular control over different types of user account. Accounts registered by IBR get assigned prosody:registered by default, while accounts provisioned by an admin (e.g. via prosodyctl shell) will receive prosody:member by default.

usermanager: Add create_user_with_role() method to atomically set initial role

mod_http_file_share: Set slot token TTL so util.jwt validates expiry Overrides the util.jwt default of 1h with the intended TTL of 10 minutes. Because util.jwt now has its own expiry checks, so the 'expiry' field is no longer used and can thus be removed.

prosodyctl: Add experimental way to reload specific modules directly Mostly thinking out loud about how various actions may use the shell This enables the following sequence of commands: prosodyctl install mod_example prosodyctl reload mod_example which is simpler than prosodyctl shell module reload example

net.server: Handle loading from outside Prosody (e.g. Verse) server_select only depending on LuaSocket generally makes it more portable, so fall back to that if util.poll can't be found.

renamening: Fix newly added imports to use the new namespace

util.jsonschema: Remove wrapper function This was to silence some Teal warning that seems to have gone away.

util.jsonschema: Silence Teal warnings about utf8 library Teal worries that we redefine the global. Also that the fallback was missing type information.

util.jsonschema: Silence Teal warnings about counting items in tables Teal thinks that these are key-value maps which are always of length zero, but that is not the case.

mod_invites: Refactor argument handling using util.argparse This makes it so that --admin and --role are no longer mutually exclusive, they the former is simply treated as another --role. This was likely a leftover from when only a single role was possible. It does however become unclear which should be the primary, since the order is not preserved by argparse. Bonus: Loading of modules is avoided with only the --help is shown.

util.argparse: Add support for repeatable parameters These are gathered into arrays

mod_storage_sql: Fix column name in index check for PostgreSQL Forgot to change the column name in 9a7523ea45cb

mod_storage_sql: Adjust indentation to align with surrounding code

mod_storage_sql: Only remove old index if it exists Avoids an error if the upgrade is performed twice..

mod_storage_sql: Be more specific when checking for old index in SQLite3 Prevents false positives in the odd case where something other than an index with this name might exist.

mod_storage_sql: Improve check for old table index on PostgreSQL The "pg_indexes" view is much simpler to inspect than "pg_class"

mod_storage_sql: Enable UPSERT with PostgreSQL Tested. Works.

doap: Fix typo in attribute name

util.pposix: Use Lua enum API for resource limit name argument Because diffstat.

mod_storage_sql: Add some TODO comments for future UPSERT work

mod_storage_sql: Do not keep track of quota when no quota is set No point in doing this expensive O(n) query if the result is not used for anything. Will still cache the total item count if an explicit query for this is performed, then try to keep it updated with new items added. Will likely forget eventually tho.

mod_storage_sql: Add setting to tune SQLite3 performance vs safety Notably the default journal_mode of DELETE is somewhat slow, some users might want to catch up to the amazing performance of internal storage.

mod_storage_sql: Record all SQLite3 compile options for potential use Knowing what features are available could be useful for future experiments. For example, with the JSON module or full text search.

util.sqlite3: Don't cache prepared statements for one-off queries The :execute method is mainly used for one-off queries such as creating tables and indices. There is no need to cache this prepared statement, as those queries are only done on startup. Further, prepared statements can't be reused without being reset, so this was likely broken anyway.

util.sqlite3: Deduplicate query methods There were 3 very similar methods: - :execute() - :execute_query() - :execute_update() The first one returns the prepared statement and is mainly used internally in the library for CREATE statements. The later two only really differ in how the results are returned. Those two are one main method and one small one that only picks out the iterator.

util.sqlite3: Fix indentation

mod_storage_sql: Compose a keyval+ store out of keyval and map store methods Removes the need for the shim in storagemanager. The methods only really access the 'store' property of the first (self) argument, so this is safe.

mod_storage_sql: Add UPSERT support Currently limited to SQLite3 for lack of testing on other databases. Adds a migration to replace the non-UNIQUE prosody_index, renaming it prosody_unique_index since ALTER INDEX does not seem to be portable.

tools: Update imports to use new prosody.* namespace

Merge 0.12->trunk

mod_http: Simplify conversion of Set to Array Avoids the _items semi-private value, that is used everywhere for some reason.

mod_http: Fix error if 'access_control_allow_origins' is set Because it changes the type of the 'opt_origins' variable from util.set to the internal _items table so next time an http app is added an error "attempt to call a nil value (method 'empty')" is triggered. The value is not used anywhere else. Noticed when reviewing uses of the '_items' set property. Not reported by any users, implying this setting is rarely used.

util.array: Expose new() on module table For consistency with other utils. Consistency is good.

util.datamanager: Halve size of list index Instead of storing (start, length) tuples, store the offset to the end of items and derive length using the previous entry.

mod_storage_internal: Use a binary search for time based ranges Iterating over an entire archive to find a few items in the far end from where iteration started is expensive, and probably more expensive with the lazy-loading of items added in the previous commit. Since we can now efficiently read items in random order, we can now use a binary search to find a better starting point for iteration.

mod_storage_internal: Lazy-load archive items while iterating Very large list files previously ran into limits of the Lua parser, or just caused Prosody to freeze while parsing. Using the new index we can parse individual items one at a time. This probably won't reduce overall CPU usage, probably the opposite, but it will reduce the number of items in memory at once and allow collection of items after we iterated past them.

util.datamanager: Add O(1) list indexing with on-disk index Index file contains offsets and lengths of each item() which allows seeking directly to each item and reading it without parsing the entire file. Also allows tricks like binary search, assuming items have some defined order. We take advantage of the 1-based indexing in tables to store a magic header in the 0 position, so that table index 1 ends up at file index 1.

storagemanager tests: Reorder test data in chronological order Why was the test data not in chronological order? Altho, maybe that was the point? Except for MAM, the data might *not* be in chronological order!

mod_admin_shell: Use new serialize preset to simplify default config Two pairs replaced by one. Blame lua-format for the line diff delta.

util.serialization: Add a "pretty" preset This is the config I want 90% of the time when just showing data in the console or so.

doc/hgrc-email: Example config for using 'hg email' to contribute The initial setup can be tricky if you don't know what and were settings should be added. This should maybe also go into site/doc/contributing

doc/hgrc: Some useful Mercurial settings Some useful settings that might benefit new contributors and get them up to speed with Modern Mercurial™ faster :)

mod_admin_shell: Warn when (un-)loading module would be undone by restart Reminder to update the configuration if the change is to be permanent.

mod_http: Make RFC 7239 Forwarded opt-in for now to be safe Supporting both methods at the same time may open to spoofing attacks, whereby a client sends a Forwarded header that is not stripped by a reverse proxy, leading Prosody to use that instead of the X-Forwarded-* headers actually sent by the proxy. By only supporting one at a time, it can be configured to match what the proxy uses. Disabled by default since implementations are sparse and X-Forwarded-* are everywhere.

mod_http: Use RFC 7239 Forwarded header to find original client IP Prefer over X-Forwarded-* since it has an actual specification. Main practical difference is that Forwarded may carry more properties than only the IP address since it is a structured header. Since we parse it into an array, it is easier to do the logical thing and iterate backwards trough proxies until an untrusted one is encountered. Compare the handling of X-Forwarded-For. The 'secure' field now accounts for the full chain of proxies, which must be secure all the way to be considered secure.

mod_http: Handle bracketed IP address format from RFC 7239 There are hints that this format might be used in X-Forwarded-For as well, so best handle it everywhere. Strips both brackets and optional port number.

util.http: Implement parser for RFC 7239 Forwarded header Standardized and structured replacement for the X-Forwarded-For, X-Forwarded-Proto set of headers. Notably, this allows per-hop protocol information, unlike X-Forwarded-Proto which is always a single value for some reason.

util.set: Remove duplicate __freeze metamethod Backs out 895a82c5d8d4 beacuse __freeze already added in a96a2fbcc6c0

Merge 0.12->trunk

util.prosodyctl.check: Fix error where hostname can't be turned into A label Where gethostname or tohostname returns an invalid name, e.g. containing underscores or something, to_ascii would reject this and return nil, which triggers an error in the dns lookup. Reported by prova2 in the chat, for whom tohostname returned a long name containing underscores.

util.startup: Record current version in a metric Useful to have this info available when juggling metrics, e.g. to see if things changed between versions.

util.startup: Remove componentmanager backwards compatibility Module was removed in 0.8.0 in c52b06de9b27

net.http.server: Return request ID in header to aid debugging Eases locating the request in logs

mod_admin_shell: Show internal URL where different from external

net.tls_luasec: Expose method for loading a certificate Further isolates LuaSec from Prosody core, with the ultimate goal of allowing LuaSec to be replaced more easily.

net.certmanager: Move LuaSec feature detection to net.tls_luasec Further isolates LuaSec from Prosody core, with the ultimate goal of allowing LuaSec to be replaced more easily.

util.dependencies: Print tables itself to reduce number of imports Rationale: See diffstat When this module is imported, it ends up calling stty via term_width() in util.human.io.table(). When this happens outside of a terminal, the following message is sent to stdout: stty: 'standard input': Inappropriate ioctl for device Not importing this module avoids that. Furthermore three is value in this module having minimal dependencies as they might not be available when it does the checks. Ref a1fed82c44b9

util.sasl: Add basic tests for OAUTHBEARER

util.sasl.oauthbearer: Tighter parsing of SASL message Previously the kvsep before and after the kvpairs would have been included in kvpairs, which is incorrect but should be harmless.

Merge 0.12->trunk

mod_s2s: Add event where resolver for s2sout can be tweaked Could be used to implement custom connection methods (c.f. mod_onions) without needing to duplicate the rest of route_to_new_session(). Adds a feature to enable detection since it can be difficult to detect support for an event otherwise.

teal: Describe http_url method It is not part of the "real" module API, but used in various places. Extending the API seems hard to describe in a type-safe way.

mod_admin_shell: Show internal URL in addition to external in http:list To help with configuring reverse proxies.

mod_http: Add way to retrieve internal URL instead of external This could be of help when configuring reverse proxies, as it is the internal URL the proxy must point at. Argument treated as an enum "internal" "external"(default) to allow for future extensibility.

util.jsonschema: Update test suite ignore rules A test case was added in the middle, so all these need to be reordered.

mod_admin_shell: Allow logging HTTP events with debug:logevents("http") Mirroring debug:events("http"), and to replace the "Firing event: GET /" log lines in net.http.server

mod_admin_shell: Allow logging global events with debug:logevents("*") Missing feature. It should behave like debug:events()

net.http.server: Remove "Firing event" logs, use event logging instead Since these are noisy and we have the thing in util.helpers to log events fired. The new status line events are meant to replace these as they include more useful info.

net.http.server: Log request and response status lines Points out the beginning and end of a request.

net.http.server: Assign each request its own log source

net.http.server: Assign an ID to each request, shared with response Goal is improve tracking of individual HTTP requests throughout its life-cycle. Having a single ID to use in logging should help here.

mod_tokenauth: Support selection of _no_ role at all If a grant does not have a role, we should not go and make one up. While not very useful for XMPP if you can't even login, it may be useful for OAuth2/OIDC.

mod_tokenauth: Return error instead of session for token without role Such a session triggers errors in module:may or other places since it is generally expected that a session must have a role.

mod_adhoc: Silence permission errors when listing commands Since throwing a pile of 'access denied', even at debug level, seems akin to calling wolf :) Cutting down on debug noise is also good. Passing a flag instead of using module:could seemed easier here.

mod_invites: Fix password reset invites Caused by roles changing from table|nil to always table in c2616274bef7

core.sessionmanager: Delay closing a replaced connection after replacement Closing the session invokes ondisconnect and session close logic, including mod_smacks hibernation and the timer that destroys the session after a timeout. By closing the connection after it has been detached from the sessions table it will no longer invoke the ondetach handler, which should prevent the above problem.

doap: Update reference for publishing of XEP-0478: Stream Limits Advertisement

mod_c2s,mod_s2s: Fix tag name for SLA (thanks mjk) The (still not published) XEP-xxxx: Stream Limits Advertisement uses the element <max-bytes/> to advertise the maximum octet size of top level stream elements. "size" was probably a leftover of an even earlier version of the (Proto)XEP.

Merge 0.12->trunk

mod_csi_simple: Disable revert-to-inactive timer when going to active mode This timer shouldn't kick in in the middle of active mode.

mod_csi_simple: Clear delayed active mode timer on disable It should not be there afterwards. Noticed that it seems to fire some time after resumption claiming that the queue size is nil, implying that it may hold a reference to an expired session somehow.

mod_admin_shell: Refactor 'cert' column Removes some dead code and hopefully simplifies a bit. There's a tree of possibilities with the two tri-state status properties, something like chain: * nil -- cert validation disabled? * invalid -- something wrong with the chain (including ee cert) * valid -- chain ok cert: * nil -- incomplete validation?? * invalid -- mismatched names or such * valid -- all good!

util.jsonschema: Fix UTF-8ness of 'minLength' and 'maxLength'

util.jsonschema: Implement 'minContains' and 'maxContains'

util.jsonschema: Add some comments wrt Lua-specifics

util.jsonschema: Implement 'luaPattern' - Lua variant of 'pattern' Like 'pattern' but uses Lua patterns instead of Regular Expressions, since only a subset of regex are also valid Lua patterns.

util.jsonschema: Implement 'luaPatternProperties' as Lua variant of 'patternProperties' Previous version of this patch used 'patternProperties' but that would only work with simpler ECMA-262 regular expressions are also valid Lua patterns.

util.jsonschema: Tweak description of disabled test This doesn't fail because of additionalProperties, looks more like some issue with recursive definitions and util.jsonpointer that I don't want feel like investigating now.

util.jsonschema: Enable passing IEEE 754 equality test

util.jsonschema: Fix NYI 'patternProperties' definition It's defined as an object mapping regex to schema, not a single schema

util.error: Add test for #1805 Checks that it doesn't fail on a stanza without <error> tag

Merge 0.12->trunk

util.error: Fix error on conversion of invalid error stanza, fix #1805 Error stanzas should have an <error> element, but if you pass a stanza without one to util.error.from_stanza() it triggers an attempt to index a nil value, which this patch avoids. In the conditional, it should be safe to assume error_tag is non-nil since condition can't have those values then.

mod_admin_shell: Fix display of remote cert status when expired etc Looks like autocomplete unhelpfully capitalized this word, but it's lowercase where it is set in mod_s2s_auth_certs

mod_tls: Drop request for client certificates on outgoing connections It is the other end who should request client certificates for these connections, we only need to send ours. Hopefully this was treated as a noop, so probably no harm in keeping it. But hey, spring cleaning? :)

mod_csi: Always advertise feature Was previously supposed to be conditionally advertised based on availability of a module handling the actual optimizations, which was removed in be9ac41f1619

mod_tokenauth: Fix parsing binary part of tokens Fixes parsing of tokens that happen to have a `;` in their secret part, otherwise it splits there and the later bit goes into the username and hitting the "Invalid token in storage" condition.

mod_tokenauth: Only check if expiry of expiring tokens Some tokens, e.g. OAuth2 refresh tokens, might not have their lifetime explicitly bounded here, but rather be bounded by the lifetime of something else, like the OAuth2 client. Open question: Would it be better to enforce a lifetime on all tokens?

mod_admin_shell: Use same wildcard matching in other s2s command Consistency is nice.

mod_admin_shell: Factor apart wildcard matching into function for reuse Applying this for s2s:close[all]() would also be nice.

mod_csi: Remove module status, doesn't work because of mod_smacks This was meant to warn in case you had only mod_csi without a logic handling module like mod_csi_simple by checking if anything hooked this event, however mod_smacks also hooks this event and so this isn't really a useful way of detecting this condition.

mod_http: Fix reliance on previous tostring() format of util.set a863e4237b91 unintentionally changed the format of HTTP CORS headers, which were apparently relying on the output of tostring(), which it shouldn't have. Explicitly serializing it this time.

util.human.io: Fix column width miscalculation Fixes that the more fixed width columns there are, the narrower the resulting table becomes. A right-aligned variable-width column at the last position should always be flush to the right side of the terminal.

util.human.io: Fix error with ellipsis to negative length Can happen if you resize the terminal too narrow that the space left for variable width columns end up negative.

tools/tb2err: Trim trailing whitespace

tools/tb2err: Rewrite prosody-modules paths to ../modules This assumes you have community modules in ../modules as I do

tools/tb2err: Add some example usage in a comment

tools/tb2err: Drop use of lua-any since it should run fine on any Lua Dependencies--; \o/

mod_admin_shell: Add config:set([host,] key, value) because why not We had config:get() but not this. > <MattJ> Yeah, why did we never implement that? Handy if you want to quickly try out settings without reloading the whole config.

mod_admin_shell: Allow wildcard matches like s2s:show("*.example.com") E.g. if you want to show connections to/from a domain, including its subdomains, this is handy.

util.sasl: Fix a singulars Thanks timeless, your mere existence inspires us to improve our spelling, tho this was more syntax.

Merge 0.12->trunk

util.argparse: Translate '-' to '_' in long option names for convenience A review of existing code suggests nothing will break. So, here we go...

prosodyctl: Fix using variable content in a format string This broke if the error message contained a format specified such as '%s'.

mod_http_file_share: use util.human.io.parse_duration Updated by Zash, the original patch by Jonas had put the duration parsing function in util.datetime but MattJ later did the same thing but differently in f4d7fe919969

mod_mam: port to use util.human.io.parse_duration Updated by Zash, the original patch by Jonas had put the duration parsing function in util.datetime but MattJ later did the same thing but differently in f4d7fe919969

util.human.io: Add parse_duration() method to parse a duration string Similar logic occurs throughout various modules in the codebase. We might even want a module:get_option_duration()??

mod_admin_shell: Allow "*" as substitute for 'nil' for easier CLI usage Since prosodyctl shell with additional arguments assumes the first two are a section:command() and any following arguments are strings, passing a bare 'nil' is not possible. In order to avoid delving into this rabbit hole, instead produce a token that alone is not really a legal JID for use as wildcard.

util.prosodyctl.shell: Fix sending terminal width with single argument E.g. when you do 'prosodyctl shell "s2s:show()"', this is the case that triggers, and it was missing the @width argument, causing confusion.

util.human.io: Coerce $COLUMNS to number os.getenv() returns a string but term_width() should return a number

util.prosodyctl.shell: Coerce terminal width to string (for util.stanza) Fixes invalid attribute value: expected string, got number

util.human.io: table: don't read $COLUMNS directly, just use term_width() ...which now reads $COLUMNS for us and does the right thing.

util.human.io: table: Return determined width as a second result This allows callers to adjust other things based on the width of the rows (such as header lines).

util.human.io: Prefer using the $COLUMNS environment variable if set (by readline) Feels like it should be faster.

util.prosodyctl.shell: Use new term_width() for width Kicks in if/when readline hasn't set $COLUMNS, e.g. when using the `prosodyctl shell command like this` form.

util.human.io: table: use term_width() to discover terminal width

util.human.io: Add term_width() method to discover the terminal width This is not standard POSIX, but apparently very widely supported. For reference: https://www.austingroupbugs.net/view.php?id=1053

mod_admin_shell: Make IP column thinner if IPv6 is disabled IPv6 addresses can be pretty long, so if they can be more compact, that's nice. But nobody would disable IPv6, would they?

mod_admin_shell: Make default column width 1 part These gets used for usernames, resources and other random session fields that don't have a column definition in `available_columns`

mod_admin_shell: Fix attempt to compare number with string Missed the # in 93c1590b5951

util.human.io: Pass the whole column definition to mapper function I forget why I wanted this, but it may allow doing things like pull settings from the column, especially when the mapper function is reused among many columns.

util.human.io: Allow defining per column ellipsis function As an alternative to doing it in the mapper function. Could be useful in cases where one may want to put the ellipsis in the middle or beginning instead of the start.

util.human.io: Pass expected width to mapper function In order to allow it to adjust its output to available space, apply its own ellipsis method or other compacting method.

mod_admin_shell: Dynamically size JIDs and hosts Reasoning: a hostname is one part, a JID is 3 parts.

mod_admin_shell: More dynamic widths calculations

mod_admin_shell: Calculate widths of columns from example values Harder to accidentally count wrong if Lua is doing the counting on a plausible input.

mod_admin_shell: Strip 'prosody:' prefix to allow narrower Role column

util.set: Change tostring format to {a, b, c} Makes it easier to make out where the set starts and ends in cases where it may get embedded and tostring()-ed in a log message. { } taken over from util.array for consistency with some other systems syntax for Sets, e.g. Python

util.array: Change tostring format to [a,b,c] Arrays in Lua do use { } but since __tostring is often user-facing it seems sensible to use [ ] instead for consistency with many other systems; as well as to allow the {a,b,c} formatting to be used by util.set without being confused with util.array.

util.human.io: Fix pattern to support fractional proportions

util.human.io: Support for dynamic "proportional" columns Instead of a percentage, this allows you to specify e.g. `width="[N]p"`, where a width="2p" will be twice the width of a width="1p" column. Compatibility with the old %-based widths is preserved, and percentages adding up to more than 100 are handled more gracefully.

mod_debug_reset: Remove now unused import of util.time (thanks luacheck)

mod_debug_reset: Don't delay operations until next tick For some unknown reason, this was required with the old mock util.time functions prior to 012d6e7b723a. After 012d6e7b723a, it breaks. So I'm happy to revert to not delaying anything. This makes tests pass again.

integration tests: Preserve unmocked time.monotonic() With monotonic() frozen, timers may fail to trigger. This caused problems after the new util.startup changes that moved the server-started event to a timer. The timer wouldn't trigger, the event didn't fire, and prosody would fail to daemonize. All the tests that depend on specific time behaviour are depending on wall clock time, so only mocking util.time.now() and os.time() fixes those.

mod_csi: Drop summary stats, doesn't work in normal module This method ends up going up for each collection and the :clear() method is only available to global modules (see e.g. mod_c2s), while regular per-host modules get scoped stats

mod_csi: Add metrics, covering changes and totals Motivation: Investigating clients that seem to forget to set CSI. Also, of course, MORE GRAPHS!

mod_tokenauth: Add API method to revoke a grant by id We probably want to refactor revoke_token() to use this one in the future.

Merge 0.12->trunk

mod_smacks: Replace existing watchdog when starting hibernation There shouldn't be one here but if there is, for some reason, it's better to close it than have it around to wake up and possibly try to destroy the session.

mod_smacks: Fix stray watchdog closing sessions Unsure exactly how this happens, but sometimes a watchdog appears to close a session that isn't hibernating, or hasn't hibernating long enough.

mod_adhoc: Remove "mod_" prefix from permission action name Other places doesn't have "mod_" there, why should it here?

util.fsm: New utility lib for finite state machines

util.set: Add missing remove function in Teal spec

net.http: Rename enum in Teal spec to avoid name clash