mod_s2s: Fix firing buffer drain events Fixes the same kind of issue as in 65563530375b but once and for all, while improving similarity between incoming and outgoing connections.

mod_smacks: Don't close resuming session when failed due to overflow

mod_smacks: Long overdue cleanup of resumption code, fixes some old TODOs

mod_admin_shell: Rename variable to avoid confusion with global function For luacheck, but it doesn't actually complain about this right now

mod_admin_shell: Fix output from user:roles() It used _G.print instead of the shell session print, which would silently write to stdout

Merge role-auth->trunk

CHANGES: Add role auth

mod_admin_shell: Ensure account has role before it is usable By creating the account first without a password it can't be used until the role has set. This is most important for restricted accounts, as a failure to set the role would lead to the account having more privileges than indented.

mod_auth_insecure: Store creation and update timestamps on account This ensures that the store is not empty in case no password is provided, so the underlying data storage won't consider the store empty.

mod_admin_shell: Update help for user:create to reflect singular role argument

mod_auth_internal_hashed: Allow creating disabled account without password Otherwise, create_user(username, nil) leads to the account being deleted.

mod_admin_shell: Update with new role management commands and help text

core.usermanager: Update argument name in authz fallback method It's not plural

core.usermanager: Remove obsolete authz fallback method

core.usermanager: Add missing methods to fallback authz provider

core.usermanager: Add scoped luacheck ignore rule to reduce clutter

mod_authz_internal: Expose convenience method to test if user can assume role

mod_authz_internal, and more: New iteration of role API These changes to the API (hopefully the last) introduce a cleaner separation between the user's primary (default) role, and their secondary (optional) roles. To keep the code sane and reduce complexity, a data migration is needed for people using stored roles in 0.12. This can be performed with prosodyctl mod_authz_internal migrate <host>

util.roles: Add Teal interface declaration

mod_admin_shell: Show session role in c2s:show

usermanager: Add back temporary is_admin to warn about deprecated API usage Goal: Introduce role-auth with minimal disruption is_admin() is unsafe in a system with per-session permissions, so it has been deprecated. Roll-out approach: 1) First, log a warning when is_admin() is used. It should continue to function normally, backed by the new role API. Nothing is really using per-session authz yet, so there is minimal security concern. The 'strict_deprecate_is_admin' global setting can be set to 'true' to force a hard failure of is_admin() attempts (it will log an error and always return false). 2) In some time (at least 1 week), but possibly longer depending on the number of affected deployments: switch 'strict_deprecate_is_admin' to 'true' by default. It can still be disabled for systems that need it. 3) Further in the future, before the next release, the option will be removed and is_admin() will be permanently disabled.

usermanager: Remove concept of global authz provider Rationale: - Removes a bunch of code! - We don't have many cases where an actor is not bound to one of our hosts - A notable exception is the admin shell, but if we ever attempt to lock those sessions down, there is a load of other work that also has to be done. And it's not clear if we would need a global authz provider for that anyway. - Removes an extra edge case from the necessary mental model for operators - Sessions that aren't bound to a host generally are anonymous or have an alternative auth model (such as by IP addres). - With the encapsulation now provided by util.roles, ad-hoc "detached roles" can still be created anyway by code that needs them.

usermanager: Fix method name of global authz provider (thanks Zash)

usermanager: Remove obsolete function from global authz provider

features: Add "permissions" feature for role-auth

usermanager: Handle local JIDs being passed to get/set_jid_role() There is no reasonable fallback for set_jid_role() because users may have multiple roles, so that's an error.

core.usermanager: Add missing stub authz methods to global authz provider Except, should we have a global authz provider at all?

moduleapi: Stricter type check for actor in permission check Non-table but truthy values would trigger "attempt to index a foo value" on the next line otherwise

moduleapi: Remove redundant expansion of ':' prefix in permission names

moduleapi: Distribute permissions set from global modules to all hosts Roles and permissions will always happen in the context of a host. Prevents error upon indexing since `hosts["*"] == nil`

mod_tokenauth: New API that better fits how modules are using token auth This also updates the module to the new role API, and improves support for scope/role selection (currently treated as the same thing, which they almost are).

mod_authz_internal: Use util.roles, some API changes and config support This commit was too awkward to split (hg record didn't like it), so: - Switch to the new util.roles lib to provide a consistent representation of a role object. - Change API method from get_role_info() to get_role_by_name() (touches sessionmanager and usermanager) - Change get_roles() to get_user_roles(), take a username instead of a JID This is more consistent with all other usermanager API methods. - Support configuration of custom roles and permissions via the config file (to be documented).

util.roles: Add new utility module to consolidate role objects and methods

usermanager, mod_auth_*: Add get_account_info() returning creation/update time This is useful for a number of things. For example, listing users that need to rotate their passwords after some event. It also provides a safer way for code to determine that a user password has changed without needing to set a handler for the password change event (which is a more fragile approach).

core.moduleapi: Expand permission name ':' prefix earlier Ensures it applies to the context as string case Somehow this fixes everything

core.moduleapi: Fixup method name `get_user_role()` did not exist anywhere else. MattJ said `get_user_default_role()` was indented

teal-src: update module.d.tl with new access control methods

Switch to a new role-based authorization framework, removing is_admin() We began moving away from simple "is this user an admin?" permission checks before 0.12, with the introduction of mod_authz_internal and the ability to dynamically change the roles of individual users. The approach in 0.12 still had various limitations however, and apart from the introduction of roles other than "admin" and the ability to pull that info from storage, not much actually changed. This new framework shakes things up a lot, though aims to maintain the same functionality and behaviour on the surface for a default Prosody configuration. That is, if you don't take advantage of any of the new features, you shouldn't notice any change. The biggest change visible to developers is that usermanager.is_admin() (and the auth provider is_admin() method) have been removed. Gone. Completely. Permission checks should now be performed using a new module API method: module:may(action_name, context) This method accepts an action name, followed by either a JID (string) or (preferably) a table containing 'origin'/'session' and 'stanza' fields (e.g. the standard object passed to most events). It will return true if the action should be permitted, or false/nil otherwise. Modules should no longer perform permission checks based on the role name. E.g. a lot of code previously checked if the user's role was prosody:admin before permitting some action. Since many roles might now exist with similar permissions, and the permissions of prosody:admin may be redefined dynamically, it is no longer suitable to use this method for permission checks. Use module:may(). If you start an action name with ':' (recommended) then the current module's name will automatically be used as a prefix. To define a new permission, use the new module API: module:default_permission(role_name, action_name) module:default_permissions(role_name, { action_name[, action_name...] }) This grants the specified role permission to execute the named action(s) by default. This may be overridden via other mechanisms external to your module. The built-in roles that developers should use are: - prosody:user (normal user) - prosody:admin (host admin) - prosody:operator (global admin) The new prosody:operator role is intended for server-wide actions (such as shutting down Prosody). Finally, all usage of is_admin() in modules has been fixed by this commit. Some of these changes were trickier than others, but no change is expected to break existing deployments. EXCEPT: mod_auth_ldap no longer supports the ldap_admin_filter option. It's very possible nobody is using this, but if someone is then we can later update it to pull roles from LDAP somehow.

mod_saslauth: Rename field from 'scope'->'role' The 'scope' term derives from OAuth, and represents a bundle of permissions. We're now setting on the term 'role' for a bundle of permissions. This change does not affect any public modules I'm aware of.

util.session: Add role management methods

net.connect: Clear TODO for Happy Eyeballs / RFC 8305, close #1246 Gotta have the DOAP references!

Merge 0.12->trunk

mod_admin_shell: Switch names for user role management commands user:roles() does not convey that this is the mutating command, it should have been called setroles from the start but wasn't due to lack of foresight. This has to accidentally removing roles when wanting to show them.

util.stanza: Add method for extracting a single attribute value Sometimes you only care about a single attribute, but the child tag itself may be optional, leading to needing `tag and tag.attr.foo` or `stanza:find("tag@foo")`. The `:find()` method is fairly complex, so avoiding it for this kind of simpler use case is a win.

mod_time: Remove obsolete XEP-0090 support Deprecated even before Prosody even started, obsolete for over a decade.

util.datetime: Update Teal interface description Integers were required before, now any number should work.

util.datetime: Remove a line No idea why the locals were declared on a line by itself. Perhaps line length considerations? But saving 6 characters in width by adding a whole line with 47 characters seems excessive. This is still within the 150 character limit set by .luacheckrc

mod_time: Return sub-second precision timestamps Because why not? Who even has this module enabled?

mod_storage_sql: Drop archive timestamp precision pending schema update The "when" column is an INTEGER which will probably be unhappy about storing higher precision timestamps, so we keep the older behavior for now.

mod_mam: Store archives with sub-second precision timestamps Changes sub-second part of example timestamp to .5 in order to avoid floating point issues. Some clients use timestamps when ordering messages which can lead to messages having the same timestamp ending up in the wrong order. It would be better to preserve the order messages are sent in, which is the order they were stored in.

util.datetime: Add support for sub-second precision timestamps Lua since 5.3 raises a fuss when time functions are handed a number with a fractional part and the underlying C functions are all based on integer seconds without support for more precision.

util.datetime: Fix argument order in tests The expected value goes first.

util.signal: Fix name conflict in Teal interface declaration

util.error: Use avoid name conflict in Teal interface declaration Naming things ... Thing or thing_t?

util.uuid: Fix syntax of Teal interface declaration file

util.timer: Add Teal interface description

util.termcolours: Add Teal interface description

util.queue: Add Teal interface description

util.logger: Add Teal interface description

util.bitcompat: Add Teal type specification

util.struct: Add Teal interface description file

util.table: Add move() to Teal interface description file

util.set: Add teal type declaration file

util.serialization: Add Teal type specification

util.dataforms: Add missing :data() to Teal definition

util.dataforms: Restructure Teal definition file The PR has been merged and there's no reason not to have nested records and other definitions.

util.human.io: Add Teal interface definition

util.promise: Add Teal interface specification file

teal: add stub util.array teal defs

net.server: Add teal description file

net.http: Add teal description files

util.human.units: Specify enum argument to format()

core.storagemanager: Convert old Typed Lua description file into Teal Still only a type definition. Typed Lua is no longer maintained. Teal is currently an active project.

util.hex: Update Teal spec for function rename in a0ff5c438e9d

doap: Update XEP versions for which no code changes appear needed XEP-0004: Partial forms are handled XEP-0045: We're already strict with GC 1.0 XEP-0060: Change in semantics wrt 'pubsub#type', but not in code XEP-0115: No protocol change XEP-0138: Specification moved to Obsolete XEP-0163: Editorial only change XEP-0215: Minor schema change XEP-0280: Editorial change XEP-0297: Had the wrong version number XEP-0106: Note missing piece for version 1.1 XEP-0313: Editorial change XEP-0363: Editorial clarification, no code change required XEP-0380: Registry additions, no code change needed XEP-0384: Not directly supported, only here because people will ask otherwise XEP-0445: Broken out of XEP-0401

various: Update IETF RFC URLs for tools.ietf.org transition See https://www.ietf.org/blog/finalizing-ietf-tools-transition/ Already done in various other places.

mod_admin_shell: Remove obsolete module:load() argument from 0.8 time This 'config' argument was removed without explanation in d8dbf569766c

mod_tls: Record STARTTLS state so it can be shown in Shell This field can be viewed using s2s:show(nil, "... starttls") even without any special support in mod_admin_shell, which can be added later to make it nicer. One can then assume that a TLS connection with an empty / nil starttls field means Direct TLS.

net.resolvers.basic: Add opt-out argument for DNSSEC security status This makes explicit which lookups can accept an unsigned response. Insecure (unsigned, as before DNSSEC) A and AAAA records can be used as security would come from TLS, but an insecure TLSA record is worthless.

Merge 0.12->trunk

mod_storage_sql: Fix summary API with Postgres (fixes #1766) The ORDER BY and LIMIT clauses are not needed and don't even make much sense. This part was most likely a leftover from the :find method. Tested with sqlite and postgres 14

storage tests: Add test for the archive:summary API Passes with memory, internal, sqlite Fails with postgres as in #1766

mod_http_files: Log warning about legacy modules using mod_http_files It is time. Most community modules should have been adjusted to work with the new (net.http.files) way. At some point this usage should be prevented. Related to #1765

util.sasl.scram: Add 'tls-exporter' as recognised channel binding method The last missing piece of #1760, otherwise SCRAM-SHA-*-PLUS is not actually advertised.

Merge 0.12->trunk

mod_saslauth: Implement RFC 9266 'tls-exporter' channel binding (#1760) Brings back SCRAM-SHA-*-PLUS from its hiatus brought on by the earlier channel binding method being undefined for TLS 1.3, and the increasing deployment of TLS 1.3. See 1bfd238e05ad and #1542 Requires future version of LuaSec, once support for this key material export method is merged. See https://github.com/brunoos/luasec/pull/187

mod_bookmarks: Reduce error about not having bookmarks to debug (thanks tom) This is happens if the account is new and doesn't have any bookmarks yet, which is not a problem. Rarely seen since most clients currently use the older version of XEP-0084 stored in XEP-0049 rather than in PEP, but at least one (Converse.js )does. One scenario in which this would show up often is with Converse.js as a guest chat using anonymous authentication, where all "accounts" would always be new and not have any bookmarks. This scenario probably does not need to have mod_bookmarks at all, but if enabled globally it would likely become loaded onto the VirtualHost unless explicitly disabled.

mod_storage_sql: Fix bypass of load procedure under prosodyctl There's no 'prosody.prosodyctl' property other than this one, introduced in 6216743c188c in 2015. Guessing that the intent was to skip this when running as a prosodyctl command. The module.command code does its own version of this initialization, so this seems likely. Thanks raja for noticing

util.table: Fix inaccurate comment Probably a duplicate of the comment next to Lmove, recorded by mistake Lpack can probably be removed at some point in the near future once we are confident it is not used anywhere.

compat: Use table.pack (there since Lua 5.2) over our util.table Added in d278a770eddc avoid having to deal with its absence in Lua 5.1. No longer needed when Lua 5.1 support is dropped.

compat: Remove handling of Lua 5.1 location of 'unpack' function

Merge 0.12->trunk

luacheck: Set expected globals to Lua 5.4 + compat Requires luacheck 0.25.0

core.s2smanager: Don't remove unrelated session on close of bidi session Normally with bidi, any outgoing connection should be the same as the incoming, hence when closing a bidi connection it should be removed as a route to the remote server. However it is not guaranteed, a remote bidi-capable server might have decided to open a new connection for some reason. This can lead to a situation where there are two bidi connections, and the s2sout route is a locally initiated s2sout connection. In this case, such a s2sout connection should be kept. Noticed in a rare case where bidi has just been enabled on a running server, and something establishes new connections immediately when a connection is closed.

Merge 0.12->trunk

luacheck: Shut up (backports 3caff1f93520, ignores module deleted in trunk)

Merge 0.12->trunk

Backport 875f73ead4e8 8e4033213c62 to deal with luacheck 0.26

Merge 0.12->trunk

util.datamapper: Improve handling of schemas with non-obvious "type" The JSON Schema specification says that schemas are objects or booleans, and that the 'type' property is optional and can be an array. This module previously allowed bare type names as schemas and did not really handle booleans. It now handles missing 'type' properties and boolean 'true' as a schema. Objects and arrays are guessed based on the presence of 'properties' or 'items' field.

util.jsonschema: Fix validation to not assume presence of "type" field MattJ reported a curious issue where validation did not work as expected. Primarily that the "type" field was expected to be mandatory, and thus leaving it out would result in no checks being performed. This was likely caused by misreading during initial development. Spent some time testing against https://github.com/json-schema-org/JSON-Schema-Test-Suite.git and discovered a multitude of issues, far too many to bother splitting into separate commits. More than half of them fail. Many because of features not implemented, which have been marked NYI. For example, some require deep comparisons e.g. when objects or arrays are present in enums fields. Some because of quirks with how Lua differs from JavaScript, e.g. no distinct array or object types. Tests involving fractional floating point numbers. We're definitely not going to follow references to remote resources. Or deal with UTF-16 sillyness. One test asserted that 1.0 is an integer, where Lua 5.3+ will disagree.

executables: Reject Lua 5.1 early Prevents attempting to load libraries that may no longer be found and crashing with a traceback. Platforms like Debian where multiple Lua versions can be installed at the same time and 'lua' pointing to one of the installed interpreters via symlinks, there's the possibility that prosody/prosodyctl may be invoked with Lua 5.1, which will no longer have any of the rest of Prosody libraries available to be require(), and thus would immediately fail with an unfriendly traceback. Checking and aborting early with a friendlier message and reference to more information is better. Part of #1600

CHANGES: Lua 5.1 support removed (closes #1600)

util.envload: Remove Lua 5.1 method Part of #1600 Is this module even needed anymore?

util-src: Remove Lua 5.1 compat macros Part of #1600

mod_storage_sql: Remove Lua 5.1 compatibility hack Part of #1600

util: Remove various Lua 5.1 compatibility hacks Part of #1600

util.dependencies: Reject Lua 5.1, Lua 5.2 or later is now required (see #1600)

tests: Remove special-casing of Lua 5.1 Part of #1600

configure: No longer accept Lua 5.1

util.dependencies: Deprecate support for Lua 5.1, this is your final warning

util.hashes: Revert to HMAC() convenience function Reverts some of 1e41dd0f8353 Seems HMAC() isn't deprecated after all? Must have been at some point according to #1589 Twice as fast for some reason.

util.hashes: Remove unused constants

util.hashes: Remove unused struct Unused since 9f1c5ae8d70b

util.hashes: Return OpenSSL error messages on failure With luck, might contain more details than just "failed"

util.hashes: Add SHA3 bindings

util.hashes: Bind BLAKE2 algoritms supported by OpenSSL

util.hashes: Refactor PBKDF2 to deduplicate code

util.hashes: Expose sha224 and sha384 HMAC functions For completeness and consistency with set of plain hash functions

util.hashes: Refactor HMAC bindings (fixes #1589) HMAC() is deprecated As with the regular hash functions, macros like this make it awkward to apply static analysis and code formatting.