mod_storage_xmlarchive/README.markdown
author Kim Alvefur <zash@zash.se>
Thu, 28 Dec 2017 22:34:41 +0100
changeset 2859 7713cd4fff8f
parent 2858 687b19cad4f5
child 3438 a6722a35f35e
permissions -rw-r--r--
mod_storage_xmlarchive/README: Fix typo
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
1807
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
     1
---
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
     2
labels:
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
     3
- 'Stage-Beta'
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
     4
- 'Type-Storage'
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
     5
- ArchiveStorage
2363
33e85070dc60 mod_storage_xmlarchive: Expand the summary a little bit
Kim Alvefur <zash@zash.se>
parents: 2295
diff changeset
     6
summary: XML file based archive storage
2821
f052b62c653c mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents: 2819
diff changeset
     7
---
1786
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
     8
1807
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
     9
Introduction
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
    10
============
1786
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
    11
1807
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
    12
This module implements stanza archives using files, similar to the
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
    13
default "internal" storage.
1786
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
    14
1944
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    15
Configuration
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    16
=============
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    17
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    18
To use this with [mod\_mam] add this to your config:
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    19
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    20
``` lua
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    21
storage = {
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    22
    archive2 = "xmlarchive"
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    23
}
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    24
```
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    25
1973
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1944
diff changeset
    26
To use it with [mod\_mam\_muc] or [mod\_http\_muc\_log]:
1944
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    27
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    28
``` lua
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    29
storage = {
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    30
    muc_log = "xmlarchive"
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    31
}
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    32
```
1786
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
    33
1973
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1944
diff changeset
    34
Refer to [Prosodys data storage documentation][doc:storage] for more
e63dba236a2a mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents: 1944
diff changeset
    35
information.
1807
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
    36
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
    37
Note that this module does not implement the "keyval" storage method and
1944
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    38
can't be used by anything other than archives.
1786
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
    39
1807
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
    40
Compatibility
4d73a1a6ba68 Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents: 1786
diff changeset
    41
=============
1786
29f3d6b7ad16 Import wiki pages
Kim Alvefur <zash@zash.se>
parents:
diff changeset
    42
2295
1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents: 2052
diff changeset
    43
  ------ ---------------
1944
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    44
  0.10   Works
2295
1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents: 2052
diff changeset
    45
  0.9    Should work
1944
453eee191e18 mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents: 1807
diff changeset
    46
  0.8    Does not work
2295
1eeda34d668b mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents: 2052
diff changeset
    47
  ------ ---------------
2819
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    48
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    49
Conversion to or from internal storage
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    50
--------------------------------------
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    51
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    52
This module stores data in a way that overlaps with the more recent
2821
f052b62c653c mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents: 2819
diff changeset
    53
archive support in `mod_storage_internal`, meaning e.g. [mod_migrate]
2819
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    54
will not be able to cleanly convert to or from the `xmlarchive` format.
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    55
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    56
To mitigate this, an migration command has been added to
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    57
`mod_storage_xmlarchive`:
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    58
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    59
``` bash
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    60
prosodyctl mod_storage_xmlarchive convert $DIR internal $STORE $JID
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    61
```
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    62
2822
88474dd1af48 Various READMEs: s/eg/e.g.g/
Kim Alvefur <zash@zash.se>
parents: 2821
diff changeset
    63
Where `$DIR` is `to` or `from`, `$STORE` is e.g. `archive` or `archive2`
2819
d48d4d9ccae7 mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents: 2363
diff changeset
    64
for MAM and `muc_log` for MUC logs. Finally, `$JID` is the JID of the
2859
7713cd4fff8f mod_storage_xmlarchive/README: Fix typo
Kim Alvefur <zash@zash.se>
parents: 2858
diff changeset
    65
user or MUC room to be migrated, which can be repeated.
2858
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    66
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    67
Data structure
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    68
==============
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    69
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    70
Data is split in three kinds of files and messages are grouped by day.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    71
Prosodys `util.datamanager` is used, so all special characters in these
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    72
filenames are escaped and reside under `hostname/store` in Prosodys Data
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    73
directory, commonly `/var/lib/prosody`.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    74
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    75
`username.list`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    76
:   A list of dates in `YYYY-MM-DD` format.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    77
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    78
`username@YYYY-MM-DD.list`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    79
:   Index containing metadata for messages stored on that day.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    80
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    81
`username@YYYY-MM-DD.xml`
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    82
:   Messages in textual XML format, separated by newlines.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    83
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    84
This makes it fairly simple and fast to find messages by timestamp.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    85
Queries that are not time based, but limited to a specific contact may
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    86
be expensive as potentially the entire archive will be read.
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    87
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    88
Each archive ID is of the form `YYYY-MM-DD-random`, making lookups by
687b19cad4f5 mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents: 2822
diff changeset
    89
archive id just as simple as time based queries.