author | Kim Alvefur <zash@zash.se> |
Sat, 25 Jan 2020 01:31:49 +0100 | |
changeset 3861 | 8752e5b5dd08 |
parent 3464 | f25b0623b276 |
child 3991 | dddcf094af5c |
permissions | -rw-r--r-- |
1807
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
1 |
--- |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
2 |
labels: |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
3 |
- 'Stage-Beta' |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
4 |
- 'Type-Storage' |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
5 |
- ArchiveStorage |
2363
33e85070dc60
mod_storage_xmlarchive: Expand the summary a little bit
Kim Alvefur <zash@zash.se>
parents:
2295
diff
changeset
|
6 |
summary: XML file based archive storage |
2821
f052b62c653c
mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents:
2819
diff
changeset
|
7 |
--- |
1786 | 8 |
|
1807
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
9 |
Introduction |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
10 |
============ |
1786 | 11 |
|
1807
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
12 |
This module implements stanza archives using files, similar to the |
3438
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
13 |
default "internal" storage. Unlike "internal", it saves messages in two |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
14 |
files per day (and per user), one containing metadata and one containing |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
15 |
the actual messages in XML format (hence the name). |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
16 |
|
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
17 |
Splitting data per day improves performance for larger archives as it |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
18 |
does not have to look through data from other days. |
1786 | 19 |
|
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
20 |
Configuration |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
21 |
============= |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
22 |
|
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
23 |
To use this with [mod\_mam] add this to your config: |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
24 |
|
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
25 |
``` lua |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
26 |
storage = { |
3455
7b97abf26612
mod_storage_xmlarchive/README: Change example to match Prosodys mod_mam
Kim Alvefur <zash@zash.se>
parents:
3438
diff
changeset
|
27 |
archive = "xmlarchive" |
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
28 |
} |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
29 |
``` |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
30 |
|
1973
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1944
diff
changeset
|
31 |
To use it with [mod\_mam\_muc] or [mod\_http\_muc\_log]: |
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
32 |
|
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
33 |
``` lua |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
34 |
storage = { |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
35 |
muc_log = "xmlarchive" |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
36 |
} |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
37 |
``` |
1786 | 38 |
|
1973
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1944
diff
changeset
|
39 |
Refer to [Prosodys data storage documentation][doc:storage] for more |
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1944
diff
changeset
|
40 |
information. |
1807
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
41 |
|
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
42 |
Note that this module does not implement the "keyval" storage method and |
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
43 |
can't be used by anything other than archives. |
1786 | 44 |
|
1807
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
45 |
Compatibility |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
46 |
============= |
1786 | 47 |
|
2295
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2052
diff
changeset
|
48 |
------ --------------- |
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
49 |
0.10 Works |
2295
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2052
diff
changeset
|
50 |
0.9 Should work |
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
51 |
0.8 Does not work |
2295
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2052
diff
changeset
|
52 |
------ --------------- |
2819
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
53 |
|
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
54 |
Conversion to or from internal storage |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
55 |
-------------------------------------- |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
56 |
|
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
57 |
This module stores data in a way that overlaps with the more recent |
2821
f052b62c653c
mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents:
2819
diff
changeset
|
58 |
archive support in `mod_storage_internal`, meaning e.g. [mod_migrate] |
2819
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
59 |
will not be able to cleanly convert to or from the `xmlarchive` format. |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
60 |
|
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
61 |
To mitigate this, an migration command has been added to |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
62 |
`mod_storage_xmlarchive`: |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
63 |
|
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
64 |
``` bash |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
65 |
prosodyctl mod_storage_xmlarchive convert $DIR internal $STORE $JID |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
66 |
``` |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
67 |
|
2822 | 68 |
Where `$DIR` is `to` or `from`, `$STORE` is e.g. `archive` or `archive2` |
2819
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
69 |
for MAM and `muc_log` for MUC logs. Finally, `$JID` is the JID of the |
2859
7713cd4fff8f
mod_storage_xmlarchive/README: Fix typo
Kim Alvefur <zash@zash.se>
parents:
2858
diff
changeset
|
70 |
user or MUC room to be migrated, which can be repeated. |
2858
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
71 |
|
3463
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3455
diff
changeset
|
72 |
::: {.alert .alert-danger} |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3455
diff
changeset
|
73 |
Since this is a destructive command, don't forget to backup your data |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3455
diff
changeset
|
74 |
first. |
3464
f25b0623b276
mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents:
3463
diff
changeset
|
75 |
|
f25b0623b276
mod_storage_xmlarchive/README: And don't have prosody running at the same time
Kim Alvefur <zash@zash.se>
parents:
3463
diff
changeset
|
76 |
Prosody should *not* be running while converting data. |
3463
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3455
diff
changeset
|
77 |
::: |
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3455
diff
changeset
|
78 |
|
471855552f49
mod_storage_xmlarchive/README: Add reminder about backups
Kim Alvefur <zash@zash.se>
parents:
3455
diff
changeset
|
79 |
|
2858
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
80 |
Data structure |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
81 |
============== |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
82 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
83 |
Data is split in three kinds of files and messages are grouped by day. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
84 |
Prosodys `util.datamanager` is used, so all special characters in these |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
85 |
filenames are escaped and reside under `hostname/store` in Prosodys Data |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
86 |
directory, commonly `/var/lib/prosody`. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
87 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
88 |
`username.list` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
89 |
: A list of dates in `YYYY-MM-DD` format. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
90 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
91 |
`username@YYYY-MM-DD.list` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
92 |
: Index containing metadata for messages stored on that day. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
93 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
94 |
`username@YYYY-MM-DD.xml` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
95 |
: Messages in textual XML format, separated by newlines. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
96 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
97 |
This makes it fairly simple and fast to find messages by timestamp. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
98 |
Queries that are not time based, but limited to a specific contact may |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
99 |
be expensive as potentially the entire archive will be read. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
100 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
101 |
Each archive ID is of the form `YYYY-MM-DD-random`, making lookups by |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
102 |
archive id just as simple as time based queries. |
3438
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
103 |
|
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
104 |
## Limitations |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
105 |
|
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
106 |
- Only XML stanzas can be stored. |
a6722a35f35e
mod_storage_xmlarchive: Expand description in README
Kim Alvefur <zash@zash.se>
parents:
2859
diff
changeset
|
107 |
- The deletion method only supports removing entire days at a time. |