author | Kim Alvefur <zash@zash.se> |
Thu, 28 Dec 2017 22:34:41 +0100 | |
changeset 2859 | 7713cd4fff8f |
parent 2858 | 687b19cad4f5 |
child 3438 | a6722a35f35e |
permissions | -rw-r--r-- |
1807
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
1 |
--- |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
2 |
labels: |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
3 |
- 'Stage-Beta' |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
4 |
- 'Type-Storage' |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
5 |
- ArchiveStorage |
2363
33e85070dc60
mod_storage_xmlarchive: Expand the summary a little bit
Kim Alvefur <zash@zash.se>
parents:
2295
diff
changeset
|
6 |
summary: XML file based archive storage |
2821
f052b62c653c
mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents:
2819
diff
changeset
|
7 |
--- |
1786 | 8 |
|
1807
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
9 |
Introduction |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
10 |
============ |
1786 | 11 |
|
1807
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
12 |
This module implements stanza archives using files, similar to the |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
13 |
default "internal" storage. |
1786 | 14 |
|
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
15 |
Configuration |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
16 |
============= |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
17 |
|
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
18 |
To use this with [mod\_mam] add this to your config: |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
19 |
|
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
20 |
``` lua |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
21 |
storage = { |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
22 |
archive2 = "xmlarchive" |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
23 |
} |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
24 |
``` |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
25 |
|
1973
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1944
diff
changeset
|
26 |
To use it with [mod\_mam\_muc] or [mod\_http\_muc\_log]: |
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
27 |
|
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
28 |
``` lua |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
29 |
storage = { |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
30 |
muc_log = "xmlarchive" |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
31 |
} |
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
32 |
``` |
1786 | 33 |
|
1973
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1944
diff
changeset
|
34 |
Refer to [Prosodys data storage documentation][doc:storage] for more |
e63dba236a2a
mod_storage_xmlarchive: Use datamanager.append_raw (had that code duplicated here)
Kim Alvefur <zash@zash.se>
parents:
1944
diff
changeset
|
35 |
information. |
1807
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
36 |
|
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
37 |
Note that this module does not implement the "keyval" storage method and |
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
38 |
can't be used by anything other than archives. |
1786 | 39 |
|
1807
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
40 |
Compatibility |
4d73a1a6ba68
Convert all wiki pages to Markdown
Kim Alvefur <zash@zash.se>
parents:
1786
diff
changeset
|
41 |
============= |
1786 | 42 |
|
2295
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2052
diff
changeset
|
43 |
------ --------------- |
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
44 |
0.10 Works |
2295
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2052
diff
changeset
|
45 |
0.9 Should work |
1944
453eee191e18
mod_storage_xmlarchive/README: Add example configuration
Kim Alvefur <zash@zash.se>
parents:
1807
diff
changeset
|
46 |
0.8 Does not work |
2295
1eeda34d668b
mod_storage_xmlarchive/README: Should work in 0.9 now
Kim Alvefur <zash@zash.se>
parents:
2052
diff
changeset
|
47 |
------ --------------- |
2819
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
48 |
|
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
49 |
Conversion to or from internal storage |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
50 |
-------------------------------------- |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
51 |
|
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
52 |
This module stores data in a way that overlaps with the more recent |
2821
f052b62c653c
mod_storage_xmlarchive/README: Minor tweak
Kim Alvefur <zash@zash.se>
parents:
2819
diff
changeset
|
53 |
archive support in `mod_storage_internal`, meaning e.g. [mod_migrate] |
2819
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
54 |
will not be able to cleanly convert to or from the `xmlarchive` format. |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
55 |
|
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
56 |
To mitigate this, an migration command has been added to |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
57 |
`mod_storage_xmlarchive`: |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
58 |
|
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
59 |
``` bash |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
60 |
prosodyctl mod_storage_xmlarchive convert $DIR internal $STORE $JID |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
61 |
``` |
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
62 |
|
2822 | 63 |
Where `$DIR` is `to` or `from`, `$STORE` is e.g. `archive` or `archive2` |
2819
d48d4d9ccae7
mod_storage_xmlarchive: Add a prosodyctl command for migrating to/from the internal storage format
Kim Alvefur <zash@zash.se>
parents:
2363
diff
changeset
|
64 |
for MAM and `muc_log` for MUC logs. Finally, `$JID` is the JID of the |
2859
7713cd4fff8f
mod_storage_xmlarchive/README: Fix typo
Kim Alvefur <zash@zash.se>
parents:
2858
diff
changeset
|
65 |
user or MUC room to be migrated, which can be repeated. |
2858
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
66 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
67 |
Data structure |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
68 |
============== |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
69 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
70 |
Data is split in three kinds of files and messages are grouped by day. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
71 |
Prosodys `util.datamanager` is used, so all special characters in these |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
72 |
filenames are escaped and reside under `hostname/store` in Prosodys Data |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
73 |
directory, commonly `/var/lib/prosody`. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
74 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
75 |
`username.list` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
76 |
: A list of dates in `YYYY-MM-DD` format. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
77 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
78 |
`username@YYYY-MM-DD.list` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
79 |
: Index containing metadata for messages stored on that day. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
80 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
81 |
`username@YYYY-MM-DD.xml` |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
82 |
: Messages in textual XML format, separated by newlines. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
83 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
84 |
This makes it fairly simple and fast to find messages by timestamp. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
85 |
Queries that are not time based, but limited to a specific contact may |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
86 |
be expensive as potentially the entire archive will be read. |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
87 |
|
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
88 |
Each archive ID is of the form `YYYY-MM-DD-random`, making lookups by |
687b19cad4f5
mod_storage_xmlarchive/README: Add description of how data is stored
Kim Alvefur <zash@zash.se>
parents:
2822
diff
changeset
|
89 |
archive id just as simple as time based queries. |