Compare commits

...
28 Commits
Author SHA1 Message Date
Paul Makles 4c00a7dfb7 chore: strip dotenv (unmaintained) from project
It is no longer needed as configuration is loaded via. TOML or direct env if appropriate.
2024-11-27 17:05:11 +00:00
Paul Makles b9ae333b02 fix(bonfire): make error handling consistent 2024-11-11 12:31:34 +00:00
Paul Makles bf0fc504a9 chore(vscode): select nix env 2024-11-11 12:31:19 +00:00
Paul Makles 705e517871 fix: db migration for webhooks does not consider missing channels
fixes #378
2024-10-31 20:31:09 +00:00
Paul Makles 6f99ac2160 chore: bump little_exif debug code 2024-10-28 15:38:07 +00:00
Paul Makles 2fcc714546 chore: add helper script for running everything at once 2024-10-28 14:15:48 +00:00
Paul Makles 1e72f7bc77 chore: copy createbuckets config from self-hosted 2024-10-28 13:45:37 +00:00
Paul Makles af0d24c7c6 fix: don't use failed file hashes
fixes #377
2024-10-28 13:40:40 +00:00
Paul Makles ab58177dfa feat: add support for path style buckets 2024-10-27 23:28:28 +00:00
Paul Makles df07426019 chore: bump version to 0.7.19 2024-10-24 17:48:11 +01:00
Paul Makles c30d9a2620 chore(bonfire): fire error event with more information 2024-10-24 17:47:17 +01:00
Paul Makles 7b44317705 docs: add new features page 2024-10-24 17:45:10 +01:00
Paul Makles 876068a37e fix: populate empty vector if clear field is missing
fixes #367
2024-10-24 17:14:30 +01:00
IAmTomahawkx 397b9878e1 fix: temp fix for spam attack via notification bug abuse, but git actually commits the changes this time 2024-10-06 13:22:13 -07:00
Paul Makles d9deadc65a fix: temp fix for spam attack via notification bug abuse 2024-10-06 02:34:47 -07:00
Paul Makles 58d3c5cc2e fix(services/january): remove image if video present and hence fix logic error
refactor(services/january): throw an error if embed fails to generate
2024-10-02 16:08:24 +01:00
Paul Makles 520fb02fb6 fix(services/january): support svg for embed generation 2024-10-02 15:20:35 +01:00
Paul Makles 2cb12a3d59 fix: do not handle error early 2024-10-02 15:05:57 +01:00
Paul Makles 18888eae5f fix: mangled symbol 2024-10-02 14:13:32 +01:00
Paul Makles f31020fb6e refactor(core): add ImageProcessingFailed error 2024-10-02 14:12:26 +01:00
Paul Makles bb202079e0 fix(services/january): put html parsing into block to prevent issues with futures 2024-10-02 14:09:41 +01:00
Paul Makles bb6bcda8bd fix(services/january): reddit embeds by spoofing as discord / mapping to old reddit
closes #360
2024-10-02 12:56:32 +01:00
Paul Makles 68099bd2b7 chore: bump version & close issues
closes #366
closes #364
closes #359
closes #356
closes #354
closes #351
closes #348
closes #345
2024-10-02 12:39:32 +01:00
Paul Makles a8db1cb40d feat(core/files): SVG rendering for thumbnails 2024-10-02 12:36:56 +01:00
Paul Makles efa7ba78ed feat(core/files): support for jixel decoding
closes #342
2024-10-02 12:20:14 +01:00
Paul Makles 25fc692dc1 chore(core/models): truncate the rest of the embed fields 2024-10-02 11:35:37 +01:00
Paul Makles 530d68fe89 feat(services/autumn): add accept-language header
closes #358
2024-10-02 11:32:55 +01:00
Paul Makles afd8c906ba fix(services/january): YouTube fallback 2024-10-02 11:31:55 +01:00
39 changed files with 772 additions and 303 deletions
+3 -2
View File
@@ -1,5 +1,6 @@
{
"editor.formatOnSave": true,
"rust-analyzer.checkOnSave.command": "clippy",
"nixEnvSelector.suggestion": false
}
"nixEnvSelector.suggestion": false,
"nixEnvSelector.nixFile": "${workspaceFolder}/default.nix"
}
Generated
+268 -43
View File
@@ -199,9 +199,9 @@ checksum = "a4c527152e37cf757a3f78aae5a06fbeefdb07ccc535c980a3208ee3060dd544"
[[package]]
name = "arrayvec"
version = "0.7.2"
version = "0.7.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8da52d66c7071e2e3fa2a1e5c6d088fec47b593032b254f5e980de8ea54454d6"
checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50"
[[package]]
name = "async-attributes"
@@ -1531,6 +1531,15 @@ version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5827cebf4670468b8772dd191856768aedcb1b0278a04f989f7766351917b9dc"
[[package]]
name = "core_maths"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e3b02505ccb8c50b0aa21ace0fc08c3e53adebd4e58caa18a36152803c7709a3"
dependencies = [
"libm",
]
[[package]]
name = "cpufeatures"
version = "0.2.2"
@@ -1676,16 +1685,6 @@ version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f3b7eb4404b8195a9abb6356f4ac07d8ba267045c8d6d220ac4dc992e6cc75df"
[[package]]
name = "ctor"
version = "0.1.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f877be4f7c9f246b183111634f75baa039715e3f46ce860677d3b19a69fb229c"
dependencies = [
"quote 1.0.37",
"syn 1.0.107",
]
[[package]]
name = "ctr"
version = "0.7.0"
@@ -1858,6 +1857,12 @@ version = "2.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ee2393c4a91429dffb4bedf19f4d6abf27d8a732c8ce4980305d782e5426d57"
[[package]]
name = "data-url"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c297a1c74b71ae29df00c3e22dd9534821d60eb9af5a0192823fa2acea70c2a"
[[package]]
name = "deadqueue"
version = "0.2.4"
@@ -1999,12 +2004,6 @@ version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0688c2a7f92e427f44895cd63841bff7b29f8d7a1648b9e7e07a4a365b2e1257"
[[package]]
name = "dotenv"
version = "0.15.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77c90badedccf4105eca100756a0b1289e191f6fcbdadd3cee1d2f614f97da8f"
[[package]]
name = "dtoa"
version = "1.0.9"
@@ -2354,6 +2353,29 @@ version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
[[package]]
name = "fontconfig-parser"
version = "0.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c1fcfcd44ca6e90c921fee9fa665d530b21ef1327a4c1a6c5250ea44b776ada7"
dependencies = [
"roxmltree",
]
[[package]]
name = "fontdb"
version = "0.22.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a3a6f9af55fb97ad673fb7a69533eb2f967648a06fa21f8c9bb2cd6d33975716"
dependencies = [
"fontconfig-parser",
"log",
"memmap2",
"slotmap",
"tinyvec",
"ttf-parser",
]
[[package]]
name = "foreign-types"
version = "0.3.2"
@@ -3595,6 +3617,16 @@ dependencies = [
"mutate_once",
]
[[package]]
name = "kurbo"
version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "89234b2cc610a7dd927ebde6b41dd1a5d4214cffaef4cf1fb2195d592f92518f"
dependencies = [
"arrayvec",
"smallvec",
]
[[package]]
name = "kv-log-macro"
version = "1.0.7"
@@ -3759,11 +3791,10 @@ dependencies = [
[[package]]
name = "log"
version = "0.4.17"
version = "0.4.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "abb12e687cfb44aa40f41fc3978ef76448f9b6038cad6aef4259d3c095a2382e"
checksum = "a7a70ba024b9dc04c27ea2f0c0548feb474ec5c54bba33a7f72f873a39d07b24"
dependencies = [
"cfg-if",
"value-bag",
]
@@ -3911,6 +3942,15 @@ version = "2.7.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78ca9ab1a0babb1e7d5695e3530886289c18cf2f87ec19a575a0abdce112e3a3"
[[package]]
name = "memmap2"
version = "0.9.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fd3f7eed9d3848f8b98834af67102b720745c4ec028fcd0aa0239277e7de374f"
dependencies = [
"libc",
]
[[package]]
name = "metrics"
version = "0.18.1"
@@ -4744,7 +4784,7 @@ version = "0.10.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6796ad771acdc0123d2a88dc428b5e38ef24456743ddb1744ed628f9815c096"
dependencies = [
"siphasher",
"siphasher 0.3.11",
]
[[package]]
@@ -4753,9 +4793,15 @@ version = "0.11.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "90fcb95eef784c2ac79119d1dd819e162b5da872ce6f3c3abe1e8ca1c082f72b"
dependencies = [
"siphasher",
"siphasher 0.3.11",
]
[[package]]
name = "pico-args"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5be167a7af36ee22fe3115051bc51f6e6c7054c9348e28deb4f49bd6f705a315"
[[package]]
name = "pin-project"
version = "1.0.10"
@@ -5509,9 +5555,26 @@ dependencies = [
"quick-error 1.2.3",
]
[[package]]
name = "resvg"
version = "0.44.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4a325d5e8d1cebddd070b13f44cec8071594ab67d1012797c121f27a669b7958"
dependencies = [
"gif",
"image-webp",
"log",
"pico-args",
"rgb",
"svgtypes",
"tiny-skia",
"usvg",
"zune-jpeg",
]
[[package]]
name = "revolt-autumn"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"axum",
"axum-macros",
@@ -5548,7 +5611,7 @@ dependencies = [
[[package]]
name = "revolt-bonfire"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"async-channel 2.3.1",
"async-std",
@@ -5578,12 +5641,11 @@ dependencies = [
[[package]]
name = "revolt-config"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"async-std",
"cached",
"config",
"dotenv",
"futures-locks",
"log",
"once_cell",
@@ -5595,7 +5657,7 @@ dependencies = [
[[package]]
name = "revolt-database"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"async-lock 2.8.0",
"async-recursion",
@@ -5643,7 +5705,7 @@ dependencies = [
[[package]]
name = "revolt-delta"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"async-channel 1.6.1",
"async-std",
@@ -5651,7 +5713,6 @@ dependencies = [
"bitfield",
"chrono",
"dashmap",
"dotenv",
"env_logger",
"futures",
"impl_ops",
@@ -5690,7 +5751,7 @@ dependencies = [
[[package]]
name = "revolt-files"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"aes-gcm 0.10.3",
"aws-config",
@@ -5700,17 +5761,20 @@ dependencies = [
"image",
"imagesize",
"jxl-oxide",
"resvg",
"revolt-config",
"revolt-result",
"tempfile",
"tiny-skia",
"tracing",
"typenum",
"usvg",
"webp",
]
[[package]]
name = "revolt-january"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"async-recursion",
"axum",
@@ -5738,7 +5802,7 @@ dependencies = [
[[package]]
name = "revolt-models"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"indexmap 1.9.3",
"iso8601-timestamp 0.2.11",
@@ -5756,7 +5820,7 @@ dependencies = [
[[package]]
name = "revolt-nodejs-bindings"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"async-std",
"neon",
@@ -5769,7 +5833,7 @@ dependencies = [
[[package]]
name = "revolt-permissions"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"async-std",
"async-trait",
@@ -5784,7 +5848,7 @@ dependencies = [
[[package]]
name = "revolt-presence"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"async-std",
"log",
@@ -5795,7 +5859,7 @@ dependencies = [
[[package]]
name = "revolt-result"
version = "0.7.17"
version = "0.7.19"
dependencies = [
"axum",
"revolt_okapi",
@@ -6106,6 +6170,12 @@ dependencies = [
"serde",
]
[[package]]
name = "roxmltree"
version = "0.20.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6c20b6793b5c2fa6553b250154b78d6d0db37e72700ae35fad9387a46f487c97"
[[package]]
name = "rsa"
version = "0.7.2"
@@ -6307,6 +6377,24 @@ version = "1.0.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5583e89e108996506031660fe09baa5011b9dd0341b89029313006d1fb508d70"
[[package]]
name = "rustybuzz"
version = "0.18.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c85d1ccd519e61834798eb52c4e886e8c2d7d698dd3d6ce0b1b47eb8557f1181"
dependencies = [
"bitflags 2.6.0",
"bytemuck",
"core_maths",
"log",
"smallvec",
"ttf-parser",
"unicode-bidi-mirroring",
"unicode-ccc",
"unicode-properties",
"unicode-script",
]
[[package]]
name = "ryu"
version = "1.0.10"
@@ -6824,18 +6912,42 @@ version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f27f6278552951f1f2b8cf9da965d10969b2efdea95a6ec47987ab46edfe263a"
[[package]]
name = "simplecss"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a11be7c62927d9427e9f40f3444d5499d868648e2edbc4e2116de69e7ec0e89d"
dependencies = [
"log",
]
[[package]]
name = "siphasher"
version = "0.3.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "38b58827f4464d87d377d175e90bf58eb00fd8716ff0a62f80356b5e61555d0d"
[[package]]
name = "siphasher"
version = "1.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "56199f7ddabf13fe5074ce809e7d3f42b42ae711800501b5b16ea82ad029c39d"
[[package]]
name = "slab"
version = "0.4.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eb703cfe953bccee95685111adeedb76fabe4e97549a58d16f03ea7b9367bb32"
[[package]]
name = "slotmap"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dbff4acf519f630b3a3ddcfaea6c06b42174d9a44bc70c620e9ed1649d58b82a"
dependencies = [
"version_check",
]
[[package]]
name = "sluice"
version = "0.5.5"
@@ -6922,6 +7034,15 @@ dependencies = [
"loom",
]
[[package]]
name = "strict-num"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6637bab7722d379c8b41ba849228d680cc12d0a45ba1fa2b48f2a30577a06731"
dependencies = [
"float-cmp",
]
[[package]]
name = "string_cache"
version = "0.8.7"
@@ -6989,6 +7110,16 @@ version = "2.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "svgtypes"
version = "0.15.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "794de53cc48eaabeed0ab6a3404a65f40b3e38c067e4435883a65d2aa4ca000e"
dependencies = [
"kurbo",
"siphasher 1.0.1",
]
[[package]]
name = "syn"
version = "0.11.11"
@@ -7234,6 +7365,32 @@ dependencies = [
"time-core",
]
[[package]]
name = "tiny-skia"
version = "0.11.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "83d13394d44dae3207b52a326c0c85a8bf87f1541f23b0d143811088497b09ab"
dependencies = [
"arrayref",
"arrayvec",
"bytemuck",
"cfg-if",
"log",
"png",
"tiny-skia-path",
]
[[package]]
name = "tiny-skia-path"
version = "0.11.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9c9e7fc0c2e86a30b117d0462aa261b72b7a99b7ebd7deb3a14ceda95c5bdc93"
dependencies = [
"arrayref",
"bytemuck",
"strict-num",
]
[[package]]
name = "tinyvec"
version = "1.6.0"
@@ -7600,6 +7757,15 @@ version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "59547bce71d9c38b83d9c0e92b6066c4253371f15005def0c30d9657f50c7642"
[[package]]
name = "ttf-parser"
version = "0.24.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5be21190ff5d38e8b4a2d3b6a3ae57f612cc39c96e83cedeaf7abc338a8bac4a"
dependencies = [
"core_maths",
]
[[package]]
name = "tungstenite"
version = "0.17.2"
@@ -7727,6 +7893,18 @@ version = "0.3.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "08f95100a766bf4f8f28f90d77e0a5461bbdb219042e7679bebe79004fed8d75"
[[package]]
name = "unicode-bidi-mirroring"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "64af057ad7466495ca113126be61838d8af947f41d93a949980b2389a118082f"
[[package]]
name = "unicode-ccc"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "260bc6647b3893a9a90668360803a15f96b85a5257b1c3a0c3daf6ae2496de42"
[[package]]
name = "unicode-ident"
version = "1.0.0"
@@ -7742,12 +7920,30 @@ dependencies = [
"tinyvec",
]
[[package]]
name = "unicode-properties"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e70f2a8b45122e719eb623c01822704c4e0907e7e426a05927e1a1cfff5b75d0"
[[package]]
name = "unicode-script"
version = "0.5.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9fb421b350c9aff471779e262955939f565ec18b86c15364e6bdf0d662ca7c1f"
[[package]]
name = "unicode-segmentation"
version = "1.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1dd624098567895118886609431a7c3b8f516e41d30e0643f03d94592a147e36"
[[package]]
name = "unicode-vo"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b1d386ff53b415b7fe27b50bb44679e2cc4660272694b7b6f3326d8480823a94"
[[package]]
name = "unicode-width"
version = "0.1.14"
@@ -7838,6 +8034,33 @@ version = "2.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da"
[[package]]
name = "usvg"
version = "0.44.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7447e703d7223b067607655e625e0dbca80822880248937da65966194c4864e6"
dependencies = [
"base64 0.22.1",
"data-url",
"flate2",
"fontdb",
"imagesize",
"kurbo",
"log",
"pico-args",
"roxmltree",
"rustybuzz",
"simplecss",
"siphasher 1.0.1",
"strict-num",
"svgtypes",
"tiny-skia-path",
"unicode-bidi",
"unicode-script",
"unicode-vo",
"xmlwriter",
]
[[package]]
name = "utf-8"
version = "0.7.6"
@@ -7978,13 +8201,9 @@ checksum = "830b7e5d4d90034032940e4ace0d9a9a057e7a45cd94e6c007832e39edb82f6d"
[[package]]
name = "value-bag"
version = "1.0.0-alpha.9"
version = "1.9.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2209b78d1249f7e6f3293657c9779fe31ced465df091bbd433a1cf88e916ec55"
dependencies = [
"ctor",
"version_check",
]
checksum = "5a84c137d37ab0142f0f2ddfe332651fdbf252e7b7dbb4e67b6c1f1b2e925101"
[[package]]
name = "vcpkg"
@@ -8492,6 +8711,12 @@ version = "0.13.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "66fee0b777b0f5ac1c69bb06d361268faafa61cd4682ae064a171c16c433e9e4"
[[package]]
name = "xmlwriter"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec7a2a501ed189703dba8b08142f057e887dfc4b2cc4db2d343ac6376ba3e0b9"
[[package]]
name = "yaml-rust"
version = "0.4.5"
+5 -1
View File
@@ -95,17 +95,21 @@ Then continue:
# start other necessary services
docker compose up -d
# run everything together
./scripts/start.sh
# .. or individually
# run the API server
cargo run --bin revolt-delta
# run the events server
cargo run --bin revolt-bonfire
# run the file server
cargo run --bin revolt-autumn
# run th proxy server
# run the proxy server
cargo run --bin revolt-january
# hint:
# mold -run <cargo build, cargo run, etc...>
# mold -run ./scripts/start.sh
```
You can start a web client by doing the following:
+4 -5
View File
@@ -32,13 +32,12 @@ services:
image: minio/mc
depends_on:
- minio
environment:
MINIO_ROOT_USER: minioautumn
MINIO_ROOT_PASSWORD: minioautumn
entrypoint: >
/bin/sh -c "
/usr/bin/mc config host add minio http://minio:9000 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD;
while ! /usr/bin/mc ready minio; do echo 'Waiting minio...' && sleep 1; done;
while ! /usr/bin/mc ready minio; do
/usr/bin/mc config host add minio http://minio:9000 minioautumn minioautumn;
echo 'Waiting minio...' && sleep 1;
done;
/usr/bin/mc mb minio/revolt-uploads;
exit 0;
"
+4 -4
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-nodejs-bindings"
version = "0.7.17"
version = "0.7.19"
description = "Node.js bindings for the Revolt software"
authors = ["Paul Makles <me@insrt.uk>"]
license = "MIT"
@@ -20,6 +20,6 @@ serde = { version = "1", features = ["derive"] }
async-std = "1.12.0"
revolt-config = { version = "0.7.17", path = "../../core/config" }
revolt-result = { version = "0.7.17", path = "../../core/result" }
revolt-database = { version = "0.7.17", path = "../../core/database" }
revolt-config = { version = "0.7.19", path = "../../core/config" }
revolt-result = { version = "0.7.19", path = "../../core/result" }
revolt-database = { version = "0.7.19", path = "../../core/database" }
+2 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-bonfire"
version = "0.7.17"
version = "0.7.19"
license = "AGPL-3.0-or-later"
edition = "2021"
@@ -41,7 +41,7 @@ revolt-result = { path = "../core/result" }
revolt-models = { path = "../core/models" }
revolt-config = { path = "../core/config" }
revolt-database = { path = "../core/database" }
revolt-permissions = { version = "0.7.17", path = "../core/permissions" }
revolt-permissions = { version = "0.7.19", path = "../core/permissions" }
revolt-presence = { path = "../core/presence", features = ["redis-is-patched"] }
# redis
+11 -8
View File
@@ -79,7 +79,9 @@ pub async fn client(db: &'static Database, stream: TcpStream, addr: SocketAddr)
// Try to authenticate the user.
let Some(token) = config.get_session_token().as_ref() else {
write
.send(config.encode(&create_error!(InvalidSession)))
.send(config.encode(&EventV1::Error {
data: create_error!(InvalidSession),
}))
.await
.ok();
return;
@@ -88,7 +90,10 @@ pub async fn client(db: &'static Database, stream: TcpStream, addr: SocketAddr)
let (user, session_id) = match User::from_token(db, token, UserHint::Any).await {
Ok(user) => user,
Err(err) => {
write.send(config.encode(&err)).await.ok();
write
.send(config.encode(&EventV1::Error { data: err }))
.await
.ok();
return;
}
};
@@ -301,25 +306,23 @@ async fn listener(
PayloadType::Json => message
.value
.as_str()
.and_then(|s| serde_json::from_str::<EventV1>(s.as_ref()).ok()),
.and_then(|s| report_internal_error!(serde_json::from_str::<EventV1>(s.as_ref())).ok()),
PayloadType::Msgpack => message
.value
.as_bytes()
.and_then(|b| rmp_serde::from_slice::<EventV1>(b).ok()),
.and_then(|b| report_internal_error!(rmp_serde::from_slice::<EventV1>(b)).ok()),
PayloadType::Bincode => message
.value
.as_bytes()
.and_then(|b| bincode::deserialize::<EventV1>(b).ok()),
.and_then(|b| report_internal_error!(bincode::deserialize::<EventV1>(b)).ok()),
};
let Some(mut event) = event else {
let err = format!(
"Failed to deserialise an event for {}! Introspection: `{:?}`",
"Failed to deserialise event for {}: `{:?}`",
message.channel,
message
.value
.as_string()
.map(|x| x.chars().take(32).collect::<String>())
);
error!("{}", err);
+2 -3
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-config"
version = "0.7.17"
version = "0.7.19"
edition = "2021"
license = "MIT"
authors = ["Paul Makles <me@insrt.uk>"]
@@ -15,7 +15,6 @@ default = ["test"]
[dependencies]
# Utility
dotenv = "0.15.0"
config = "0.13.3"
cached = "0.44.0"
once_cell = "1.18.0"
@@ -35,4 +34,4 @@ pretty_env_logger = "0.4.0"
sentry = "0.31.5"
# Core
revolt-result = { version = "0.7.17", path = "../result", optional = true }
revolt-result = { version = "0.7.19", path = "../result", optional = true }
+4
View File
@@ -132,6 +132,7 @@ emojis = [128, 128]
#
# Backblaze B2:
# - endpoint is listed on the "Buckets" page
# - path_style_buckets is set to true
# - region is `eu-central-003` string from endpoint URL
# - access_key_id is keyID generated on the "Application Keys" page
# - secret_access_key is token generated on the "Application Keys" page
@@ -139,6 +140,9 @@ emojis = [128, 128]
# S3 protocol endpoint
endpoint = "http://minio:9000"
# Whether to use path-style buckets
# Generally true, except for MinIO
path_style_buckets = false
# S3 region name
region = "minio"
# S3 protocol key ID
+1 -2
View File
@@ -175,6 +175,7 @@ pub struct FilesLimit {
#[derive(Deserialize, Debug, Clone)]
pub struct FilesS3 {
pub endpoint: String,
pub path_style_buckets: bool,
pub region: String,
pub access_key_id: String,
pub secret_access_key: String,
@@ -286,8 +287,6 @@ pub async fn config() -> Settings {
/// Configure logging and common Rust variables
pub async fn setup_logging(release: &'static str, dsn: String) -> Option<sentry::ClientInitGuard> {
dotenv::dotenv().ok();
if std::env::var("RUST_LOG").is_err() {
std::env::set_var("RUST_LOG", "info");
}
+6 -6
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-database"
version = "0.7.17"
version = "0.7.19"
edition = "2021"
license = "AGPL-3.0-or-later"
authors = ["Paul Makles <me@insrt.uk>"]
@@ -24,15 +24,15 @@ default = ["mongodb", "async-std-runtime", "tasks"]
[dependencies]
# Core
revolt-config = { version = "0.7.17", path = "../config", features = [
revolt-config = { version = "0.7.19", path = "../config", features = [
"report-macros",
] }
revolt-result = { version = "0.7.17", path = "../result" }
revolt-models = { version = "0.7.17", path = "../models", features = [
revolt-result = { version = "0.7.19", path = "../result" }
revolt-models = { version = "0.7.19", path = "../models", features = [
"validator",
] }
revolt-presence = { version = "0.7.17", path = "../presence" }
revolt-permissions = { version = "0.7.17", path = "../permissions", features = [
revolt-presence = { version = "0.7.19", path = "../presence" }
revolt-permissions = { version = "0.7.19", path = "../permissions", features = [
"serde",
"bson",
] }
+13 -22
View File
@@ -1,25 +1,16 @@
use authifier::AuthifierEvent;
use revolt_result::Error;
use serde::{Deserialize, Serialize};
use revolt_models::v0::{
AppendMessage, Channel, ChannelUnread, Emoji, FieldsChannel, FieldsMember, FieldsMessage, FieldsRole, FieldsServer, FieldsUser, FieldsWebhook, Member, MemberCompositeKey, Message, PartialChannel, PartialMember, PartialMessage, PartialRole, PartialServer, PartialUser, PartialWebhook, RemovalIntention, Report, Server, User, UserSettings, Webhook
AppendMessage, Channel, ChannelUnread, Emoji, FieldsChannel, FieldsMember, FieldsMessage,
FieldsRole, FieldsServer, FieldsUser, FieldsWebhook, Member, MemberCompositeKey, Message,
PartialChannel, PartialMember, PartialMessage, PartialRole, PartialServer, PartialUser,
PartialWebhook, RemovalIntention, Report, Server, User, UserSettings, Webhook,
};
use revolt_result::Error;
use crate::Database;
/// WebSocket Client Errors
#[derive(Serialize, Deserialize, Debug, Clone)]
#[serde(tag = "error")]
pub enum WebSocketError {
LabelMe,
InternalError { at: String },
InvalidSession,
OnboardingNotFinished,
AlreadyAuthenticated,
MalformedData { msg: String },
}
/// Ping Packet
#[derive(Serialize, Deserialize, Debug, Clone)]
#[serde(untagged)]
@@ -28,14 +19,6 @@ pub enum Ping {
Number(usize),
}
/// Untagged Error
#[derive(Serialize)]
#[serde(untagged)]
pub enum ErrorEvent {
Error(WebSocketError),
APIError(Error),
}
/// Fields provided in Ready payload
#[derive(PartialEq)]
pub enum ReadyPayloadFields {
@@ -55,6 +38,8 @@ pub enum ReadyPayloadFields {
pub enum EventV1 {
/// Multiple events
Bulk { v: Vec<EventV1> },
/// Error event
Error { data: Error },
/// Successfully authenticated
Authenticated,
@@ -89,6 +74,7 @@ pub enum EventV1 {
id: String,
channel: String,
data: PartialMessage,
#[serde(default)]
clear: Vec<FieldsMessage>,
},
@@ -140,6 +126,7 @@ pub enum EventV1 {
ServerUpdate {
id: String,
data: PartialServer,
#[serde(default)]
clear: Vec<FieldsServer>,
},
@@ -150,6 +137,7 @@ pub enum EventV1 {
ServerMemberUpdate {
id: MemberCompositeKey,
data: PartialMember,
#[serde(default)]
clear: Vec<FieldsMember>,
},
@@ -168,6 +156,7 @@ pub enum EventV1 {
id: String,
role_id: String,
data: PartialRole,
#[serde(default)]
clear: Vec<FieldsRole>,
},
@@ -178,6 +167,7 @@ pub enum EventV1 {
UserUpdate {
id: String,
data: PartialUser,
#[serde(default)]
clear: Vec<FieldsUser>,
event_id: Option<String>,
},
@@ -212,6 +202,7 @@ pub enum EventV1 {
ChannelUpdate {
id: String,
data: PartialChannel,
#[serde(default)]
clear: Vec<FieldsChannel>,
},
@@ -11,6 +11,7 @@ use bson::oid::ObjectId;
use futures::StreamExt;
use rand::seq::SliceRandom;
use revolt_permissions::DEFAULT_WEBHOOK_PERMISSIONS;
use revolt_result::{Error, ErrorType};
use serde::{Deserialize, Serialize};
use unicode_segmentation::UnicodeSegmentation;
@@ -20,7 +21,7 @@ struct MigrationInfo {
revision: i32,
}
pub const LATEST_REVISION: i32 = 30;
pub const LATEST_REVISION: i32 = 31;
pub async fn migrate_database(db: &MongoDb) {
let migrations = db.col::<Document>("migrations");
@@ -1139,53 +1140,7 @@ pub async fn run_migrations(db: &MongoDb, revision: i32) -> i32 {
.expect("Failed to create attachment_hashes index.");
}
if revision <= 29 {
info!("Running migration [revision 29 / 29-09-2024]: Add creator_id to webhooks.");
#[derive(serde::Serialize, serde::Deserialize)]
struct WebhookShell {
_id: String,
channel_id: String,
}
let invites = db
.db()
.collection::<WebhookShell>("channel_webhooks")
.find(doc! {}, None)
.await
.expect("webhooks")
.filter_map(|s| async { s.ok() })
.collect::<Vec<WebhookShell>>()
.await;
for invite in invites {
let channel = db.fetch_channel(&invite.channel_id).await.expect("channel");
let creator_id = match channel {
Channel::Group { owner, .. } => owner,
Channel::TextChannel { server, .. } | Channel::VoiceChannel { server, .. } => {
let server = db.fetch_server(&server).await.expect("server");
server.owner
}
_ => unreachable!("not server or group channel!"),
};
db.db()
.collection::<Document>("channel_webhooks")
.update_one(
doc! {
"_id": invite._id,
},
doc! {
"$set" : {
"creator_id": creator_id
}
},
None,
)
.await
.expect("update webhook");
}
}
// Revision 29 omitted due to bug.
if revision <= 30 {
info!("Running migration [revision 30 / 29-09-2024]: Add index for used_for.id to attachments.");
@@ -1209,7 +1164,68 @@ pub async fn run_migrations(db: &MongoDb, revision: i32) -> i32 {
.expect("Failed to create attachments index.");
}
// Need to migrate fields on attachments, change `user_id`, `object_id`, etc to `parent`.
if revision <= 31 {
info!("Running migration [revision 31 / 31-10-2024]: Add creator_id to webhooks and delete those whose channels don't exist.");
#[derive(serde::Serialize, serde::Deserialize)]
struct WebhookShell {
_id: String,
channel_id: String,
}
let webhooks = db
.db()
.collection::<WebhookShell>("channel_webhooks")
.find(doc! {}, None)
.await
.expect("webhooks")
.filter_map(|s| async { s.ok() })
.collect::<Vec<WebhookShell>>()
.await;
for webhook in webhooks {
match db.fetch_channel(&webhook.channel_id).await {
Ok(channel) => {
let creator_id = match channel {
Channel::Group { owner, .. } => owner,
Channel::TextChannel { server, .. }
| Channel::VoiceChannel { server, .. } => {
let server = db.fetch_server(&server).await.expect("server");
server.owner
}
_ => unreachable!("not server or group channel!"),
};
db.db()
.collection::<Document>("channel_webhooks")
.update_one(
doc! {
"_id": webhook._id,
},
doc! {
"$set" : {
"creator_id": creator_id
}
},
None,
)
.await
.expect("update webhook");
}
Err(Error {
error_type: ErrorType::NotFound,
..
}) => {
db.db()
.collection::<WebhookShell>("channel_webhooks")
.delete_one(doc! { "_id": webhook._id }, None)
.await
.expect("failed to delete invalid webhook");
}
Err(err) => panic!("{err:?}"),
}
}
}
// Reminder to update LATEST_REVISION when adding new migrations.
LATEST_REVISION.max(revision)
@@ -1,4 +1,4 @@
use std::collections::HashSet;
use std::{collections::HashSet, hash::RandomState};
use indexmap::{IndexMap, IndexSet};
use iso8601_timestamp::Timestamp;
@@ -338,7 +338,34 @@ impl Message {
}
if !mentions.is_empty() {
message.mentions.replace(mentions.into_iter().collect());
// FIXME: temp fix to stop spam attacks
match channel {
Channel::DirectMessage { ref recipients, .. }
| Channel::Group { ref recipients, .. } => {
let recipients_hash: HashSet<&String, RandomState> =
HashSet::from_iter(recipients.iter());
mentions.retain(|m| recipients_hash.contains(m));
}
Channel::TextChannel { ref server, .. }
| Channel::VoiceChannel { ref server, .. } => {
let mentions_vec = Vec::from_iter(mentions.iter().cloned());
let valid_members = db.fetch_members(server.as_str(), &mentions_vec[..]).await;
if let Ok(valid_members) = valid_members {
let valid_ids: HashSet<String, RandomState> = HashSet::from_iter(
valid_members.iter().map(|member| member.id.user.clone()),
);
mentions.retain(|m| valid_ids.contains(m));
} else {
revolt_config::capture_error(&valid_members.unwrap_err());
}
}
Channel::SavedMessages { .. } => mentions.clear(),
}
if !mentions.is_empty() {
message.mentions.replace(mentions.into_iter().collect());
}
}
if !replies.is_empty() {
+8 -3
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-files"
version = "0.7.17"
version = "0.7.19"
edition = "2021"
license = "AGPL-3.0-or-later"
authors = ["Paul Makles <me@insrt.uk>"]
@@ -20,14 +20,19 @@ typenum = "1.17.0"
aws-config = "1.5.5"
aws-sdk-s3 = { version = "1.46.0", features = ["behavior-version-latest"] }
revolt-config = { version = "0.7.17", path = "../config", features = [
revolt-config = { version = "0.7.19", path = "../config", features = [
"report-macros",
] }
revolt-result = { version = "0.7.17", path = "../result" }
revolt-result = { version = "0.7.19", path = "../result" }
# image processing
jxl-oxide = "0.8.1"
image = { version = "0.25.2" }
# svg rendering
usvg = "0.44.0"
resvg = "0.44.0"
tiny-skia = "0.11.4"
# encoding
webp = "0.3.0"
+88 -25
View File
@@ -4,7 +4,7 @@ use aes_gcm::{
aead::{AeadCore, AeadMutInPlace, OsRng},
Aes256Gcm, Key, KeyInit, Nonce,
};
use image::DynamicImage;
use image::{DynamicImage, ImageBuffer};
use revolt_config::{config, report_internal_error, FilesS3};
use revolt_result::{create_error, Result};
@@ -15,6 +15,7 @@ use aws_sdk_s3::{
use base64::prelude::*;
use tempfile::NamedTempFile;
use tiny_skia::Pixmap;
/// Size of the authentication tag in the buffer
pub const AUTHENTICATION_TAG_SIZE_BYTES: usize = 16;
@@ -33,6 +34,7 @@ pub fn create_client(s3_config: FilesS3) -> Client {
let config = Config::builder()
.region(Region::new(s3_config.region))
.endpoint_url(s3_config.endpoint)
.force_path_style(s3_config.path_style_buckets)
.credentials_provider(creds)
.build();
@@ -123,13 +125,24 @@ pub fn image_size(f: &NamedTempFile) -> Option<(usize, usize)> {
}
/// Determine size of image with buffer
pub fn image_size_vec(v: &[u8]) -> Option<(usize, usize)> {
if let Ok(size) = imagesize::blob_size(v)
.inspect_err(|err| tracing::error!("Failed to generate image size! {err:?}"))
{
Some((size.width, size.height))
} else {
None
pub fn image_size_vec(v: &[u8], mime: &str) -> Option<(usize, usize)> {
match mime {
"image/svg+xml" => {
let tree =
report_internal_error!(usvg::Tree::from_data(v, &Default::default())).ok()?;
let size = tree.size();
Some((size.width() as usize, size.height() as usize))
}
_ => {
if let Ok(size) = imagesize::blob_size(v)
.inspect_err(|err| tracing::error!("Failed to generate image size! {err:?}"))
{
Some((size.width, size.height))
} else {
None
}
}
}
}
@@ -152,38 +165,88 @@ pub fn video_size(f: &NamedTempFile) -> Option<(i64, i64)> {
}
/// Decode image from reader
pub fn decode_image<R: Read + BufRead + Seek>(
reader: &mut R,
is_jxl: bool,
) -> Result<DynamicImage> {
if is_jxl {
Err(create_error!(LabelMe))
} else {
pub fn decode_image<R: Read + BufRead + Seek>(reader: &mut R, mime: &str) -> Result<DynamicImage> {
match mime {
// Read image using jxl-oxide crate
"image/jxl" => {
let jxl_image = report_internal_error!(jxl_oxide::JxlImage::builder().read(reader))?;
if let Ok(frame) = jxl_image.render_frame(0) {
match frame.color_channels().len() {
3 => Ok(DynamicImage::ImageRgb8(
DynamicImage::ImageRgb32F(
ImageBuffer::from_vec(
jxl_image.width(),
jxl_image.height(),
frame.image().buf().to_vec(),
)
.ok_or_else(|| create_error!(ImageProcessingFailed))?,
)
.to_rgb8(),
)),
4 => Ok(DynamicImage::ImageRgba8(
DynamicImage::ImageRgba32F(
ImageBuffer::from_vec(
jxl_image.width(),
jxl_image.height(),
frame.image().buf().to_vec(),
)
.ok_or_else(|| create_error!(ImageProcessingFailed))?,
)
.to_rgba8(),
)),
_ => Err(create_error!(ImageProcessingFailed)),
}
} else {
Err(create_error!(ImageProcessingFailed))
}
}
// Read image using resvg
"image/svg+xml" => {
// usvg doesn't support Read trait so copy to buffer
let mut buf = Vec::new();
report_internal_error!(reader.read_to_end(&mut buf))?;
let tree = report_internal_error!(usvg::Tree::from_data(&buf, &Default::default()))?;
let size = tree.size();
let mut pixmap = Pixmap::new(size.width() as u32, size.height() as u32)
.ok_or_else(|| create_error!(ImageProcessingFailed))?;
let mut pixmap_mut = pixmap.as_mut();
resvg::render(&tree, Default::default(), &mut pixmap_mut);
Ok(DynamicImage::ImageRgba8(
ImageBuffer::from_vec(
size.width() as u32,
size.height() as u32,
pixmap.data().to_vec(),
)
.ok_or_else(|| create_error!(ImageProcessingFailed))?,
))
}
// Check if we can read using image-rs crate
report_internal_error!(report_internal_error!(
_ => report_internal_error!(report_internal_error!(
image::ImageReader::new(reader).with_guessed_format()
)?
.decode())
.decode()),
}
}
/// Check whether given reader has a valid image
pub fn is_valid_image<R: Read + BufRead + Seek>(reader: &mut R, is_jxl: bool) -> bool {
if is_jxl {
pub fn is_valid_image<R: Read + BufRead + Seek>(reader: &mut R, mime: &str) -> bool {
match mime {
// Check if we can read using jxl-oxide crate
jxl_oxide::JxlImage::builder()
"image/jxl" => jxl_oxide::JxlImage::builder()
.read(reader)
.inspect_err(|err| tracing::error!("Failed to read JXL! {err:?}"))
.is_ok()
} else {
!matches!(
// Check if we can read using image-rs crate
.is_ok(),
// Check if we can read using image-rs crate
_ => !matches!(
image::ImageReader::new(reader)
.with_guessed_format()
.inspect_err(|err| tracing::error!("Failed to read image! {err:?}"))
.map(|f| f.decode()),
Err(_) | Ok(Err(_))
)
),
}
}
+3 -3
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-models"
version = "0.7.17"
version = "0.7.19"
edition = "2021"
license = "MIT"
authors = ["Paul Makles <me@insrt.uk>"]
@@ -20,8 +20,8 @@ default = ["serde", "partials", "rocket"]
[dependencies]
# Core
revolt-config = { version = "0.7.17", path = "../config" }
revolt-permissions = { version = "0.7.17", path = "../permissions" }
revolt-config = { version = "0.7.19", path = "../config" }
revolt-permissions = { version = "0.7.19", path = "../permissions" }
# Utility
regex = "1"
+12
View File
@@ -168,6 +168,14 @@ auto_derived!(
impl WebsiteMetadata {
/// Truncate strings in metadata
pub fn truncate(&mut self) {
if let Some(s) = self.url.as_mut() {
s.truncate(256);
}
if let Some(s) = self.original_url.as_mut() {
s.truncate(256);
}
if let Some(s) = self.title.as_mut() {
s.truncate(100);
}
@@ -180,6 +188,10 @@ impl WebsiteMetadata {
s.truncate(32);
}
if let Some(s) = self.icon_url.as_mut() {
s.truncate(256);
}
if let Some(s) = self.colour.as_mut() {
s.truncate(32);
}
+2 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-permissions"
version = "0.7.17"
version = "0.7.19"
edition = "2021"
license = "MIT"
authors = ["Paul Makles <me@insrt.uk>"]
@@ -21,7 +21,7 @@ async-std = { version = "1.8.0", features = ["attributes"] }
[dependencies]
# Core
revolt-result = { version = "0.7.17", path = "../result" }
revolt-result = { version = "0.7.19", path = "../result" }
# Utility
auto_ops = "0.3.0"
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-presence"
version = "0.7.17"
version = "0.7.19"
edition = "2021"
license = "AGPL-3.0-or-later"
authors = ["Paul Makles <me@insrt.uk>"]
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-result"
version = "0.7.17"
version = "0.7.19"
edition = "2021"
license = "MIT"
authors = ["Paul Makles <me@insrt.uk>"]
+3
View File
@@ -1,4 +1,5 @@
use axum::{http::StatusCode, response::IntoResponse, Json};
use rocket::http::Status;
use crate::{Error, ErrorType};
@@ -78,6 +79,8 @@ impl IntoResponse for Error {
ErrorType::FileTooSmall => StatusCode::UNPROCESSABLE_ENTITY,
ErrorType::FileTooLarge { .. } => StatusCode::UNPROCESSABLE_ENTITY,
ErrorType::FileTypeNotAllowed => StatusCode::BAD_REQUEST,
ErrorType::ImageProcessingFailed => StatusCode::INTERNAL_SERVER_ERROR,
ErrorType::NoEmbedData => StatusCode::BAD_REQUEST,
};
(status, Json(&self)).into_response()
+2
View File
@@ -162,6 +162,8 @@ pub enum ErrorType {
max: usize,
},
FileTypeNotAllowed,
ImageProcessingFailed,
NoEmbedData,
// ? Legacy errors
VosoUnavailable,
+2
View File
@@ -84,6 +84,8 @@ impl<'r> Responder<'r, 'static> for Error {
ErrorType::FileTooSmall => Status::UnprocessableEntity,
ErrorType::FileTooLarge { .. } => Status::UnprocessableEntity,
ErrorType::FileTypeNotAllowed => Status::BadRequest,
ErrorType::ImageProcessingFailed => Status::InternalServerError,
ErrorType::NoEmbedData => Status::BadRequest,
};
// Serialize the error data structure into JSON.
+1 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-delta"
version = "0.7.17"
version = "0.7.19"
license = "AGPL-3.0-or-later"
authors = ["Paul Makles <paulmakles@gmail.com>"]
edition = "2018"
@@ -16,7 +16,6 @@ redis-kiss = "0.1.4"
lru = "0.7.0"
url = "2.2.2"
log = "0.4.11"
dotenv = "0.15.0"
dashmap = "5.2.0"
linkify = "0.6.0"
once_cell = "1.17.1"
-2
View File
@@ -19,8 +19,6 @@ pub struct TestHarness {
impl TestHarness {
pub async fn new() -> TestHarness {
dotenv::dotenv().ok();
let client = Client::tracked(crate::web().await)
.await
.expect("valid rocket instance");
+6 -6
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-autumn"
version = "0.7.17"
version = "0.7.19"
edition = "2021"
[dependencies]
@@ -13,7 +13,7 @@ webp = "0.3.0"
sha2 = "0.10.8"
jxl-oxide = "0.8.1"
kamadak-exif = "0.5.4"
# revolt_little_exif = "0.4.0"
# revolt_little_exif = "0.5.1"
image = { version = "0.25.2" } # avif encode requires dav1d system library: features = ["avif-native"]
# File processing
@@ -42,12 +42,12 @@ tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
# Core crates
revolt-files = { version = "0.7.17", path = "../../core/files" }
revolt-config = { version = "0.7.17", path = "../../core/config" }
revolt-database = { version = "0.7.17", path = "../../core/database", features = [
revolt-files = { version = "0.7.19", path = "../../core/files" }
revolt-config = { version = "0.7.19", path = "../../core/config" }
revolt-database = { version = "0.7.19", path = "../../core/database", features = [
"axum-impl",
] }
revolt-result = { version = "0.7.17", path = "../../core/result", features = [
revolt-result = { version = "0.7.19", path = "../../core/result", features = [
"utoipa",
"axum",
] }
+25 -8
View File
@@ -11,7 +11,6 @@ use axum::{
Json, Router,
};
use axum_typed_multipart::{FieldData, TryFromMultipart, TypedMultipart};
use image::ImageReader;
use lazy_static::lazy_static;
use revolt_config::{config, report_internal_error};
use revolt_database::{iso8601_timestamp::Timestamp, Database, FileHash, Metadata, User};
@@ -208,16 +207,27 @@ async fn upload_file(
};
// Find an existing hash and use that if possible
if let Ok(file_hash) = db
let file_hash_exists = if let Ok(file_hash) = db
.fetch_attachment_hash(&format!("{original_hash:02x}"))
.await
{
let tag: &'static str = tag.into();
db.insert_attachment(&file_hash.into_file(id.clone(), tag.to_owned(), filename, user.id))
if !file_hash.iv.is_empty() {
let tag: &'static str = tag.into();
db.insert_attachment(&file_hash.into_file(
id.clone(),
tag.to_owned(),
filename,
user.id,
))
.await?;
return Ok(Json(UploadResponse { id }));
}
return Ok(Json(UploadResponse { id }));
}
true
} else {
false
};
// Determine the mime type for the file
let mime_type = determine_mime_type(&mut file.contents, &buf, &filename);
@@ -275,7 +285,10 @@ async fn upload_file(
size: new_file_size as isize,
};
db.insert_attachment_hash(&file_hash).await?;
// Add attachment hash if it doesn't exist
if !file_hash_exists {
db.insert_attachment_hash(&file_hash).await?;
}
// Upload the file to S3 and commit nonce to database
let upload_start = Instant::now();
@@ -355,7 +368,11 @@ async fn fetch_preview(
let data = retrieve_file_by_hash(&hash).await?;
// Read image and create thumbnail
let data = create_thumbnail(decode_image(&mut Cursor::new(data), false)?, tag).await;
let data = create_thumbnail(
decode_image(&mut Cursor::new(data), &file.content_type)?,
tag,
)
.await;
Ok((
[
+18 -17
View File
@@ -17,23 +17,24 @@ pub async fn strip_metadata(
) -> Result<(Vec<u8>, Metadata)> {
match &metadata {
Metadata::Image { width, height } => match mime {
// little_exif does not appear to parse JPEGs correctly? had 2/2 files fail
/* "image/jpeg" | "image/png" => {
// use little_exif to strip metadata except for orientation and colour profile
// PNGs must also be re-encoded to mitigate CVE-2023-21036
let metadata = revolt_little_exif::metadata::Metadata::new_from_path_with_filetype(
file.path(),
match mime {
"image/jpeg" => revolt_little_exif::filetype::FileExtension::JPEG,
"image/png" => revolt_little_exif::filetype::FileExtension::PNG {
as_zTXt_chunk: true,
},
_ => unreachable!(),
},
)
.unwrap();
dbg!(metadata.data());
} */
// // little_exif does not appear to parse JPEGs correctly? had 2/2 files fail
// "image/jpeg" | "image/png" => {
// // use little_exif to strip metadata except for orientation and colour profile
// // PNGs must also be re-encoded to mitigate CVE-2023-21036
// let metadata = revolt_little_exif::metadata::Metadata::new_from_path_with_filetype(
// file.path(),
// match mime {
// "image/jpeg" => revolt_little_exif::filetype::FileExtension::JPEG,
// "image/png" => revolt_little_exif::filetype::FileExtension::PNG {
// as_zTXt_chunk: true,
// },
// _ => unreachable!(),
// },
// )
// .unwrap();
// dbg!(metadata.data());
// todo!()
// }
// Apply orientation manually & strip all other EXIF data
"image/jpeg" | "image/png" | "image/avif" | "image/tiff" => {
// Create a reader
+5 -1
View File
@@ -19,7 +19,11 @@ pub fn determine_mime_type(f: &mut NamedTempFile, buf: &[u8], file_name: &str) -
// See if the file is actually just plain Unicode/ASCII text
if mime_type == "application/octet-stream" && simdutf8::basic::from_utf8(buf).is_ok() {
return "plain/text";
if file_name.to_lowercase().ends_with(".svg") {
return "image/svg+xml";
} else {
return "plain/text";
}
}
mime_type
+5 -5
View File
@@ -1,6 +1,6 @@
[package]
name = "revolt-january"
version = "0.7.17"
version = "0.7.19"
edition = "2021"
[dependencies]
@@ -31,13 +31,13 @@ tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
# Core crates
revolt-config = { version = "0.7.17", path = "../../core/config" }
revolt-models = { version = "0.7.17", path = "../../core/models" }
revolt-result = { version = "0.7.17", path = "../../core/result", features = [
revolt-config = { version = "0.7.19", path = "../../core/config" }
revolt-models = { version = "0.7.19", path = "../../core/models" }
revolt-result = { version = "0.7.19", path = "../../core/result", features = [
"utoipa",
"axum",
] }
revolt-files = { version = "0.7.17", path = "../../core/files" }
revolt-files = { version = "0.7.19", path = "../../core/files" }
# Axum / web server
axum = { version = "0.7.5" }
+7 -2
View File
@@ -1,6 +1,7 @@
use axum::{extract::Query, response::IntoResponse, routing::get, Json, Router};
use reqwest::header;
use revolt_result::Result;
use revolt_models::v0::Embed;
use revolt_result::{create_error, Result};
use serde::{Deserialize, Serialize};
use utoipa::ToSchema;
@@ -87,5 +88,9 @@ async fn embed(
Query(UrlQuery { url }): Query<UrlQuery>,
// TypedHeader(Authorization(_bearer)): TypedHeader<Authorization<Bearer>>,
) -> Result<impl IntoResponse> {
Request::generate_embed(url).await.map(Json)
match Request::generate_embed(url).await {
Ok(Embed::None) => Err(create_error!(NoEmbedData)),
result => result,
}
.map(Json)
}
+2
View File
@@ -66,6 +66,8 @@ async fn main() -> Result<(), std::io::Error> {
.nest("/", api::router().await);
// Configure TCP listener and bind
tracing::info!("Listening on 0.0.0.0:14705");
tracing::info!("Play around with the API: http://localhost:14705/scalar");
let address = SocketAddr::from((Ipv4Addr::UNSPECIFIED, 14705));
let listener = TcpListener::bind(&address).await?;
axum::serve(listener, app.into_make_service()).await
+46 -20
View File
@@ -33,7 +33,10 @@ lazy_static! {
.expect("reqwest Client");
/// Spoof User Agent as Discord
static ref RE_USER_AGENT_SPOOFING_AS_DISCORD: Regex = Regex::new("^(?:(?:https?:)?//)?(?:(?:vx|fx)?twitter|(?:fixv|fixup)?x).com").expect("valid regex");
static ref RE_USER_AGENT_SPOOFING_AS_DISCORD: Regex = Regex::new("^(?:(?:https?:)?//)?(?:(?:vx|fx)?twitter|(?:fixv|fixup)?x|(?:old\\.|new\\.|www\\.)reddit).com").expect("valid regex");
/// Regex for matching new Reddit URLs
static ref RE_URL_NEW_REDDIT: Regex = Regex::new("^(?:(?:https?:)?//)?(?:(?:new\\.|www\\.)?reddit).com").expect("valid regex");
/// Cache for proxy results
static ref PROXY_CACHE: moka::future::Cache<String, Result<(String, Vec<u8>)>> = moka::future::Cache::builder()
@@ -63,7 +66,7 @@ impl Request {
let Request { response, mime } = Request::new(url).await?;
if matches!(mime.type_(), mime::IMAGE | mime::VIDEO) {
let bytes = response.bytes().await.map_err(|_| create_error!(LabelMe));
let bytes = report_internal_error!(response.bytes().await);
let result = match bytes {
Ok(bytes) => {
@@ -71,16 +74,19 @@ impl Request {
let reader = &mut Cursor::new(&bytes);
if matches!(mime.subtype(), mime::GIF) {
if is_valid_image(reader, false) {
if is_valid_image(reader, "image/gif") {
Ok(("image/gif".to_owned(), bytes.to_vec()))
} else {
Err(create_error!(LabelMe))
Err(create_error!(FileTypeNotAllowed))
}
} else {
Ok((
"image/webp".to_owned(),
create_thumbnail(decode_image(reader, false)?, "attachments")
.await,
create_thumbnail(
decode_image(reader, mime.as_ref())?,
"attachments",
)
.await,
))
}
} else {
@@ -90,7 +96,7 @@ impl Request {
if video_size(&file).is_some() {
Ok((mime.to_string(), bytes.to_vec()))
} else {
Err(create_error!(LabelMe))
Err(create_error!(FileTypeNotAllowed))
}
}
}
@@ -100,7 +106,7 @@ impl Request {
PROXY_CACHE.insert(url.to_owned(), result.clone()).await;
result
} else {
Err(create_error!(LabelMe))
Err(create_error!(FileTypeNotAllowed))
}
}
}
@@ -116,20 +122,21 @@ impl Request {
_ => Ok(None),
}
} else {
let response = if let Some(Request { response, .. }) = request {
response
let request = if let Some(request) = request {
request
} else {
let Request { response, mime } = Request::new(url).await?;
if matches!(mime.type_(), mime::IMAGE) {
response
let request = Request::new(url).await?;
if matches!(request.mime.type_(), mime::IMAGE) {
request
} else {
return Err(create_error!(LabelMe));
return Err(create_error!(FileTypeNotAllowed));
}
};
if let Some((width, height)) =
image_size_vec(&report_internal_error!(response.bytes().await)?)
{
if let Some((width, height)) = image_size_vec(
&report_internal_error!(request.response.bytes().await)?,
request.mime.as_ref(),
) {
Ok(Some(Image {
url: url.to_owned(),
width,
@@ -156,11 +163,11 @@ impl Request {
let response = if let Some(Request { response, .. }) = request {
response
} else {
let Request { response, mime } = Request::new(&url).await?;
let Request { response, mime } = Request::new(url).await?;
if matches!(mime.type_(), mime::VIDEO) {
response
} else {
return Err(create_error!(LabelMe));
return Err(create_error!(FileTypeNotAllowed));
}
};
@@ -182,7 +189,16 @@ impl Request {
}
/// Generate embed for a given URL
pub async fn generate_embed(url: String) -> Result<Embed> {
pub async fn generate_embed(mut url: String) -> Result<Embed> {
// Re-map certain links for better metadata generation
if RE_URL_NEW_REDDIT.is_match(&url) {
url = RE_URL_NEW_REDDIT
// Reddit has a bunch of clickbait-y marketing on the new URLs, so we use the old site instead
.replace(&url, "https://old.reddit.com")
.to_string();
}
// Generate the actual embed
if let Some(hit) = EMBED_CACHE.get(&url).await {
Ok(hit)
} else {
@@ -240,6 +256,7 @@ impl Request {
"Mozilla/5.0 (compatible; January/2.0; +https://github.com/revoltchat/backend)"
},
)
.header("Accept-Language", "en-US,en;q=0.5")
.send()
.await
.map_err(|_| create_error!(ProxyError))?;
@@ -262,4 +279,13 @@ impl Request {
Ok(Request { response, mime })
}
/// Check if something exists
pub async fn exists(url: &str) -> bool {
if let Ok(response) = CLIENT.head(url).send().await {
response.status().is_success()
} else {
false
}
}
}
+87 -56
View File
@@ -9,34 +9,38 @@ use scraper::{Html, Selector};
/// Create website metadata from URL and document
pub async fn create_website_embed(original_url: &str, document: &str) -> Option<WebsiteMetadata> {
let document = Html::parse_document(document);
let (mut meta, mut link) = {
let document = Html::parse_document(document);
// create selectors
let meta_selector = Selector::parse("meta").ok()?;
let link_selector = Selector::parse("link").ok()?;
// create selectors
let meta_selector = Selector::parse("meta").ok()?;
let link_selector = Selector::parse("link").ok()?;
// extract meta tags
let mut meta = HashMap::new();
for el in document.select(&meta_selector) {
let node = el.value();
// extract meta tags
let mut meta = HashMap::new();
for el in document.select(&meta_selector) {
let node = el.value();
if let (Some(property), Some(content)) = (
node.attr("property").or_else(|| node.attr("name")),
node.attr("content"),
) {
meta.insert(property.to_string(), content.to_string());
if let (Some(property), Some(content)) = (
node.attr("property").or_else(|| node.attr("name")),
node.attr("content"),
) {
meta.insert(property.to_string(), content.to_string());
}
}
}
// extract rel links
let mut link = HashMap::new();
for el in document.select(&link_selector) {
let node = el.value();
// extract rel links
let mut link = HashMap::new();
for el in document.select(&link_selector) {
let node = el.value();
if let (Some(property), Some(content)) = (node.attr("rel"), node.attr("href")) {
link.insert(property.to_string(), content.to_string());
if let (Some(property), Some(content)) = (node.attr("rel"), node.attr("href")) {
link.insert(property.to_string(), content.to_string());
}
}
}
(meta, link)
};
// build metadata
let mut metadata = WebsiteMetadata {
@@ -137,32 +141,41 @@ pub async fn create_website_embed(original_url: &str, document: &str) -> Option<
};
// populate extra metadata for popular websites
populate_special(original_url.to_owned(), &mut metadata);
// TODO: these do not work because compiler is tripping:
populate_special(original_url.to_owned(), &mut metadata).await;
// fetch video size if missing
// if let Some(Video { width, height, url }) = &metadata.video {
// if width == &0 || height == &0 {
// metadata.video = match crate::requests::Request::fetch_video_metadata(url, None).await {
// Ok(Some(video)) => Some(video),
// _ => None,
// }
// }
// }
if metadata.special.is_none() {
if let Some(Video { width, height, url }) = &metadata.video {
if width == &0 || height == &0 {
metadata.video =
match crate::requests::Request::fetch_video_metadata(url, None).await {
Ok(Some(video)) => Some(video),
_ => None,
}
}
}
}
// remove image if video exists
if metadata.video.is_some() {
metadata.image.take();
}
// fetch image size if missing
// if let Some(Image {
// width, height, url, ..
// }) = &metadata.image
// {
// if width == &0 || height == &0 {
// metadata.image = match crate::requests::Request::fetch_image_metadata(url, None).await {
// Ok(Some(image)) => Some(image),
// _ => None,
// }
// }
// }
if metadata.special.is_none() {
if let Some(Image {
width, height, url, ..
}) = &metadata.image
{
if width == &0 || height == &0 {
metadata.image =
match crate::requests::Request::fetch_image_metadata(url, None).await {
Ok(Some(image)) => Some(image),
_ => None,
}
}
}
}
// truncate data
metadata.truncate();
@@ -175,7 +188,7 @@ pub async fn create_website_embed(original_url: &str, document: &str) -> Option<
}
}
pub fn populate_special(original_url: String, metadata: &mut WebsiteMetadata) {
pub async fn populate_special(original_url: String, metadata: &mut WebsiteMetadata) {
lazy_static! {
static ref RE_YOUTUBE: Regex = Regex::new("^(?:(?:https?:)?//)?(?:(?:www|m)\\.)?(?:(?:youtube\\.com|youtu.be))(?:/(?:[\\w\\-]+\\?v=|embed/|v/)?)([\\w\\-]+)(?:\\S+)?$").unwrap();
@@ -207,24 +220,42 @@ pub fn populate_special(original_url: String, metadata: &mut WebsiteMetadata) {
id: captures[1].to_string(),
})
} else if let Some(captures) = RE_YOUTUBE.captures_iter(url).next() {
let id = captures[1].to_string();
lazy_static! {
static ref RE_TIMESTAMP: Regex = Regex::new("(?:\\?|&)(?:t|start)=([\\w]+)").unwrap();
}
if let Some(video) = &metadata.video {
if let Some(timestamp_captures) = RE_TIMESTAMP.captures_iter(&video.url).next() {
Some(Special::YouTube {
id: captures[1].to_string(),
timestamp: Some(timestamp_captures[1].to_string()),
})
} else {
Some(Special::YouTube {
id: captures[1].to_string(),
timestamp: None,
})
// YouTube now blocks datacentre IPs from fetching information
// This is a fallback to prevent the embed from looking weird
if metadata.video.is_none() {
metadata.title.replace("YouTube".to_owned());
metadata.description.take();
metadata.colour.take();
metadata.icon_url.take();
metadata.site_name.take();
// Verify the video exists
if !crate::requests::Request::exists(&format!(
"http://img.youtube.com/vi/{}/sddefault.jpg",
id
))
.await
{
return;
}
}
if let Some(timestamp_captures) = RE_TIMESTAMP.captures_iter(url).next() {
Some(Special::YouTube {
id,
timestamp: Some(timestamp_captures[1].to_string()),
})
} else {
None
Some(Special::YouTube {
id,
timestamp: None,
})
}
} else if let Some(captures) = RE_LIGHTSPEED.captures_iter(url).next() {
Some(Special::Lightspeed {
+1
View File
@@ -2,6 +2,7 @@
- [Introduction](./hello.md)
- [Project Structure]()
- [Creating new API features](./new_features.md)
- [Testing]()
- [Writing a new database test]()
- [Writing a new API test]()
+15
View File
@@ -0,0 +1,15 @@
# New API features
New API features must be documented where appropriate, this document aims to cover everywhere you need to update for new features.
Before writing new API features, generally a good idea to:
- Consult with other developers in the [Revolt Developers space](https://rvlt.gg/API)
- If it's a relatively big feature, also [write an RFC](https://github.com/revoltchat/rfcs)
When your feature is ready to release, ensure to:
- Update backend documentation (what you're reading now!) if applicable
- Update the [developers documentation](https://github.com/revoltchat/wiki) if applicable
- Update the Feature Matrix (or ask someone that can to do so)
- Ensure it is properly listed in the backend release changelog
+2
View File
@@ -24,6 +24,7 @@ docker build -t ghcr.io/revoltchat/base:latest -f Dockerfile.useCurrentArch .
docker build -t ghcr.io/revoltchat/server:$TAG - < crates/delta/Dockerfile
docker build -t ghcr.io/revoltchat/bonfire:$TAG - < crates/bonfire/Dockerfile
docker build -t ghcr.io/revoltchat/autumn:$TAG - < crates/services/autumn/Dockerfile
docker build -t ghcr.io/revoltchat/january:$TAG - < crates/services/january/Dockerfile
if [ "$DEBUG" = "true" ]; then
git restore Cargo.toml
@@ -32,3 +33,4 @@ fi
docker push ghcr.io/revoltchat/server:$TAG
docker push ghcr.io/revoltchat/bonfire:$TAG
docker push ghcr.io/revoltchat/autumn:$TAG
docker push ghcr.io/revoltchat/january:$TAG
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -e
cargo build \
--bin revolt-delta \
--bin revolt-bonfire \
--bin revolt-autumn \
--bin revolt-january
trap 'pkill -f revolt-' SIGINT
cargo run --bin revolt-delta &
cargo run --bin revolt-bonfire &
cargo run --bin revolt-autumn &
cargo run --bin revolt-january