@grandeto/vrchats

Description

HTTP/WebSocket real-time message processor based on Socket.io

Requirements

• Node.js version: refer to package.json engines.node

Features

• HTTP event handler
• WebSocket multicast emitter
• Publishers: Trusted Origins, CIDRs, PROXY
• Subscribers: WebSocket Auth, Rate Limiter

Environment

Tune OS (Linux) and Node.js for websocket connections

https://tcminh.com/600k-concurrent-websocket-connections-on-aws-using-node-js/

sudo nano /etc/security/limits.d/custom.conf

root soft nofile 1000000
root hard nofile 1000000
* soft nofile 1000000
* hard nofile 1000000

sudo nano /etc/sysctl.conf

fs.file-max = 1000000
fs.nr_open = 1000000
net.ipv4.netfilter.ip_conntrack_max = 1048576
net.nf_conntrack_max = 1048576

sudo nano /etc/sysctl.d/net.ipv4.ip_local_port_range.conf

net.ipv4.ip_local_port_range = 10000 65535

• check ulimits

ulimit -Sa && echo -e "\n" && ulimit -Ha

• set memlock if too low

  • example of setting 15.5GB memlock in /etc/docker/daemon.json

  *NOTE: in daemon.json memlock soft:hard values should be in binary bytes representation e.g. 15872000000 B instead of 15500000 KB - https://www.gbmb.org/kilobytes

  "default-ulimits": {
      "memlock": {
          "Hard": 15872000000,
          "Name": "memlock",
          "Soft": 15872000000
      }
  }

  *NOTE in /etc/security/limits.d/custom.conf memlock soft:hard values should be in decimal kilobytes representation e.g. 15500000 KB

  root soft memlock 500000
  root hard memlock 500000
  * soft memlock 15500000
  * hard memlock 15500000

• Create an .env file from .env_example

chmod 600 .env

• Add pubkey.pem, privkey.pem, ca.pem in ./certs

chmod 600 ./certs/*.pem

• Whitelist .env ports in firewall

Docker

Prod

• Build an image

docker build -t "vrchats:vX.X.X" .

• Start a container

docker run -d --name vrchats --restart always -p 8443:8443 -p 2053:2053 vrchats:vX.X.X

• applying changes

  • re-run build step followed by:

  docker stop vrchats && docker rm vrchats && docker run -d --name vrchats --restart always -p 8443:8443 -p 2053:2053 vrchats:vX.X.X

• *NOTE: Consider adding a cronjob in order to handle https://github.com/grandeto/vrchats#pm2-knows-issues

  0 0 * * * docker restart vrchats

Dev

npm run dev.run

npm run dev.remove

npm run dev.update

pm2

• Save pm2 settings

pm2 save

• pm2 deamon

pm2 startup

• Copy and execute the output command generated by pm2 startup then:

sudo nano /etc/systemd/system/pm2-$USER.service and:

add Wants=network-online.target above After=network.target

add RequiresMountsFor=/home above After=network.target

change After=network.target to After=network.target network-online.target

change WantedBy=multi-user.target to WantedBy=multi-user.target network-online.target

add under [Service] the following env variables:

Environment=NODE_ENV=production
Environment=CLUSTER_MODE=1
Environment=CLUSTER_INSTANCES="-1"
Environment=PRODUCER_PORT=2053
Environment=CONSUMER_PORT=8443
Environment=STANDALONE_PORT=8443
Environment=TRUST_ORIGIN_LIST="https://example.com"
Environment=TRUST_CIDR_LIST="123.123.123.123/32,127.0.0.1/32,::1/128"
Environment=USE_PROXY=1
Environment=TRUST_PROXY_LIST="103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,104.16.0.0/13,104.24.0.0/14,108.162.192.0/18,131.0.72.0/22,141.101.64.0/18,162.158.0.0/15,172.64.0.0/13,173.245.48.0/20,188.114.96.0/20,190.93.240.0/20,197.234.240.0/22,198.41.128.0/17,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2c0f:f248::/32,2a06:98c0::/29"
Environment=PUB_KEY_PATH=/var/www/vrchats/pubkey.pem
Environment=PRIV_KEY_PATH=/var/www/vrchats/privkey.pem
Environment=CA_PATH=/var/www/vrchats/ca.pem
Environment=VERIFY_ORIGIN=1
Environment=WEBSOCKET_AUTH_TOKEN_RENEW_START_HOUR=0
Environment=WEBSOCKET_AUTH_TOKEN_RENEW_INTERVAL=86400000
Environment=WEBSOCKET_AUTH_TOKEN_SECRET="some-nasty-secret"

• Execute sudo systemctl daemon-reload

• reboot

• after reboot test the service is operational

pm2 ls - should display list of running instances having status online

pm2 applying changes

• pm2 ls && pm2 stop all && pm2 delete all

• pm2 save

• sudo systemctl restart pm2-$USER

• pm2 ls - verify pm2 started successfuly the app instances with new pids

Test connectivity

https://example.com:2053/status - should return 200 OK

https://example.com:8443 - should return 404 "Cannot GET /"

Monitoring

• set ENABLE_DEBUG=1

• rebuild and deploy

• attach to container

pm2 monit and pm2 logs

• TODO export and collect metrics

• TODO collect log stream

Troubleshooting and known issues

pm2 knows issues

• It seems pm2 v5.1.2 has a weird timezone behaviour, as seen from .pm2/pm2.log:

  An example from sequent restarts of pm2-$USER service:

  • 2021-12-19T14:53:56: PM2 log: --- New PM2 Daemon started - has the correct TZ in UTC
  • 2021-12-19T18:24:12: PM2 log: --- New PM2 Daemon started - the next one executed right after the previous one has started all of the nodes, but outputs an incorrect UTC TZ

  Thus, auto-refreshing of the auth_token becomes unreliable and the simplest workaround is a root cron job scheduled to restart the pm2-$USER service daily. For instance:

  0 0 * * * systemctl restart pm2-$USER

• pm2 docs - journalctl -u pm2-$USER.service

• stackoverflow 43786412

• If your home directory is encrypted in order to demonize with pm2 startup all the node.js, npm and app raleted files should be outside of the home dir or try link1 link2 link3

Client

• check the client_example dir and socket.io docs

Release

• create branch vX.X.X
• open PR master <- vX.X.X
• merge PR
• wait for github actions
• delete vX.X.X
• draft new release -> create tag -> publish
• git checkout master && git pull
• npm login
• npm publish --access public