I'm running 8 services on one server with PM2 and an Apache reverse proxy
I run several services on one server with PM2 and split them by domain with an Apache reverse proxy. Covers VirtualHost config, WebSocket proxying and a wildcard certificate.
#Infrastructure #Apache #PM2 #Reverse proxy
Type pm2 list once and eight rows come up: blog, crypto, couples, game, monitoring, portfolio, tarot and travel planner. The domains are all different, but in reality they all run on a single computer. At first I wondered whether this would even work, but once I tried it, it turned out to be no big deal once the structure was in place. In this post I'll go over how I start each service and how I tie it to a domain. Each service runs on its own port First I gave each Node server a port of its own. Blog 5000, game 5005, crypto 5006, couples 5010, travel 5011, portfolio 5015, monitoring 5020, tarot 5025, and so on, so that nothing overlaps. There's no rule to it, I just assigned them in the 5000s in the order I built things. But if you start these with plain node app.js, they die the moment you close the terminal. So I manage them with PM2 . It runs them in the background, restarts them automatically if they die, and brings them back up by itself even when the server reboots. # register a service pm2 start app.js --name blog-server # see the overall status at a glance pm2 list # restart after changing code pm2 restart blog-server # come back up automatically after a reboot pm2 save pm2 startup Typing pm2 list shows a table with whether each service is online, how much memory it's using and how many times it has restarted. This screenshot was taken now, long after I wrote this post, and it's handy because this one view tells me the state of the whole server. In the restart count column of the table, game-server is at 113 and travel-server is at 99. I went a while without looking at these numbers and got burned pretty badly later, which I wrote up separately in the PM2 crash loop post. So who connects the domains? The real problem starts here. Users come in through coin.jaeyonging.com, not through port 5006. Something has to hand requests that arrive on 80 and 443 over to the right internal port, and the Apache reverse proxy does that job. You create one VirtualHost per domain and pass requests for that domain to the matching port. The config for the crypto site looks roughly like this. VirtualHost *:443 ServerName coin.jaeyonging.com SSLEngine on SSLCertificateFile /.../wildcard.jaeyonging.com.crt.pem SSLCertificateKeyFile /.../wildcard.jaeyonging.com.key.pem ProxyPreserveHost On ProxyPass / http://localhost:5006/ ProxyPassReverse / http://localhost:5006/ /VirtualHost The core of it is the single line ProxyPass / http://localhost:5006/ . It passes every request for the coin domain to internal port 5006. With 8 domains, you make 8 of these VirtualHosts. ProxyPreserveHost On is an option that passes along the original request's Host header as is, and it has to be on for the Node side to know which domain the request came in on. The VirtualHost for port 80 is kept separately, and all it does is a 301 redirect to https. So each domain ends up with two config files. WebSockets need separate attention The crypto site uses Socket.IO for live chat and prices, and WebSockets connect differently from regular HTTP, so I had to set up the proxy for them separately. Without this the chat keeps dropping. RewriteEngine On RewriteCond %{HTTP:Upgrade} websocket [NC] RewriteCond %{HTTP:Connection} upgrade [NC] RewriteRule ^/?(.*) ws://localhost:5006/$1 [P,L] ProxyPass /socket.io/ http://localhost:5006/socket.io/ ProxyPassReverse /socket.io/ http://localhost:5006/socket.io/ When the Upgrade header is websocket, it's passed separately over the ws:// protocol, and only after adding this did the realtime connection become stable. If you're putting a service that uses WebSockets behind a proxy, this is close to mandatory. Later, when I added SSE to the monitoring server, I tripped over the proxy once more in a similar way, but that's a story about the flushpackets option, so I wrote it up separately in the SSE post. SSL with a single wildcard certificate Having several domains didn't mean I needed several certificates. A single *.jaeyonging.com wildcard certificate covers every subdomain. Each VirtualHost just has to point at the same certificate path. As a bonus I also set the HSTS header to force browsers to connect over HTTPS only. To see which service is in what state right now, I look at a monitoring page I built separately. It keeps the response time and 90 days of uptime history for each domain. It's not 8 anymore It was 8 when I wrote this post, but counting sites-enabled now, there are 24 config files. Each service has one for port 80 and one for port 443, so 12 domains became 24 files. On the PM2 side there are 10 running as well, and one of them is just named node, so I need to take another look at what it is. As they grew, it started to get confusing. I can't tell from the file names alone which one goes to which port, so these days I start with grep before changing anything. grep -rn "ProxyPass " /etc/apache2/sites-enabled/*-ssl.conf | grep -v Reverse Running it gives this. This one line works as a map of the whole server. If you look, default-ssl.conf, in other words the root domain, points to the blog (5000) and not the portfolio (5015). I switched it temporarily for the AdSense review, forgot about it, and only found out much later that auto deploy had stopped. I wrote about that at the end of the deploy post. And there's one more thing called catchall. I made it to catch requests for domains that don't match anything, but I can't remember now what I actually meant to do with it. This is what happens once you pass twenty config files. Around the point where you go past three or four services, I'd recommend making a table with one line per domain saying which port it goes to.