- Go to portal.azure.com
- Click "Create a resource" → "Virtual Machine"
-
Fill in:
- Resource Group: Create new → scrapper-rg
- VM Name: scrapper-vm
- Region: Choose nearest (e.g., East US, Central India)
- Image: Ubuntu Server 22.04 LTS
- Size: Standard_B2s (2 vCPUs, 4GB RAM)
- Username: azureuser
- SSH public key source: Generate new key pair
- Key pair name: scrapper-vm_key
-
Click "Review + Create" → "Create"
-
Download the .pem file when prompted and save it safely!
- After VM is created, go to VM → "Networking" (left sidebar)
- Click "Add inbound port rule" and add these 3 rules:
-
Rule 1:
- Destination port: 22
- Name: SSH
-
Rule 2:
- Destination port: 3000
- Name: App_Port
-
Rule 3:
- Destination port: 80
- Name: HTTP
- Go to VM Overview page
- Copy the Public IP address
On your Arch Linux machine:
# Move key to .ssh directory
mkdir -p ~/.ssh
mv ~/Downloads/scrapper-vm_key.pem ~/.ssh/
# Set correct permissions
chmod 400 ~/.ssh/scrapper-vm_key.pem
# Connect to VM (replace <VM_IP> with your actual IP)
ssh -i ~/.ssh/scrapper-vm_key.pem azureuser@<VM_IP>Type yes when asked about fingerprint.
- Step 4: Install Node.js 24
- On VM:
# Update system
sudo apt update && sudo apt upgrade -y
# Install Node.js 24
curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash -
sudo apt install -y nodejs
# Verify installation
node --version # Should show v24.x.x
npm --version
# Install build tools and git
sudo apt install -y build-essential git- Go to github.com → Your profile → Settings
- Developer settings → Personal access tokens → Tokens (classic)
- Generate new token (classic)
- Settings:
Note: Azure VM scrapper
Expiration: 90 days (or No expiration)
Scopes: Check ✅ repo
Click Generate token
⚠️ COPY THE TOKEN (starts with ghp\_...) - You won't see it again!
- On VM:
# Clone using PAT (replace with your details)
git clone https://<GITHUB_USERNAME>:<YOUR_PAT>@github.com/<USERNAME>/<REPO_NAME>.git
# Example:
# git clone https://john:ghp_abc123xyz@github.com/john/scrapper-app.git
# Go to project directory
cd <REPO_NAME># Store credentials for future pulls
git config --global credential.helper store
git config --global user.name "Your Name"
git config --global user.email "your-email@example.com"# Install dependencies
npm install
# Build TypeScript
npm run build
# Create .env file (if you have environment variables)
nano .envnote : In nano editor, add your variables
Test run your app:
node dist/index.js
Test in browser: Open http://<VM_IP>:3000 Stop the app: Press Ctrl+C
# Install PM2 globally
sudo npm install -g pm2
# Start your app with PM2
pm2 start dist/index.js --name scrapper-app
# Make PM2 start on system reboot
pm2 startup systemd
# ⚠️ IMPORTANT: Copy and run the command that PM2 shows
# It looks like: sudo env PATH=$PATH:/usr/bin ...
# Save PM2 process list
pm2 save
# Check app status
pm2 status
# View logs
pm2 logs scrapper-app# Install Nginx
sudo apt install -y nginx
# Create Nginx configuration file
sudo nano /etc/nginx/sites-available/scrapperPaste this configuration:
server {
listen 80;
server_name _;
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_cache_bypass $http_upgrade;
# Timeouts (important for scrappers)
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
}
}# Create symbolic link
sudo ln -s /etc/nginx/sites-available/scrapper /etc/nginx/sites-enabled/
# Remove default configuration
sudo rm /etc/nginx/sites-enabled/default
# Test Nginx configuration
sudo nginx -t
# If test passes, restart Nginx
sudo systemctl restart nginx
# Enable Nginx to start on boot
sudo systemctl enable nginx- Go to Azure Portal → Your VM → Networking
- Add inbound port rule:
- Destination port: 443
- Protocol: TCP
- Priority: 1003
- Name: HTTPS
ssh -i ~/.ssh/scrapper-vm_key.pem azureuser@<VM_IP># Update and install Certbot
sudo apt update
sudo apt install -y certbot python3-certbot-nginxEdit Nginx config
sudo nano /etc/nginx/sites-available/scrapperserver {
listen 80;
server_name scrapper-app.eastasia.cloudapp.azure.com; # ← Change this
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_cache_bypass $http_upgrade;
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
}
}Save: Ctrl+X, Y, Enter
sudo nginx -t
sudo systemctl reload nginxsudo certbot --nginx -d scrapper-app.eastasia.cloudapp.azure.comsudo systemctl status certbot.timer
# Test renewal (dry run - doesn't actually renew)
sudo certbot renew --dry-run