Skip to content

Setup Varnish with NGINX for static site with and admin area

This is for a simple site that has no login so it doesn’t need to remember cookies except for in an admin area.

First install Varnish

1
sudo apt install varnish -y

Then modify the Varnish settings, this will add the hit or miss header, remove cookies on every page that doesn’t include “admin”  allow purging of individual pages and the entire domain if an X-Purge-Method header of regex is included

1
sudo nano /etc/varnish/default.vcl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
sub vcl_recv {
  if (req.url ~ "admin") {
    # Don't cache admin
    return (pass);
  } else {
    unset req.http.cookie;
    unset req.http.Accept-Encoding;
  }
 
  if (req.method == "PURGE") {
    if (req.http.X-Purge-Method == "regex") {
      ban("req.http.host == " + req.http.host);
      return (synth(200, "Full cache cleared"));
    }
    return (purge);
  }
}
 
sub vcl_backend_response {
  if (bereq.url ~ "admin") {
  } else {
   unset beresp.http.Set-Cookie;
  }
}
 
sub vcl_deliver {
  if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS
    set resp.http.X-Cache = "HIT";
  } else {
    set resp.http.X-Cache = "MISS";
  }
  return (deliver);
}

To BAN all cache you would just need to run

1
curl -i -XPURGE  -H "X-Purge-Method: regex" "https://example.com/"

Then adjust Varnish to use port 80, and increase its ram. I’ve seen it recommended to set it to use 75% of your server’s total memory, but I set it for a more conservative 30% at 1G for this server.

1
sudo nano /etc/default/varnish
1
2
DAEMON_OPTS="-a :6081 \
-s malloc,1G"
1
sudo nano /lib/systemd/system/varnish.service
1
On the 'ExecStart' line, leave the varnish port at 6081 and change 256 to 1G

Anytime you make changes to the Varnish settings you need to reload the daemon and restart

1
2
sudo systemctl daemon-reload
sudo systemctl restart varnish

Then, on the NGINX side you essentially have to move the config from 443 to 8080, then proxy pass on 443 through 8080. 8080 port is specified in varnish in file /etc/varnish/default.vcl

1
2
cd /etc/nginx/sites-enabled
sudo nano example.com
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
server {
  listen [::]:443 ssl http2;
  listen 443 ssl http2;
 
  ssl_certificate ...; 
  ssl_certificate_key ...; 
 
  server_name example.com;
  port_in_redirect off;
 
  location / {
    proxy_pass http://127.0.0.1:6081;
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header HTTPS "on";
  }
}
 
server {
  #Varnish proxy
 
  listen 8080;
  listen [::]:8080;
 
  server_name example.com;
  root /www/hosts/example.com/public/;
  index index.php index.html index.htm;
  port_in_redirect off;
 
  location / {
    try_files $uri $uri/ /index.php$is_args$args;
  }
 
  location ~ \.php$ {
    try_files $uri /index.php =404;
    fastcgi_pass unix:/run/php/php7.3-fpm.sock;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
  }
}

Then restart NGINX and your should be good to go

1
2
3
sudo nginx -t
sudo systemctl reset-failed nginx
sudo service nginx restart

Sources

https://www.howtoforge.com/tutorial/ubuntu-nginx-varnish/
https://www.linode.com/docs/guides/use-varnish-and-nginx-to-serve-wordpress-over-ssl-and-http-on-debian-8/
https://stackoverflow.com/questions/65013805/cant-purge-entire-domain-in-varnish-but-can-purge-individual-pages-do-i-have
https://stackoverflow.com/questions/38892021/how-to-clear-complete-cache-in-varnish

Use a private controller method from artisan tinker

Sometimes I’ve found myself with the need to call a controller method by hand within artisan tinker.  It’s not too hard to do, you just need to use make to create an instance of the controller and then you can call a specific method and include parameters as well.

1
2
$controller = app()->make('App\Http\Controllers\Controller');
$return = app()->call([$controller, 'method'], [$variable1, $variable2]);

Use ffmpeg to compress video with multiple tracks to specific size

I had some multi-track videos I needed to compress, and I found this awesome script for using ffmpeg to compress videos to specific sizes:
https://stackoverflow.com/questions/29082422/ffmpeg-video-compression-specific-file-size/61146975#61146975

And it works great!  But, only includes the first track.  So I made a few slight changes to make it include multiple audio tracks, and factor them in to the total size of the end result.

I added this part to calculate the total number of audio tracks.  Courtesy of this answer.

1
2
3
4
5
6
7
#number of audio streams
O_ACOUNT=$(\
 ffprobe \
 -v error \
 -select_streams a \
 -show_entries stream=index \
 -of csv=p=0 "$1" | wc -w)

And, then multiplied the audio_rate by the number of tracks

1
2
3
4
5
6
7
8
# Calculate target video rate - MB -> KiB/s
T_VRATE=$(\
 awk \
 -v size="$T_SIZE" \
 -v duration="$O_DUR" \
 -v audio_rate="$O_ARATE" \
 -v audio_count="$O_ACOUNT" \
 'BEGIN { print ( ( size * 8192.0 ) / ( 1.048576 * duration ) - (audio_rate * audio_count)) }')

And I added -map 0 to the export at the end so it will capture all the audio streams

1
2
3
4
5
6
7
8
9
ffmpeg \
 -i "$1" \
 -c:v libx264 \
 -b:v "$T_VRATE"k \
 -pass 2 \
 -c:a aac \
 -map 0 \
 -b:a "$T_ARATE"k \
 $T_FILE

So the final script ended up as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
#!/bin/bash
#
# Re-encode a video to a target size in MB.
# Example:
# ./this_script.sh video.mp4 15
#
# Filename CANNOT contain any spaces
 
T_SIZE="$2" # target size in MB
T_FILE="${1%.*}-$2MB.mp4" # filename out
 
# Original duration in seconds
O_DUR=$(\
 ffprobe \
 -v error \
 -show_entries format=duration \
 -of csv=p=0 "$1")
 
# Original audio rate
O_ARATE=$(\
 ffprobe \
 -v error \
 -select_streams a:0 \
 -show_entries stream=bit_rate \
 -of csv=p=0 "$1")
 
# Original audio rate in KiB/s
O_ARATE=$(\
 awk \
 -v arate="$O_ARATE" \
 'BEGIN { printf "%.0f", (arate / 1024) }')
 
#number of audio streams
O_ACOUNT=$(\
 ffprobe \
 -v error \
 -select_streams a \
 -show_entries stream=index \
 -of csv=p=0 "$1" | wc -w)
 
# Target size is required to be less than the size of the original audio stream
T_MINSIZE=$(\
 awk \
 -v arate="$O_ARATE" \
 -v duration="$O_DUR" \
 'BEGIN { printf "%.2f", ( (arate * duration) / 8192 ) }')
 
# Equals 1 if target size is ok, 0 otherwise
IS_MINSIZE=$(\
 awk \
 -v size="$T_SIZE" \
 -v minsize="$T_MINSIZE" \
 'BEGIN { print (minsize < size) }')
 
# Give useful information if size is too small
if [[ $IS_MINSIZE -eq 0 ]]; then
 printf "%s\n" "Target size ${T_SIZE}MB is too small!" >&2
 printf "%s %s\n" "Try values larger than" "${T_MINSIZE}MB" >&2
 exit 1
fi
 
# Set target audio bitrate
T_ARATE=$O_ARATE
 
 
# Calculate target video rate - MB -> KiB/s
T_VRATE=$(\
 awk \
 -v size="$T_SIZE" \
 -v duration="$O_DUR" \
 -v audio_rate="$O_ARATE" \
 -v audio_count="$O_ACOUNT" \
 'BEGIN { print ( ( size * 8192.0 ) / ( 1.048576 * duration ) - (audio_rate * audio_count)) }')
 
# Perform the conversion
ffmpeg \
 -y \
 -i "$1" \
 -c:v libx264 \
 -b:v "$T_VRATE"k \
 -pass 1 \
 -an \
 -f mp4 \
 /dev/null \
&& \
ffmpeg \
 -i "$1" \
 -c:v libx264 \
 -b:v "$T_VRATE"k \
 -pass 2 \
 -c:a aac \
 -map 0 \
 -b:a "$T_ARATE"k \
 $T_FILE

Change server terminal command prompt

If you deal with a lot of server, sometimes it can get tricky to keep track of which one you’re on…so a simple way to help that is to color code and name them in the command prompt.

All you need to do is:

  1. nano ~/.bashrc
  2. uncomment #force_color_prompt=yes
  3. Find a line like this:
    1
    
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
  4. And Change it to something like this:
    1
    
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;35;40m\]Staging \u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '

For this I made it pink and added “Staging” before the hostname.  01;32m was the default color and I just needed to change it to 01;35m.  Below is a handy chart I found with all the ubuntu color codes.

VSQru

Sources:

https://askubuntu.com/questions/123268/changing-colors-for-user-host-directory-information-in-terminal-command-prompt
https://askubuntu.com/questions/27314/script-to-display-all-terminal-colors

Automatic Timestamping in MySQL

Setup table schema with automatic timestamping

1
2
3
4
5
6
7
CREATE TABLE `users` (
  `id`              BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
  `full_name`       VARCHAR(100) NOT NULL,
 
  `created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
)

Automatic properties are specified using the DEFAULT CURRENT_TIMESTAMP and ON UPDATE CURRENT_TIMESTAMP clauses in column definitions.

The DEFAULT CURRENT_TIMESTAMP clause assigns the current timestamp as the default value of the column.
The ON UPDATE CURRENT_TIMESTAMP clause updates the column to the current timestamp when the value of any other column in the row is changed from its current value.

CURRENT_TIMESTAMP is a synonym for NOW() and returns the current date and time. It returns a constant time that indicates the time at which a statement began to execute.

 

This was shamelessly stolen :X  I really liked their formatting, and just wanted to make a backup
https://habenamare.com/guides/mysql/automatic-timestamping/