WordPress Multisite on CentOS 7.2, Nginx 1.9.5, PHP7, batcache, and HTTP/2

It was time for an overhaul of my go-to WordPress stack. For a long while, I’ve been running PHP 5.4 on CentOS 6 with Varnish. Technical times are changing and it was overdue for an upgrade; so I bring you this!

My first impression of PHP 7 – WOW, Just WOW! The speed increase is phenomenal compared to PHP 5. It seems like the browser is no longer waiting on the server, but the other way around. WordPress is so fast on this setup that it feels like a native app. Perhaps even better/faster than a native app. I almost can’t believe it.

As for HTTP/2 – supposedly this new protocol is faster, but I don’t really see a huge difference with it on versus off. I’m sure its advantages will become apparent over time when we start to integrate more technology into our websites.

I decided to go away from varnish since I’m now running Batcache. Previously, I was a fan of W3TC, but have migrated away from that plugin (it’s become a bloated/buymebuyme nightmare now). This means you lose support for gzip, browser caching, and CDN settings from within WordPress, but you don’t really need to worry about those settings after they’re manually set up in Nginx anyways. Anyways, Batcache versus Varnish – I don’t really see a huge difference. If anything, the cache invalidation is much easier with Batcache. It’s a set-it-and-forget-it type of system and it “just works”.

On one of my production machines running this stack, I show load averages dropping from 3-4 down to less than 1. This is on a 16-core server, so that’s a significant performance gain. Again – I almost can’t believe it, but the numbers speak for themselves. As for memory usage, I don’t see much difference, but everything looks good and clean.

Specifications

Prerequisites

sudo yum update # update the system
sudo iptables -F # Flush iptables - we'll rebuild later
sudo yum install epel-release # Install EPEL repo
sudo yum install https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
sudo yum install zlib-devel make gcc

Now, download the latest nginx source to a convenient location and compile it.

The Nginx Build

./configure \
--user=nginx \
--group=nginx \
--prefix=/etc/nginx \
--sbin-path=/usr/sbin/nginx \
--conf-path=/etc/nginx/nginx.conf \
--pid-path=/var/run/nginx.pid \
--lock-path=/var/run/nginx.lock \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--with-http_gzip_static_module \
--with-http_stub_status_module \
--with-http_ssl_module \
--with-pcre \
--with-file-aio \
--with-http_realip_module \
--with-http_v2_module \
--without-http_scgi_module \
--without-http_uwsgi_module

Then, of course, make and make install and nginx should compile. You can test this with /usr/sbin/nginx -c /etc/nginx/nginx.conf. You’ll want to kill that process after you test it.

Next, you need to be able to start/stop/reload nginx. Add this script to /etc/init.d/nginx:

#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig:   - 85 15
# description:  NGINX is an HTTP(S) server, HTTP(S) reverse \
#               proxy and IMAP/POP3 proxy server
# processname: nginx
# config:      /etc/nginx/nginx.conf
# config:      /etc/sysconfig/nginx
# pidfile:     /var/run/nginx.pid

# Source function library.
. /etc/rc.d/init.d/functions

# Source networking configuration.
. /etc/sysconfig/network

# Check that networking is up.
[ "$NETWORKING" = "no" ] & exit 0

nginx="/usr/sbin/nginx"
prog=$(basename $nginx)

NGINX_CONF_FILE="/etc/nginx/nginx.conf"

[ -f /etc/sysconfig/nginx ] & . /etc/sysconfig/nginx

lockfile=/var/lock/subsys/nginx

make_dirs() {
   # make required directories
   user=`$nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`
   if [ -z "`grep $user /etc/passwd`" ]; then
       useradd -M -s /bin/nologin $user
   fi
   options=`$nginx -V 2>&1 | grep 'configure arguments:'`
   for opt in $options; do
       if [ `echo $opt | grep '.*-temp-path'` ]; then
           value=`echo $opt | cut -d "=" -f 2`
           if [ ! -d "$value" ]; then
               # echo "creating" $value
               mkdir -p $value & chown -R $user $value
           fi
       fi
   done
}

start() {
    [ -x $nginx ] || exit 5
    [ -f $NGINX_CONF_FILE ] || exit 6
    make_dirs
    echo -n $"Starting $prog: "
    daemon $nginx -c $NGINX_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] & touch $lockfile
    return $retval
}

stop() {
    echo -n $"Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] & rm -f $lockfile
    return $retval
}

restart() {
    configtest || return $?
    stop
    sleep 1
    start
}

reload() {
    configtest || return $?
    echo -n $"Reloading $prog: "
    killproc $nginx -HUP
    RETVAL=$?
    echo
}

force_reload() {
    restart
}

configtest() {
  $nginx -t -c $NGINX_CONF_FILE
}

rh_status() {
    status $prog
}

rh_status_q() {
    rh_status > /dev/null 2>&1
}

case "$1" in
    start)
        rh_status_q & exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart|configtest)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
            ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
        exit 2
esac

Okay, the hard part is over with. If you can start and stop nginx with service nginx start and service nginx stop, then you’re good to go. Time to have some fun. Let’s install some more software.

# Install memcached
sudo yum install -y memcached

# Install mariadb (mysql)
sudo yum install -y mariadb-server mariadb-client

# Install PHP 7
sudo yum install -y php70w php70w-fpm php70w-mysql php70w-opcache php70w-devel php70w-gd php70w-mbstring php70w-xml

# Install ntp, set timezone, set date
sudo yum install -y ntp ntpdate
sudo mv /etc/localtime /etc/localtime.bk
sudo ln -s /usr/share/zoneinfo/America/New_York /etc/localtime
sudo ntpdate pool.ntp.org

# Install some other tools that you'll want to have ready
sudo yum install -y vim htop screen

Configure Nginx

Now you’ll want to set up your nginx.conf and conf.d/* files how you want them. Some key things to remember:

  • If you want to use HTTP/2, you’ll need SSL. If you want free (and easy) SSL certificates, check out LetsEncrypt.
  • On WordPress Multisite (subdomain installs), you’ll want a certificate for each subdomain and each mapped domain. That means you will need a minimum of two certificates for your root domain, and two certificates for each subsequent mapped domain. Each *.yourdomain.com subdomain requires it’s own certificate, and each domain you map to a subdomain requires yet another.

The default nginx.conf is probably okay, though nginx may complain about one or more listen directives (if so, just change whatever it says to change). Here a sample default.conf that includes WordPress rewrite support, concatenation, browser caching, and gzip:

server {
	listen 80;
	server_name {{ actual_hostname }};

	root {{ actual_webroot }};
	index index.php index.html index.htm;

	client_max_body_size 100M;

	gzip on;
	gzip_disable "msie6";
	gzip_vary on;
	gzip_proxied any;
	gzip_comp_level 6;
	gzip_buffers 16 8k;
	gzip_http_version 1.1;
	gzip_types text/plain application/json application/x-javascript application/xml application/xml+rss text/javascript;

	error_page 404 /404.html;
	location /404.html {
		root {{ actual_webroot }};
	}

	error_page 500 502 503 504 /50x.html;
	location = /50x.html {
		root {{ actual_webroot }};
	}

	location / {
		try_files $uri $uri/ /index.php?q=$uri&$args;
	}

	# Allow access to script concatenation engine
	location /_static/ {
		fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
		include /etc/nginx/fastcgi_params;
		fastcgi_param SCRIPT_FILENAME $document_root/wp-content/mu-plugins/http-concat/ngx-http-concat.php;
		include fastcgi_params;
	}

	# Browser cache static assets and do not access log
	location ~* \.(jpg|jpeg|gif|png|css|js|ico|svg)$ {
		expires max;
		access_log off;
		log_not_found off;
	}

	# Block access to PHP files in uploads
	location ~* /(?:uploads|files)/.*\.php$ {
		deny all;
	}

	# Whitelist IPs for nginx status
	location /nginx_status {
		stub_status on;
		access_log off;
		allow 127.0.0.1;
		deny all;
	}

	# All PHP files
	location ~ \.php$ {
		try_files $uri $uri/ /index.php?q=$uri&$args;
		fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
		fastcgi_index index.php;
		fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
		include fastcgi_params;
	}
}

This file came from my ansible install.yml so you’ll need to make a few obvious string replacements. You’ll also want to duplicate this into ssl.conf and modify accordingly to accommodate your domain(s) and SSL requirements. Activating HTTP/2 is as easy as modifying the listen directive to reflect listen 443 ssl http2;.

There are a ton of other configuration items to address and tweak, such as php.ini and /etc/php-fpm.d/www.conf, and don’t forget to set up /etc/sysconfig/memcached… but I’ll leave those fine details to you. Remember to build your firewall also.

As always, if you need pointers, feel free to hit me up in the comments.

Get Up and Running with LetsEncrypt.org Certificates on CentOS 6 and nginx

If you don’t have git installed, use Yum to install it.

sudo yum install git

Stop nginx, if it’s running. LetsEncrypt needs to bind to port 80 and/or 443 to verify your web server.

sudo service nginx stop

Switch to root and git clone the letsencrypt repository.

sudo su
cd ~
git clone https://github.com/letsencrypt/letsencrypt

Now cd to letsencrypt and run letsencrypt-auto.

cd letsencrypt
./letsencrypt-auto certonly --debug

You’ll be greeted with several questions, first to enter your email address, second to agree to the TOS, and third to enter the domain name that you wish to encrypt.

letsencrypt0

letsencrypt1

letsencrypt2

Note: At this time, even though the prompt specifies you can input multiple domain names, it only works with a single domain name.

LetsEncrypt will do it’s thing for about 10-20 seconds and you’ll be dropped back to the command line with several cryptic looking messages. Don’t stress, do this:

cd /etc/letsencrypt/live/
ls -lha

You should now be looking at a directory for your domain name. If you cd into it, you should be looking at several pem files:

  • cert.pem – This is your certificate file
  • chain.pem – This is your certificate chain
  • fullchain.pem – This is the full certificate chain
  • privkey.pem – This is your private key file

If you see all of those files, you’re in good shape! Time to configure nginx.

Now, nginx configurations come in many flavors, and mine is probably much different than yours, so I’ll just give you a crash course here. Feel free to ask specific questions in the comments below and I’d be happy to help you along.

Navigate to /etc/nginx/conf.d and create a copy of your NON-SSL configuration file (call it whatever you want, just append .conf to it). Depending on how you’re set up, it could be default.conf or virtual.conf. You want to modify whichever configuration file that contains your server blocks. Now vi/vim/nano your copy, modify the listen directive inside your server block to reflect the following:

listen 443 ssl;

Now add a few lines below your server_name directive as follows:

ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;

Obviously use the correct domain name in the ssl_certificate and ssl_certificate_key directives. The entire thing should look something like this:

server {
	listen 443 ssl;
	server_name yourdomain.com;
	ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
	...

Save the file, take a deep breath, and issue the following command:

sudo service nginx configtest

If you see syntax is ok and test is successful, start nginx by typing

sudo service nginx start

Now your domain should be accessible via HTTPS! Check it out at https://yourdomain.com.

Now, this is only the beginning of your journey to SSL integration. There are likely several tasks left that I suggest you take care of:

  1. Don’t disable vanilla HTTP yet, but maybe redirect HTTP traffic to it’s corresponding HTTPS URL. There are several ways to do this, but I suggest an nginx approach.
  2. If you’re on WordPress or another CMS, you will likely need to reconfigure your site’s URL within that system. On WordPress, there are a number of ways to do this, my favorite is define( 'wp_home', 'https://yourdomain.com' ); and define( 'wp_siteurl', 'https://yourdomain.com' ); which can be placed inside wp-config.php in your sites root. You may also want to install a plugin to handle all the fancy SSL stuff (like str_replace()ing all your content urls so you don’t get mixed content warnings). I recommend Really Simple SSL by Rogier Lankhorst – it worked right out of the box on my WP Multisite.
  3. If you’re on a WordPress multisite, or another site that has more than one domain name, you should generate a certificate for each of those domain names. For instance, on WP Multisite with subdomains enabled and domain mapping, you will have two domain names which need SSL protection, one for the back end and one for the front end.

In future releases of the LetsEncrypt.org libraries, generation of SSL certificates will be far more automated. This is what worked for me in the public beta phase, but there are probably other methods that work as well. Feel free to share what works for you below.

Footnotes:

  • LetsEncrypt.org certificates currently have a 3-month lifetime. This means you’ll need to renew your certificate quarterly for now. But it’s very easy to do so, and as mentioned above, there will be some pretty cool automation coming soon to make this seamless.
  • If you are indeed running WordPress, get excited – Zack Tollman is working on a plugin to integrate WP-CLI with LetsEncrypt, which will make generating and configuring certificates with WordPress as easy as wp cert new!

Now go forth and be secure!

Web Server iptables Script

Here is an iptables script to set up a solid firewall on a CentOS web server. Remember to change the ip address on line 26 with your IP address, or you will lock yourself out of your own server!

#!/bin/bash

# Backup current iptables rules
iptables-save > /etc/sysconfig/iptables-previous

# Flush tables
iptables -F

# Block null packets
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP

# Reject syn-flood attacks
iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP

# Reject XMAS packets
iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP

# Allow specific traffic
iptables -A INPUT -i lo -j ACCEPT

# HTTP and HTTPS traffic
iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

# SSH from specified clients
iptables -A INPUT -p tcp -s 123.45.67.89 -m tcp --dport 22 -j ACCEPT

# Allow related connections
iptables -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow outgoing connections
iptables -P OUTPUT ACCEPT

# Block all other connections
iptables -P INPUT DROP

# Save iptables and restart service
iptables-save | sudo tee /etc/sysconfig/iptables

NFS Server & Client Setup on CentOS 6.4

Install NFS on the server

sudo su
yum install nfs-utils nfs-utils-lib
chkconfig nfs on
service rpcbind start
service nfs start

Edit /etc/exports on the server

/directory/to/share 10.0.0.2(rw,sync,no_root_squash,no_subtree_check)
/directory/to/share 10.0.0.3(rw,sync,no_root_squash,no_subtree_check)
/directory/to/share 10.0.0.4(rw,sync,no_root_squash,no_subtree_check)
/another/shared/dir 10.0.0.2(rw,sync,no_root_squash,no_subtree_check)
/another/shared/dir 10.0.0.3(rw,sync,no_root_squash,no_subtree_check)
/another/shared/dir 10.0.0.4(rw,sync,no_root_squash,no_subtree_check)

Export the shares from bash

exportfs -a

Install NFS on the client(s)

sudo su
yum install nfs-utils nfs-utils-lib
chkconfig nfs on
service rpcbind start
service nfs start
mkdir -p /directory/to/share
mount 10.0.0.1:/directory/to/share /directory/to/share
mount 10.0.0.1:/another/shared/dir/ /another/shared/dir

If there are no errors, check to see if your mount is active on the client with:

df -h

Test the NFS Mount from the client

touch /directory/to/share/somefile.txt

And now that file should be on both server and client. All Done!

Graphite Graphs for WordPress

It’s no secret that I love Graphite. My co-workers think it’s a bit ridiculous how much data I feed into our graphite server. I may be a bit excessive, but I have a monitor in my office which allows me to see at a glance every little detail about our network in real time – server health, pageviews, Google Pagespeed Insight scores, etc. I can’t even describe how wonderful a tool it is.

I wanted an easy way to display graphite graphs on the WordPress dashboard. So I rolled out a plugin over the weekend. I give you WordPress Graphite Graphs – I admit I [somewhat] got the idea from WPVIP’s “page generation time” graph in the VIP dashboard, but I’m sure they won’t mind. This plugin allows me (or you) to rapidly deploy monitoring graphs across many different sites without a lot of hassle and NO custom code for each site. Just plug in the URL of your graphite server, the metrics you want to display, and viola – Graphy delight! While I was at it, I went ahead and added support for graphs on the front-end in the sidebar that can be hidden from non-users if needed. Why not, right?

Shoot me a tweet if you’re interested in additional features or get in on the action and fork it on github.

A little PHP script to watch for 404 errors

This is a quick script which reads a standard RSS feed and then grabs the response header for each feed item and makes a log file of 404 errors. Useful in debugging sticky situations.

<?php 

$link_queue = array(); 
$exception_queue = array(); 

$timestamp = date( 'Y-m-d H:i:s' ); 
$rss = simplexml_load_file( $argv[1] ); 

foreach ( $rss-&gt;channel-&gt;item as $item )
  array_push( $link_queue, $item-&gt;link );

foreach ( $link_queue as $url ) {
  $handle = curl_init( $url );
  curl_setopt( $handle, CURLOPT_RETURNTRANSFER, TRUE );
  $response = curl_exec( $handle );
  $response_code = curl_getinfo( $handle, CURLINFO_HTTP_CODE );

  if ( $response_code !== 200 )
    array_push( $exception_queue, &quot;{$url} returned status code {$response_code}&quot; );

  curl_close( $handle );
}

if ( count( $exception_queue ) ) {
  foreach ( $exception_queue as $e )
    echo $timestamp . ' &gt; ' . $e . &quot;n&quot;;
} else {
  echo $timestamp . &quot; &gt; All URLs returned status code 200n&quot;;
}

die();

// omit

Throw it in cron like this:
*/30 * * * * php /path/to/script.php http://somesite.com/feed/ >> /path/to/log 2>&1

Or run it on the command line like this:
php /path/to/the/script.php http://somesite.com/feed/

Some Helpful Varnish Commands

Look at incoming requests for URL
varnishlog -c -m RxURL:"/somepage.html"

Look at requests made to the backend for URL
varnishlog -b -m TxURL:"/somepage.html"

Requests for one specific host
varnishlog -c -m RxHeader:"Host: somehost.com"

See cache age for a specific host
varnishlog -c -m RxHeader:"Host: somehost.com" | grep Age

Test VCL compilation
varnishd -C -f /path/to/file.vcl

Ban all URLs
varnishadm -S /path/to/secret -T :6082 "ban.url /"

View Ban List
varnishadm -S /path/to/secret -T :6082 "ban.list"

Real time URLs being sent to backend
varnishtop -b -i TxURL

List backend traffic
varnishlog -b -o

List URLs going to backend
varnishlog -b -o -i TxURL | grep TxURL | awk '{print $4}'