Category: Linux

  • Get disk usage information for MySQL databases

    Ever wonder how much space your MySQL database uses? Here you go:

    SELECT table_schema "Database",
    SUM( data_length + index_length ) / 1024 / 1024 "MB used",
    SUM( data_free ) / 1024 / 1024 "MB free"
    FROM information_schema.TABLES
    GROUP BY table_schema;
  • WordPress Multisite on CentOS 7.2, Nginx 1.9.5, PHP7, batcache, and HTTP/2

    It was time for an overhaul of my go-to WordPress stack. For a long while, I’ve been running PHP 5.4 on CentOS 6 with Varnish. Technical times are changing and it was overdue for an upgrade; so I bring you this!

    My first impression of PHP 7 – WOW, Just WOW! The speed increase is phenomenal compared to PHP 5. It seems like the browser is no longer waiting on the server, but the other way around. WordPress is so fast on this setup that it feels like a native app. Perhaps even better/faster than a native app. I almost can’t believe it.

    As for HTTP/2 – supposedly this new protocol is faster, but I don’t really see a huge difference with it on versus off. I’m sure its advantages will become apparent over time when we start to integrate more technology into our websites.

    I decided to go away from varnish since I’m now running Batcache. Previously, I was a fan of W3TC, but have migrated away from that plugin (it’s become a bloated/buymebuyme nightmare now). This means you lose support for gzip, browser caching, and CDN settings from within WordPress, but you don’t really need to worry about those settings after they’re manually set up in Nginx anyways. Anyways, Batcache versus Varnish – I don’t really see a huge difference. If anything, the cache invalidation is much easier with Batcache. It’s a set-it-and-forget-it type of system and it “just works”.

    On one of my production machines running this stack, I show load averages dropping from 3-4 down to less than 1. This is on a 16-core server, so that’s a significant performance gain. Again – I almost can’t believe it, but the numbers speak for themselves. As for memory usage, I don’t see much difference, but everything looks good and clean.

    Specifications

    Prerequisites

    sudo yum update # update the system
    sudo iptables -F # Flush iptables - we'll rebuild later
    sudo yum install epel-release # Install EPEL repo
    sudo yum install https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
    sudo yum install zlib-devel make gcc

    Now, download the latest nginx source to a convenient location and compile it.

    The Nginx Build

    ./configure \
    --user=nginx \
    --group=nginx \
    --prefix=/etc/nginx \
    --sbin-path=/usr/sbin/nginx \
    --conf-path=/etc/nginx/nginx.conf \
    --pid-path=/var/run/nginx.pid \
    --lock-path=/var/run/nginx.lock \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --with-http_gzip_static_module \
    --with-http_stub_status_module \
    --with-http_ssl_module \
    --with-pcre \
    --with-file-aio \
    --with-http_realip_module \
    --with-http_v2_module \
    --without-http_scgi_module \
    --without-http_uwsgi_module

    Then, of course, make and make install and nginx should compile. You can test this with /usr/sbin/nginx -c /etc/nginx/nginx.conf. You’ll want to kill that process after you test it.

    Next, you need to be able to start/stop/reload nginx. Add this script to /etc/init.d/nginx:

    #!/bin/sh
    #
    # nginx - this script starts and stops the nginx daemon
    #
    # chkconfig:   - 85 15
    # description:  NGINX is an HTTP(S) server, HTTP(S) reverse \
    #               proxy and IMAP/POP3 proxy server
    # processname: nginx
    # config:      /etc/nginx/nginx.conf
    # config:      /etc/sysconfig/nginx
    # pidfile:     /var/run/nginx.pid
    
    # Source function library.
    . /etc/rc.d/init.d/functions
    
    # Source networking configuration.
    . /etc/sysconfig/network
    
    # Check that networking is up.
    [ "$NETWORKING" = "no" ] & exit 0
    
    nginx="/usr/sbin/nginx"
    prog=$(basename $nginx)
    
    NGINX_CONF_FILE="/etc/nginx/nginx.conf"
    
    [ -f /etc/sysconfig/nginx ] & . /etc/sysconfig/nginx
    
    lockfile=/var/lock/subsys/nginx
    
    make_dirs() {
       # make required directories
       user=`$nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`
       if [ -z "`grep $user /etc/passwd`" ]; then
           useradd -M -s /bin/nologin $user
       fi
       options=`$nginx -V 2>&1 | grep 'configure arguments:'`
       for opt in $options; do
           if [ `echo $opt | grep '.*-temp-path'` ]; then
               value=`echo $opt | cut -d "=" -f 2`
               if [ ! -d "$value" ]; then
                   # echo "creating" $value
                   mkdir -p $value & chown -R $user $value
               fi
           fi
       done
    }
    
    start() {
        [ -x $nginx ] || exit 5
        [ -f $NGINX_CONF_FILE ] || exit 6
        make_dirs
        echo -n $"Starting $prog: "
        daemon $nginx -c $NGINX_CONF_FILE
        retval=$?
        echo
        [ $retval -eq 0 ] & touch $lockfile
        return $retval
    }
    
    stop() {
        echo -n $"Stopping $prog: "
        killproc $prog -QUIT
        retval=$?
        echo
        [ $retval -eq 0 ] & rm -f $lockfile
        return $retval
    }
    
    restart() {
        configtest || return $?
        stop
        sleep 1
        start
    }
    
    reload() {
        configtest || return $?
        echo -n $"Reloading $prog: "
        killproc $nginx -HUP
        RETVAL=$?
        echo
    }
    
    force_reload() {
        restart
    }
    
    configtest() {
      $nginx -t -c $NGINX_CONF_FILE
    }
    
    rh_status() {
        status $prog
    }
    
    rh_status_q() {
        rh_status > /dev/null 2>&1
    }
    
    case "$1" in
        start)
            rh_status_q & exit 0
            $1
            ;;
        stop)
            rh_status_q || exit 0
            $1
            ;;
        restart|configtest)
            $1
            ;;
        reload)
            rh_status_q || exit 7
            $1
            ;;
        force-reload)
            force_reload
            ;;
        status)
            rh_status
            ;;
        condrestart|try-restart)
            rh_status_q || exit 0
                ;;
        *)
            echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
            exit 2
    esac

    Okay, the hard part is over with. If you can start and stop nginx with service nginx start and service nginx stop, then you’re good to go. Time to have some fun. Let’s install some more software.

    # Install memcached
    sudo yum install -y memcached
    
    # Install mariadb (mysql)
    sudo yum install -y mariadb-server mariadb-client
    
    # Install PHP 7
    sudo yum install -y php70w php70w-fpm php70w-mysql php70w-opcache php70w-devel php70w-gd php70w-mbstring php70w-xml
    
    # Install ntp, set timezone, set date
    sudo yum install -y ntp ntpdate
    sudo mv /etc/localtime /etc/localtime.bk
    sudo ln -s /usr/share/zoneinfo/America/New_York /etc/localtime
    sudo ntpdate pool.ntp.org
    
    # Install some other tools that you'll want to have ready
    sudo yum install -y vim htop screen

    Configure Nginx

    Now you’ll want to set up your nginx.conf and conf.d/* files how you want them. Some key things to remember:

    • If you want to use HTTP/2, you’ll need SSL. If you want free (and easy) SSL certificates, check out LetsEncrypt.
    • On WordPress Multisite (subdomain installs), you’ll want a certificate for each subdomain and each mapped domain. That means you will need a minimum of two certificates for your root domain, and two certificates for each subsequent mapped domain. Each *.yourdomain.com subdomain requires it’s own certificate, and each domain you map to a subdomain requires yet another.

    The default nginx.conf is probably okay, though nginx may complain about one or more listen directives (if so, just change whatever it says to change). Here a sample default.conf that includes WordPress rewrite support, concatenation, browser caching, and gzip:

    server {
    	listen 80;
    	server_name {{ actual_hostname }};
    
    	root {{ actual_webroot }};
    	index index.php index.html index.htm;
    
    	client_max_body_size 100M;
    
    	gzip on;
    	gzip_disable "msie6";
    	gzip_vary on;
    	gzip_proxied any;
    	gzip_comp_level 6;
    	gzip_buffers 16 8k;
    	gzip_http_version 1.1;
    	gzip_types text/plain application/json application/x-javascript application/xml application/xml+rss text/javascript;
    
    	error_page 404 /404.html;
    	location /404.html {
    		root {{ actual_webroot }};
    	}
    
    	error_page 500 502 503 504 /50x.html;
    	location = /50x.html {
    		root {{ actual_webroot }};
    	}
    
    	location / {
    		try_files $uri $uri/ /index.php?q=$uri&$args;
    	}
    
    	# Allow access to script concatenation engine
    	location /_static/ {
    		fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
    		include /etc/nginx/fastcgi_params;
    		fastcgi_param SCRIPT_FILENAME $document_root/wp-content/mu-plugins/http-concat/ngx-http-concat.php;
    		include fastcgi_params;
    	}
    
    	# Browser cache static assets and do not access log
    	location ~* \.(jpg|jpeg|gif|png|css|js|ico|svg)$ {
    		expires max;
    		access_log off;
    		log_not_found off;
    	}
    
    	# Block access to PHP files in uploads
    	location ~* /(?:uploads|files)/.*\.php$ {
    		deny all;
    	}
    
    	# Whitelist IPs for nginx status
    	location /nginx_status {
    		stub_status on;
    		access_log off;
    		allow 127.0.0.1;
    		deny all;
    	}
    
    	# All PHP files
    	location ~ \.php$ {
    		try_files $uri $uri/ /index.php?q=$uri&$args;
    		fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
    		fastcgi_index index.php;
    		fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    		include fastcgi_params;
    	}
    }

    This file came from my ansible install.yml so you’ll need to make a few obvious string replacements. You’ll also want to duplicate this into ssl.conf and modify accordingly to accommodate your domain(s) and SSL requirements. Activating HTTP/2 is as easy as modifying the listen directive to reflect listen 443 ssl http2;.

    There are a ton of other configuration items to address and tweak, such as php.ini and /etc/php-fpm.d/www.conf, and don’t forget to set up /etc/sysconfig/memcached… but I’ll leave those fine details to you. Remember to build your firewall also.

    As always, if you need pointers, feel free to hit me up in the comments.

  • Get Up and Running with LetsEncrypt.org Certificates on CentOS 6 and nginx

    If you don’t have git installed, use Yum to install it.

    sudo yum install git

    Stop nginx, if it’s running. LetsEncrypt needs to bind to port 80 and/or 443 to verify your web server.

    sudo service nginx stop

    Switch to root and git clone the letsencrypt repository.

    sudo su
    cd ~
    git clone https://github.com/letsencrypt/letsencrypt

    Now cd to letsencrypt and run letsencrypt-auto.

    cd letsencrypt
    ./letsencrypt-auto certonly --debug

    You’ll be greeted with several questions, first to enter your email address, second to agree to the TOS, and third to enter the domain name that you wish to encrypt.

    letsencrypt0

    letsencrypt1

    letsencrypt2

    Note: At this time, even though the prompt specifies you can input multiple domain names, it only works with a single domain name.

    LetsEncrypt will do it’s thing for about 10-20 seconds and you’ll be dropped back to the command line with several cryptic looking messages. Don’t stress, do this:

    cd /etc/letsencrypt/live/
    ls -lha

    You should now be looking at a directory for your domain name. If you cd into it, you should be looking at several pem files:

    • cert.pem – This is your certificate file
    • chain.pem – This is your certificate chain
    • fullchain.pem – This is the full certificate chain
    • privkey.pem – This is your private key file

    If you see all of those files, you’re in good shape! Time to configure nginx.

    Now, nginx configurations come in many flavors, and mine is probably much different than yours, so I’ll just give you a crash course here. Feel free to ask specific questions in the comments below and I’d be happy to help you along.

    Navigate to /etc/nginx/conf.d and create a copy of your NON-SSL configuration file (call it whatever you want, just append .conf to it). Depending on how you’re set up, it could be default.conf or virtual.conf. You want to modify whichever configuration file that contains your server blocks. Now vi/vim/nano your copy, modify the listen directive inside your server block to reflect the following:

    listen 443 ssl;

    Now add a few lines below your server_name directive as follows:

    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    

    Obviously use the correct domain name in the ssl_certificate and ssl_certificate_key directives. The entire thing should look something like this:

    server {
    	listen 443 ssl;
    	server_name yourdomain.com;
    	ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    	ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    	...
    

    Save the file, take a deep breath, and issue the following command:

    sudo service nginx configtest

    If you see syntax is ok and test is successful, start nginx by typing

    sudo service nginx start

    Now your domain should be accessible via HTTPS! Check it out at https://yourdomain.com.

    Now, this is only the beginning of your journey to SSL integration. There are likely several tasks left that I suggest you take care of:

    1. Don’t disable vanilla HTTP yet, but maybe redirect HTTP traffic to it’s corresponding HTTPS URL. There are several ways to do this, but I suggest an nginx approach.
    2. If you’re on WordPress or another CMS, you will likely need to reconfigure your site’s URL within that system. On WordPress, there are a number of ways to do this, my favorite is define( 'wp_home', 'https://yourdomain.com' ); and define( 'wp_siteurl', 'https://yourdomain.com' ); which can be placed inside wp-config.php in your sites root. You may also want to install a plugin to handle all the fancy SSL stuff (like str_replace()ing all your content urls so you don’t get mixed content warnings). I recommend Really Simple SSL by Rogier Lankhorst – it worked right out of the box on my WP Multisite.
    3. If you’re on a WordPress multisite, or another site that has more than one domain name, you should generate a certificate for each of those domain names. For instance, on WP Multisite with subdomains enabled and domain mapping, you will have two domain names which need SSL protection, one for the back end and one for the front end.

    In future releases of the LetsEncrypt.org libraries, generation of SSL certificates will be far more automated. This is what worked for me in the public beta phase, but there are probably other methods that work as well. Feel free to share what works for you below.

    Footnotes:

    • LetsEncrypt.org certificates currently have a 3-month lifetime. This means you’ll need to renew your certificate quarterly for now. But it’s very easy to do so, and as mentioned above, there will be some pretty cool automation coming soon to make this seamless.
    • If you are indeed running WordPress, get excited – Zack Tollman is working on a plugin to integrate WP-CLI with LetsEncrypt, which will make generating and configuring certificates with WordPress as easy as wp cert new!

    Now go forth and be secure!

  • Web Server iptables Script

    Here is an iptables script to set up a solid firewall on a CentOS web server. Remember to change the ip address on line 26 with your IP address, or you will lock yourself out of your own server!

    #!/bin/bash
    
    # Backup current iptables rules
    iptables-save > /etc/sysconfig/iptables-previous
    
    # Flush tables
    iptables -F
    
    # Block null packets
    iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP
    
    # Reject syn-flood attacks
    iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP
    
    # Reject XMAS packets
    iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP
    
    # Allow specific traffic
    iptables -A INPUT -i lo -j ACCEPT
    
    # HTTP and HTTPS traffic
    iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
    iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
    
    # SSH from specified clients
    iptables -A INPUT -p tcp -s 123.45.67.89 -m tcp --dport 22 -j ACCEPT
    
    # Allow related connections
    iptables -I INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    
    # Allow outgoing connections
    iptables -P OUTPUT ACCEPT
    
    # Block all other connections
    iptables -P INPUT DROP
    
    # Save iptables and restart service
    iptables-save | sudo tee /etc/sysconfig/iptables
  • Nginx Load Balancer Config

    http {
    	upstream myproject {
    		server 10.0.0.2:8000 weight=2;
    		server 10.0.0.3:8000 weight=2;
    		server 10.0.0.4:8000 weight=2;
    	}
    
    	server {
    		listen 80;
    		server_name .mydomain.com
    		location / {
    			proxy_pass http://myproject;
    		}
    	}
    }
  • NFS Server & Client Setup on CentOS 6.4

    Install NFS on the server

    sudo su
    yum install nfs-utils nfs-utils-lib
    chkconfig nfs on
    service rpcbind start
    service nfs start

    Edit /etc/exports on the server

    /directory/to/share 10.0.0.2(rw,sync,no_root_squash,no_subtree_check)
    /directory/to/share 10.0.0.3(rw,sync,no_root_squash,no_subtree_check)
    /directory/to/share 10.0.0.4(rw,sync,no_root_squash,no_subtree_check)
    /another/shared/dir 10.0.0.2(rw,sync,no_root_squash,no_subtree_check)
    /another/shared/dir 10.0.0.3(rw,sync,no_root_squash,no_subtree_check)
    /another/shared/dir 10.0.0.4(rw,sync,no_root_squash,no_subtree_check)

    Export the shares from bash

    exportfs -a

    Install NFS on the client(s)

    sudo su
    yum install nfs-utils nfs-utils-lib
    chkconfig nfs on
    service rpcbind start
    service nfs start
    mkdir -p /directory/to/share
    mount 10.0.0.1:/directory/to/share /directory/to/share
    mount 10.0.0.1:/another/shared/dir/ /another/shared/dir

    If there are no errors, check to see if your mount is active on the client with:

    df -h

    Test the NFS Mount from the client

    touch /directory/to/share/somefile.txt

    And now that file should be on both server and client. All Done!

  • Send load average to Graphite


    #!/bin/bash
    LOAD=`cat /proc/loadavg | awk '{print $1}'`
    DATETIME=`date +%s`
    echo "some.system.metric.name $LOAD $DATETIME" | nc graphite.somedomain.com 2003

  • Graphite Graphs for WordPress

    It’s no secret that I love Graphite. My co-workers think it’s a bit ridiculous how much data I feed into our graphite server. I may be a bit excessive, but I have a monitor in my office which allows me to see at a glance every little detail about our network in real time – server health, pageviews, Google Pagespeed Insight scores, etc. I can’t even describe how wonderful a tool it is.

    I wanted an easy way to display graphite graphs on the WordPress dashboard. So I rolled out a plugin over the weekend. I give you WordPress Graphite Graphs – I admit I [somewhat] got the idea from WPVIP’s “page generation time” graph in the VIP dashboard, but I’m sure they won’t mind. This plugin allows me (or you) to rapidly deploy monitoring graphs across many different sites without a lot of hassle and NO custom code for each site. Just plug in the URL of your graphite server, the metrics you want to display, and viola – Graphy delight! While I was at it, I went ahead and added support for graphs on the front-end in the sidebar that can be hidden from non-users if needed. Why not, right?

    Shoot me a tweet if you’re interested in additional features or get in on the action and fork it on github.

  • A little PHP script to watch for 404 errors

    This is a quick script which reads a standard RSS feed and then grabs the response header for each feed item and makes a log file of 404 errors. Useful in debugging sticky situations.

    <?php 
    
    $link_queue = array(); 
    $exception_queue = array(); 
    
    $timestamp = date( 'Y-m-d H:i:s' ); 
    $rss = simplexml_load_file( $argv[1] ); 
    
    foreach ( $rss-&gt;channel-&gt;item as $item )
      array_push( $link_queue, $item-&gt;link );
    
    foreach ( $link_queue as $url ) {
      $handle = curl_init( $url );
      curl_setopt( $handle, CURLOPT_RETURNTRANSFER, TRUE );
      $response = curl_exec( $handle );
      $response_code = curl_getinfo( $handle, CURLINFO_HTTP_CODE );
    
      if ( $response_code !== 200 )
        array_push( $exception_queue, &quot;{$url} returned status code {$response_code}&quot; );
    
      curl_close( $handle );
    }
    
    if ( count( $exception_queue ) ) {
      foreach ( $exception_queue as $e )
        echo $timestamp . ' &gt; ' . $e . &quot;n&quot;;
    } else {
      echo $timestamp . &quot; &gt; All URLs returned status code 200n&quot;;
    }
    
    die();
    
    // omit
    

    Throw it in cron like this:
    */30 * * * * php /path/to/script.php http://somesite.com/feed/ >> /path/to/log 2>&1

    Or run it on the command line like this:
    php /path/to/the/script.php http://somesite.com/feed/

  • Some Helpful Varnish Commands

    Look at incoming requests for URL
    varnishlog -c -m RxURL:"/somepage.html"

    Look at requests made to the backend for URL
    varnishlog -b -m TxURL:"/somepage.html"

    Requests for one specific host
    varnishlog -c -m RxHeader:"Host: somehost.com"

    See cache age for a specific host
    varnishlog -c -m RxHeader:"Host: somehost.com" | grep Age

    Test VCL compilation
    varnishd -C -f /path/to/file.vcl

    Ban all URLs
    varnishadm -S /path/to/secret -T :6082 "ban.url /"

    View Ban List
    varnishadm -S /path/to/secret -T :6082 "ban.list"

    Real time URLs being sent to backend
    varnishtop -b -i TxURL

    List backend traffic
    varnishlog -b -o

    List URLs going to backend
    varnishlog -b -o -i TxURL | grep TxURL | awk '{print $4}'