Automate/Gather Statistics for Multiple Websites in BASH

Introduction

For one reason or another, Im sure you will find yourself in a position when you need to obtain statistics for a collection of websites.
Today, we will show you steps required in building a BASH script that will do just that. Lets go….

Output Format

Within our script we will use curl. Curl allows us to define the output variables that are returned. We will define this variables within a separate file named curl_format. Within this file add the following,

%{http_code},%{num_connects},%{num_redirects},%{redirect_url},%{remote_ip},%{remote_port},%{url 
_effective},%{time_total}

Input File

Next we create a file containing the websites we want to analysis. Within this example we call the file websites.txt.

https://www.fir3net.com
http://bbc.com
http://fake.com

Script

Now lets create the script. Create a file called fetch_url_stats.sh and add the following,

#!/bin/bash

# ASSIGN PATH 
PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# CHECK FOR ARGUMENTS
if [ -z "${1}" ]; then
        echo "Usage: $0 [FILENAME]";
        exit
fi

# ASSIGN VARIABLE
WEBSITES="${1}"

# PRINT HEADER
echo target_url,http_code,num_connects,num_redirects,redirect_url,remote_ip,remote_port,url_effective,time_total 

# FETCH STATS
for target_url in `cat ${WEBSITES}`
do 
    response=$(curl -L -w "@curl_format" -o /dev/null -s $target_url)
    echo $target_url,$response
done

Example

Below shows an example. We provide the file containing the list of websites as an argument, the scripts run and the results are provided as a CSV.

laptop:~ felix001$ ./fetch_url_stats.sh websites.txt
target_url,http_code,num_connects,num_redirects,redirect_url,remote_ip,remote_port,url_effective,time_total
https://www.fir3net.com,200,2,1,,149.126.74.98,443,https://www.fir3net.com/,3.759
http://bbc.com,200,2,1,,212.58.244.69,80,http://www.bbc.com/,0.933
http://fake.com,200,2,1,,83.138.157.142,80,http://www.fake.com/,0.843

 

Rick Donato
Latest posts by Rick Donato (see all)