fir3net
PPS-Firenetbanner-780.5x190-30-03-17

How to Automate/Gather Statistics for Multiple Websites in BASH

Contents[Hide]

Introduction

For one reason or another, Im sure you will find yourself in a position when you need to obtain statistics for a collection of websites. 
Today, we will show you steps required in building a BASH script that will do just that. Lets go....

Output Format

Within our script we will use curl. Curl allows us to define the output variables that are returned. We will define this variables within a separate file named curl_format. Within this file add the following,

%{http_code},%{num_connects},%{num_redirects},%{redirect_url},%{remote_ip},%{remote_port},%{url_effective},%{time_total}

Input File

Next we create a file containing the websites we want to analysis. Within this example we call the file websites.txt.

https://fir3net.com
http://bbc.com
http://fake.com

Script

Now lets create the script. Create a file called fetch_url_stats.sh and add the following,

#!/bin/bash

# ASSIGN PATH 
PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# CHECK FOR ARGUMENTS
if [ -z "${1}" ]; then
        echo "Usage: $0 [FILENAME]";
        exit
fi

# ASSIGN VARIABLE
WEBSITES="${1}"

# PRINT HEADER
echo target_url,http_code,num_connects,num_redirects,redirect_url,remote_ip,remote_port,url_effective,time_total 

# FETCH STATS
for target_url in `cat ${WEBSITES}`
do 
    response=$(curl -L -w "@curl_format" -o /dev/null -s $target_url)
    echo $target_url,$response
done

Example

Below shows an example. We provide the file containing the list of websites as an argument, the scripts run and the results are provided as a CSV.

laptop:~ felix001$ ./fetch_url_stats.sh websites.txt
target_url,http_code,num_connects,num_redirects,redirect_url,remote_ip,remote_port,url_effective,time_total
https://fir3net.com,200,2,1,,149.126.74.98,443,https://www.fir3net.com/,3.759
http://bbc.com,200,2,1,,212.58.244.69,80,http://www.bbc.com/,0.933
http://fake.com,200,2,1,,83.138.157.142,80,http://www.fake.com/,0.843

 

Tags: BASH, Curl

About the Author

RDonato

R Donato

Rick Donato is the Founder and Chief Editor of Fir3net.com. He currently works as a Principal Network Security Engineer and has a keen interest in automation and the cloud.

You can find Rick on Twitter @f3lix001