Introduction
For one reason or another, Im sure you will find yourself in a position when you need to obtain statistics for a collection of websites.
Today, we will show you steps required in building a BASH script that will do just that. Lets go….
Output Format
Within our script we will use curl. Curl allows us to define the output variables that are returned. We will define this variables within a separate file named curl_format. Within this file add the following,
%{http_code},%{num_connects},%{num_redirects},%{redirect_url},%{remote_ip},%{remote_port},%{url _effective},%{time_total}
Input File
Next we create a file containing the websites we want to analysis. Within this example we call the file websites.txt.
https://www.fir3net.com http://bbc.com http://fake.com
Script
Now lets create the script. Create a file called fetch_url_stats.sh and add the following,
#!/bin/bash # ASSIGN PATH PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin # CHECK FOR ARGUMENTS if [ -z "${1}" ]; then echo "Usage: $0 [FILENAME]"; exit fi # ASSIGN VARIABLE WEBSITES="${1}" # PRINT HEADER echo target_url,http_code,num_connects,num_redirects,redirect_url,remote_ip,remote_port,url_effective,time_total # FETCH STATS for target_url in `cat ${WEBSITES}` do response=$(curl -L -w "@curl_format" -o /dev/null -s $target_url) echo $target_url,$response done
Example
Below shows an example. We provide the file containing the list of websites as an argument, the scripts run and the results are provided as a CSV.
laptop:~ felix001$ ./fetch_url_stats.sh websites.txt target_url,http_code,num_connects,num_redirects,redirect_url,remote_ip,remote_port,url_effective,time_total https://www.fir3net.com,200,2,1,,149.126.74.98,443,https://www.fir3net.com/,3.759 http://bbc.com,200,2,1,,212.58.244.69,80,http://www.bbc.com/,0.933 http://fake.com,200,2,1,,83.138.157.142,80,http://www.fake.com/,0.843
- How to Configure a BIND Server on Ubuntu - March 15, 2018
- What is a BGP Confederation? - March 6, 2018
- Cisco – What is BGP ORF (Outbound Route Filtering)? - March 5, 2018