Tag Archives: shell script

Shell Script to Detect If the IP Address Is Googlebot

1. Introduction

Google has explained here on how to verify if the given IP belongs to Googlebot. So instead of hard-coding all the IPs which might change eventually, Google has suggested doing a DNS lookup using host command on Linux. The steps suggested in the article are:

  1. reverse DNS lookup using the IP to get domain name
  2. check if the domain name contains googlebot.com or google.com. The types of bots and their names can be found here.
  3. forward DNS lookup using the domain name obtained in step 1 to get the IP and verify that this IP is same as the IP you initially started with.

2. Implementation Approach

And I wanted to check if the IPs ( I had around 45) belonged to google bot. One option was to run host for each of the IPs, as suggested in the above steps, but this was practically not possible and if possible it would be time-consuming. So I came up with a simple shell script to do this job.

  1. reverse DNS lookup to get the domain name
    #hostName is the complete domain name
    hostName = host $line | cut -d" " -f 5

    So a sample response of host command would be domain name pointer crawl-66-249-66-246.googlebot.com. And we can extract the domain name using the cut command as shown above. This gives us hostName = crawl-66-249-66-246.googlebot.com.

  2. forward DNS lookup to get the IP
    hostIp=host $hostName | cut -d" " -f 4

    A sample response to the host command, in this case, would be: crawl-66-249-66-246.googlebot.com has address And we extract the IP using the cut command shown above which gives us the hostIp =

  3. verify the domain name to contain googlebot and the IP obtained in step 2 is same as the IP we started with in step 1
    if [ $line == $hostIp ] && [ $domainName == "googlebot.com" ]
        echo "Googlebot: $hostIp -> $hostName"

3. Complete Shell Script

Let the IPs be in the file googlebots, for example, let us add these IPs:

The shell script is given below:

while read -r line
    hostName=`host $line | cut -d" " -f 5`
    domainName=`echo $hostName | cut -d"." -f2,3`
    #echo $domainName
    #echo "$hostName"
    hostIp=`host $hostName | cut -d" " -f 4`
    #echo "$hostIp"
    if [ $line == $hostIp ] && [ $domainName == "googlebot.com" ]
        echo "Googlebot: $hostIp -> $hostName"
done < "$file"

Shell script for clearing the Trash

I was facing a problem while clearing the Trash in Ubuntu. I wasn’t able to delete certaion folder and i got the message stating “Access Denied”. So i was wondering that there should be some way to do it from the command prompt using sudo and some command. I did a google for the requirement and found the following shell snippet:

rm -rf ~/.local/share/Trash/

Source: Ubuntu Forum.