I recently had to determine if a large set of URLs resolved to the same set of IP addresses. Knowing a little about a few different scripting languages/technologies let me know that this was definitely possible, so I just had to figure out which one I wanted to use for the most efficient results. I decided to just go with Bash shell scripting (since I’ve recently gotten a little used to it).

Say I have a file called urls.txt (newline delimited). This file started with more than just the hostname (e.g., www.example.com/foo/bar.aspx). I had to pare those down so I opened the list in vim and did a little replacement. I could have done the same thing with sed but I’m most familiar with doing this in vim. In vim, open the file and type:

:%s#/.*##g

Ok, ready for some nslookup action in a Cygwin Bash loop. Note that this does not output what I’d like to do, which is the format of “url,ip” and output to a .csv file. It’s also important to note that the loop will take the input line-by-line, so the output file will have all the results in the right order. I could do this in perl very quickly and get the results I’m looking for, but it takes that much more time to write the script. Anyway, here it is:

$ for i in `cat urls.txt`; do nslookup $i 198.6.1.4 2>/dev/null | grep Address | tail -n 1 | cut -d " " -f 3; done > ips.txt

The “$” is the prompt, so don’t write that. For all values (i.e. lines) in urls.txt, do an nslookup of the url and specify the uu.net DNS server to avoid any problems with local resolution (I was having trouble, so had to go outside my corporate DNS server). Redirect all nslookup STDERR messages to /dev/null (there’s a header). Only show me the lines with “Address” and only give me the last of those lines (head & tail working together). Then strip the “Address: ” from the remaining line to give me the IP address by itself. And when you’re done with all that work, redirect to ips.txt instead of STDOUT.

Now I open an Excel or Calc spreadsheet and drop the values from urls.txt right next to ips.txt and I’m cooking with gas.