Linux tools to parse through files

Situation: You have a lot of results in a file that you need to move through and single out and add or subtract to fields, compare entries, or many other needs

Complication: There are a TON of Linux tools to do this

Question: What can do what?

Answer: I will add to this post with specific examples consistently

 

Example 1: Search through a file list and return only the filename, not the full path

Let’s say you want to just look for missing files in a non-similar directory structure between multiple file lists. One way is to look through each file without the file path and get just the file name. Let’s say filelist1 has this list:

/tmp/derp/directoryone/file1
/tmp/derp/directoryone/file2
/tmp/derp/directoryone/file3
/tmp/derp/directoryone/file4

Let’s now say filelist2 has this list:

/tmp/der/directorytwo/file1
/tmp/der/directorytwo/file2
/tmp/der/directorytwo/file3

In this case you want to just figure out that you’re missing file 4, so that you could sync just that file and not all the contents of directoryone. So to do this you could use cat to show the file contents, and pipe it to cut to get just the filename

less filelist1 | cut -d/ -f4

The -d/ option is saying to use the as the delimiter, and the -f4 is saying to return the 4th entry

This would return

file1
file2
file3
file4

You could then send it to a new file that had just the filenames with >>

less filelist1 | cut -d/ -f4 >> filelist1_clean
less filelist2 | cut -d/ -f4 >> filelist2_clean

You can now easily find just the files missing by running

diff filelist1_clean filelist2_clean

You would then get

file4

Now you can make your sync off of this

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s