cli Archives - Yet Another Linux Blog

Lightweight Command Line Downloading with Aria2

Sometimes you just want a quick and easy way of downloading large files. If you’re like me, you want this with as little of a memory footprint as possible. Aria2 gives me this ability. When downloading a torrent for a recent Linux release, I was able to do this with only 5MB of memory being used. No other download programs can give me this.

What’s nice about aria2 is that you can download the same file from multiple sources (mirrors) and cut your download times with each source. You can also open multiple pipes to the same download which shortens the time as well. Let’s take a quick look at what aria2 can do for your downloads.

Downloading with Aria2

For this test I used KDE4 iso’s from OpenSuse. First, I established a baseline using wget:

wget http://www.gtlib.gatech.edu/pub/opensuse/distribution/12.3-RC2/iso/openSUSE-12.3-KDE-Live-Build0094-x86_64.iso

This took 15 minutes 47 seconds to complete. The file size is 941MB. My Internet connection at home has a max download of 10MB and upload of 1MB.

Using aria2, the same file took 10 minutes 32 seconds to complete. Here is the command I used for this:

aria2c -x2 http://www.gtlib.gatech.edu/pub/opensuse/distribution/12.3-RC2/iso/openSUSE-12.3-KDE-Live-Build0094-x86_64.iso

The -x2 in the above command pipelines the download of the ISO into 2 separate threads. This speeds things up considerably. Be wary of using too many threads though because many websites out there will throttle you down in speed should you open more than 3-4 threads.

Aria2 supports more protocols than you can shake a stick at including magnet links, bittorrent, metalink and even ftp. There are many command line flags and options you can use and you can even call aria2 using JSON-RPC and XML-RPC through the web. All together, aria2 is scalable, flexible and lightweight…there isn’t much it cannot do. If you’re looking for a lightweight download utility, aria2 has you covered.

Creating Symlinks – How and Why

As part of your Linux journey, you’ve probably heard of symlinks which are also known as symbolic links. I figured that since I fixed an error using symbolic links to setup an environment to allow my son to learn program. I am using something called HacketyHack which can be found here: http://hackety.com

The problem is that on Ubuntu or Debian, the libssl and libcrypto libraries are out of date. Hackety Hack’s program requires versions greater than 1.0.0 and 0.9.8 is installed. The fix is of course a symbolic link. But how do we do this and more importantly, WHY do we have to do this to fix it? Let’s go through what they are first.

What is a Symbolic Link?

Look on your computers’ desktop right now. If you’re like most people, you’ll have many shortcuts to different programs that you access daily. On my Windows 7 machine at work, I have around 40-50 shortcuts to commonly used tools and places I access to accomplish my job. Those are, in a nutshell, what symlinks are. They’re pretty much just advanced shortcuts and with the case I’m going to present today…shortcuts without an icon. Symlinks redirect a computer to an end location OR make a computer think the end location is where the shortcut is…and since they perform these 2 functions, there are 2 types of them.

Hard Links
Soft Links

Soft Link – When you click on/open a soft link, you’re redirected to the location it is pointing to. For example, if you click on ‘My Documents’ on your desktop, you’re redirected to a path on the C: drive where your documents are stored.

Hard Link – A hard link makes the computer think that the shortcut is the actual end location. So, using our ‘My Documents’ example above…the computer would look at the ‘My Documents’ shortcut and it would see it as the actual end location instead of a pointer to the end location.

What Would I Use a Symbolic Link For?

Do you use dropbox or box.net or any other cloud storage system to share files/store files/backup files? Then a symbolic link might be a good option for you. Imagine that you setup a folder on your desktop that is named ‘send-to CLOUD’ and when you drag and drop files to that folder, it sends it directly to those cloud storage systems. This is something that symbolic links can accomplish.

Another case might be if you need files stored in 2 different locations. Maybe you want to have settings files for an application be redirected to dropbox so that you can access it and use it on another computer. As you can see, there are many different reasons for using symbolic links.

How Do I Setup a Symbolic Link?

In Linux, you use the command ‘ln’. To setup a soft link, you use the -s flag like this:

ln -s /usr/lib/library.so.0.9.8 /usr/lib/library.so.1.0.0

So, in the example above, the file library.so.1.0.0 is LINKED or pointed back to the actual file library.so.0.9.8. To setup a hard link you’d drop the -s flag:

ln /tmp /other/location/tmp

In the above example, your /tmp folder will now appear in 2 locations…both /tmp and the /other/location/tmp. Please understand that /other/location/tmp has to actually exist <em>before</em> you issue the command.

To remove a symbolic link, just use the ‘rm’ command. I usually use -rf as flags so that it recursively deletes and forces it to occur without confirmation but it is up to you:

rm /other/location/tmp

How Do I Fix Hackety Hack on Debian?

As promised, the solution to fixing Hackety Hack on Debian. First, you need to find/locate the libraries that it complains about. In the first error I received, it was looking for libssl.so.1.0.0. I use the mlocate package which has the command ‘locate’ to find libssl.so as follows:

devnet@lostlap:~$ locate libcrypto
/lib/libssl.so.0.9.8
/usr/lib/libssl.so.0.9.8

The output tells me that there is a libssl.so.0.9.8 in two locations: /usr/lib and /lib. I’ll need to symbolically link both of those with a soft link so that when the program looks for the file libssl.so.1.0.0 it finds it and the link points it back to libssl.so.0.9.8.

sudo ln -s /usr/lib/libssl.so.0.9.8 /usr/lib/libssl.so.1.0.0

sudo ln -s /lib/libssl.so.0.9.8 /lib/libssl.so.1.0.0

Now that those to locations are created, we need to follow up with libcrypto which resides in the same two directories as libssl.

sudo ln -s /usr/lib/libcrypto.so.0.9.8 /usr/lib/libcrypto.so.1.0.0

sudo ln -s /lib/libcrypto.so.0.9.8 /lib/libcrypto.so.1.0.0

Now that both of those are linked to our actual ssl and crypto libraries, you can try running the file from Hackety Hack again.

For me, this fixed the initial two problems but I still have a failure when the installer does a hard check for OpenSSL 1.0.0 and unfortunately, I don’t have a complete solution for it yet. So, I suppose I lied a bit with the ‘fix’ for Hackety Hack. The above information is good though for other programs that might require libraries similar to the ones we linked.

Hopefully, you now have a decent understand of how and why to use symbolic links.

Some Random Helpful Hints

I’ve been slowly collecting a few commands that are useful to me for various things while using Linux. I figured that I would share some of these handy commands. In no particular order, they are:

To copy, preserving permissions AND structure AND recursively, from a remote system to your local system:

rsync -r -a -v -e ssh server1.address.com:/dir/youwant/to/copy/ /local/location/for/directory/

Please note that the code above assumes that you are using key based authentication and not password. For password based authentication, it would look more like this:

rsync -r -a -v -e ssh remoteuser@server1.address.com:/dir/youwant/to/copy/ /local/location/for/directory/

To remove all files matching a certain extension (xml in my example) in a directory:

find . -type f -name "*.xml" -exec rm -f {} \;

To go into a location, find all files that match a certain extension (jpg in my example) and move them to a different directory:

find . -name "*.jpg" | xargs -i mv '{}' /location/to/move/them/to/

To recursively remove empty directory from the directory you are currently in (your pwd):

find -depth -type d -empty -exec rmdir {} \;

These are a few of the commands that I’ve found useful in the past few weeks. I hope you find them useful as well. I’ll be test driving quite a few different distributions and reporting back what I find as well as experimenting with various different commands…I really like find because it is so powerful so look for some more posts with uses of the find command. Thanks for reading and sorry for my lapse in posting this past month!

Chasing Your ‘Tail’ With Linux

‘GNU tail’ is a small utility which prints (by default) the last 10 lines of any file. This an amazing piece of software not only allows you to see the last part of a file but also enables you to monitor a file’s changes without opening the file.

‘tail’ can be used alone or can be combined with other commands like ‘grep’, ‘ls’ etc.

To use ‘tail’, let’s first create a text file. You can create the file by issuing following command in terminal;

touch my_file

Now open my_file with your favorite text editor (nano in my case) and write some lines. For this article, I have written the following 15 lines;

this is the 1st line
this is the 2nd line
this is the 3rd line
this is the 4th line
this is the 5th line
this is the 6th line
this is the 7th line
this is the 8th line
this is the 9th line
this is the 10th line
this is the 11th line
this is the 12th line
this is the 13th line
this is the 14th line
this is the 15th line

Now issue the following command in terminal;

tail my_file

It will print the last 10 lines which would be the “this is the 6th line” through “this is the 15th line”.

You can control the number of lines which ‘tail’ will print. You can either increase or decrease the number of lines. For example, if you want ‘tail’ to show only last 3 lines, you can do this by issuing the following command;

tail -n 3 my_file

Now it will print only last 3 lines. You can use any number of lines instead of 3. Or you can use a plus sign like;

tail -n+7 my_file

‘tail’ will start printing from 7th line to the end of the file.

You can view the desired file with respect to size. Issue the following command in terminal;

tail -c 14 my_file

And it will show the output of last 14 bytes. In my case, the output was;

the 15th line

‘tail’ not only displays the static output of a file but it can also monitor the file for changes. A ‘-f’ option is used with ‘tail’ and it starts acting like a monitoring tool which not only displays the last few lines but also constantly updates the output as the file changes. Here is a very popular example;

tail -f /var/log/message

‘tail’ will print the last 10 lines of ‘message’ file. If you now plug-in you USB stick, you will notice that the change in ‘message’ file will instantly be reported by ‘tail’. To release the cursor press Ctrl+c.

There are many other useful options which you can use with ‘tail’ like;

tail -q my_file        # never output headers

tail -v my_file        # always outputs headers

You can combine ‘tail’ with other utilities like ‘ls’, ‘grep’, ‘head’ etc.

You can combine ‘tail’ with ‘grep’ to get lines with some specific ‘word’.

tail -n 5 my_file | grep 14

It will print only those lines out of last 5 which contains the word ’14’. In my case the output was:

this is 14th line # ’14’ will be highlighted

‘tail’ can also be combined with ‘ls’ to get the list of last few files/folders. For example, if you issue the following command;

ls -l | tail -n 2

It will give a long listing of files/folders but will show the last 2 entries of the working directory.

These are just two examples of combining ‘tail’ with other utilities. There are countless examples of combination of ‘tail’ and other softwares.

‘GNU tail’ is a very handy tool. It can output any amount of data depending upon the options used. It makes the work of an ordinary user much easer and helps him/her find information in files more efficiently. To become an expert in Linux, this is a mandatory utility over which a user must have complete mastery. Hopefully, this tutorial gets you started chasing your tail!

Using ‘Alias’ in Linux

There comes a time in every Linux users’ life when you will open the Terminal more often than not because you have realized that it is faster, more efficient and more powerful than GUI (Graphical User Interface). You’ll have started to learn more and more commands and now feel more comfortable with command prompt. The command prompt is all about commands – short commands as well as long commands. If you are like me then you may not like to type the long commands (or even small commands) 🙂

You may be thinking about some way to avoid typing commands over and over. Enter the ‘alias’.

The ‘alias’ tool is a way to make the complicated things simple (and simple things simpler). You can use ‘alias’ instead of long (or even short) commands. Now let’s see how the ‘alias’ works.

‘alias’ can make difficult and lengthy commands easy. The general format of ‘alias’ is:

alias Any_Word=”Command”

It means you linked an existing command to a (New) Word. This ‘Any_Word’ may contain anything – any alpha-numeric symbol, ‘Any_Word’ as well as ‘Command’ are interchangeable and can be used for the same purpose.

Simple Commands Made Simpler

As an example, ‘ls -l’ is used for listing directory contents in ‘long listing format’. This ‘ls -l’ can be replaced with a simpler alias. You can set the ‘alias’ for ‘ls –l’ as follows:

alias ll=”ls –l”

Now you just have to type ‘ll’ (without quotes) to get ‘long listing format’.

Or if you frequently misspell ‘ls’ as ‘sl’ and don’t want to install ‘sl’ package, then, you can use the following alias:

alias sl=”ls”

Now, whenever you type ‘sl’ in terminal, it will give you same results as ‘ls’.

Now consider even simpler example. To close a Terminal (or logout), you have to type ‘exit’ in Terminal. This ‘exit’ command can be made even simpler by using the following ‘alias’:

alias x="exit"

Now, you only have to type ‘x’ in Terminal to ‘exit’

Other examples of ‘alias’ are:

alias cp="cp -iv"
#make copy operation interactive and verbose

alias rm="rm -iv"
#make remove operation interactive and verbose

alias mv="mv -iv"
#make move operation interactive and verbose

Make Package Management A Bit Simpler

If you use Debian (or its derivatives) then you will be familiar with APT. It is an excellent package manager.

In Ubuntu, to install software using APT, you have to use the following command:

sudo apt-get install <sofware_name>

It is a long command and consumes a lot of your time and energy 🙂

You can shorten this command by using ‘alias’. Issue the following command in Terminal:

alias Install=”sudo apt-get install”

You can obviously use your own word instead of ‘Install’.

Now, you just have to type:

Install <software_name>

to install the (same) software. Simple, isn’t it?

You can simplify other aspects of APT. For example, you can use the following ‘alias’:

alias Remove=”sudo apt-get remove”

to uninstall a software.

Some other examples of attaching APT with ‘alias’ are:

alias Update=”sudo apt-get update”

alias Upgrade=”sudo apt-get upgrade”

alias Search=”apt-cache search”

alias Autoremove=”sudo apt-get autoremove”

alias Autoclean=”sudo apt-get autoclean”

alias Purge=”sudo apt-get remove –purge”

and so on…

A Very Interesting ‘alias’ For A Difficult Keyboard Button

On some keyboards, the dot (.) button is at very difficult position and if you have to use it more than once, it becomes even more difficult. That’s why ‘cd ..’ is the command which I mistype the most. This complication can be easily removed by using following ‘alias’ (you can use any other word instead of a dot):

alias .=”cd ..”

alias ..=”cd ../..”

alias ...=”cd ../../..”

alias ....=”cd ../../../..”

Using Internet From Terminal

If you regularly use lynx to browse the internet in terminal then you have to type long urls with lynx to visit the web pages. You can simplify these long urls by using ‘alias’:

alias Google=”lynx http://www.google.com/”

alias Yahoo=”lynx http://www.yahoo.com/”

alias yalb =”lynx http://linux-blog.org/”

and so on…

Now just type Google, Yahoo or yalb to visit the respective web sites.

Simple ‘alias’ For More Complicated Commands

Long commands are not only difficult to remember but also take more time to type; when you have to use them on daily basis, you become frustrated when typing them again and again and again… So, ‘alias’ are more suitable for long and complicated commands.

Let’s consider an example.

To find the top 10 largest files in your system, you can set the following ‘alias’:

alias top10files=”find . -type f -exec ls -sh {} \; | sort -n -r | head -10”

You can even mix different commands with ‘alias’. For instance, if you regularly use ‘tail’ and direct its output to file to later view that file, you can set a very simple ‘alias’ to do this cumbersome operation in 1 word:

alias Tail=”tail /var/log/messages > hello.txt;cat hello.txt”

Now just enter ‘Tail’ and viola! All is done at once.

You can use any file with tail and direct its output and you can even use ‘nano’ or ‘vi’ to view/edit its output.

Here’s another example… ‘alias’ to connect to a remote server:

alias any_name=”ssh <remote_server_address> -l <username> -p <port>”

You can even create ‘alias’ for your bash scripts, like:

alias clc=”sh /home/user/myscripts/calc.sh”

Now that you have set a few different ‘alias’ you might want to check that which ‘alias’ are set on your system. To do that, just issue the following command:

alias

and it will list all the set ‘alias’ you have.

To remove an ‘alias’, just issue the ‘unalias’ command, like:

unalias Google

and now typing Google in terminal will do nothing (as it was set with lynx).

To remove all the ‘alias’, issue the following command and all the ‘alias’ are gone:

unalias –a

We have discussed the way of setting the ‘alias’ for different kinds of commands. But setting ‘alias’ in this way be temporary. When you reboot you PC, all the ‘alias’ which you have set will be gone. This does not mean that you have to set all the ‘alias’ every time you boot your PC. If you have set an ‘alias’ and you liked it so much that you want it to permanently reside in you PC, just add this alias in ‘.bashrc’ file in you home directory. For example, if you want ‘alias’:

Install <software_name>

to permanently reside in your PC then user your favorite text editor and add the following line in your ‘~/.bashrc’ file:

alias Install=”sudo apt-get install”

Now this ‘alias’ will not vanish into thin air when you reboot your PC. Only those ‘alias’ which are listed in ‘~/.bashrc’ file will be permanent.

This guide is just a preview about ‘alias’. It is just about basic ways of using ‘alias’ to make your life simpler. ‘GNU alias’ is a tool which can simplify your life immensely. But unfortunately this tool is not given the attention it deserves. In short, it is such a powerful tool that if you give it proper time, it can make you forget typing 🙂

GNU find – A Multidimensional Tool

Beginners are mostly afraid of command prompt. Whenever they see a command prompt, they immediately say “its very difficult”. But it’s not true. The Command prompt is as friendly as GUI (Graphical User Interface), provided if you use it with proper procedure.

Most people use GUI tools to search for files. They don’t realize that they can use command line tools to search for them as well! GNU ‘find’ is such like a tool which can not only search files but can even copy, move or delete these files on the fly.

So let’s see that how ‘find’ works.

Find Your Lost Files!

Let’s start from a simple example:

Suppose you want to search for a file named ‘master.txt’ in your home directory.

Open the Terminal and issue the following command:

find . -name “master.txt”

‘find’ will immediately show the results. If ‘find’ does not show any result, this means that the file, in our case, ‘master.txt’, does not exist. It is not always the case that you want to find something in you home directory. The lost/desired file may be anywhere in your computer. Suppose you want to find a file named ‘space-01.jpg’ and you only know that its located somewhere in /usr directory. You can find it by issuing following command in Terminal:

find /usr -name “space-01.jpg”

and ‘find’ will tell you that this is located under /usr/share/backgrounds.

Using Wildcards

Maybe you want to search for a file but you don’t know its exact name? Don’t worry! You can still locate the file using ‘GNU find’ and wildcard will help you in this regard. Wildcards are a way of searching files when you don’t know much about your desired file.

One of the commonly used wildcard is asterisk (*). Lets consider an example to better understand the things.

Suppose you want to search a file named ‘Jumping_Flowers’ but you only remember the ‘Jumping‘ part of the file name. So issue the following command in Terminal:

find . -name “Jumping*”

And it will display all the files starting with the word ‘Jumping’. You can use asterisk (*) anywhere with a file name. For example:

find . -name “*Jumping*”

And it will display all the files which contains the word ‘Jumping’.

Here are some more examples of use of a wildcard:

find . -name “Jumping*Flowers*”

find . -name “*Jumping*Flowres.mp3”

Searching For Different File Types

Sometimes you are not looking for some specific file but you are looking for a group of files. For example, you may be looking for all the .txt files in your home directory. To find all the .txt files, you will give the following command in Terminal:

find . -name *.txt

In case of mp3 files, the above command will be:

find . -name *.mp3

When You Want to Search with Respect to Time

If you want to search for files by the last time they were accessed, you can use -amin flag with ‘find’. In this case you have to put a minus (-) sign before the time. The time here is in minutes. In order to search for .doc files which were accessed in last 10 minutes, you will give the following command:

find . -amin -10 -name "*.doc"

Similarly, to search for .doc files which were modified in last 20 minutes, you will use -mmin option as follows:

find . -mmin -20 -name “*.doc”

Search For Files which are Eating Your Hard Disk

There may be files on your system which are not only huge in size but also located obscure places. You may also may not know when they were last accessed. You have to use -size option with ‘find’ to locate them.

Let’s see how we can do this:

find . -size +100M

It will list all those files which are greater than 100 Megabytes. You can replace ‘M’ with ‘G’ (for Gigabyte) or with ‘k’ (for Kilobyte)

Copy, Move, or Delete Unwanted Files on the Fly

Copy – ‘find’ can also be used to copy or backup your files. You can use ‘find’ to copy certain files from one location to other with one simple command.

Suppose you want copy all of your mp3 songs from your home directory to your Windows Partition. Enter the following command in Terminal:

find . -name "*.mp3" -exec cp {} /path/to/Windows_Drive \;

And all of your mp3 files will be copied to the desired Drive/Folder.

Move – There may be situations that you quickly want to move all of your document files from your Hard Disk to your USB to keep them safe. To move all of your documents from your home directory to your USB, you will issue the following command:

find . -name "*.doc" -exec cp {} /path/to/USB \;

Delete – Suppose there are a lot of .tmp files and you want to get rid of them at once. Again ‘GNU find’ is at your service and does the work for you. Issue the following in Terminal and all of the .tmp files are gone…

find . -name '*.tmp' -exec rm {} \;

Which Files are Owned by You and Which Are Not?

There may be a situation when you want to know that which files in some other directories (or even in your home directory) are owned by some other user of your computer.

Suppose there is another user named ‘blackstar’ with whom you are sharing your PC. Now you want to know that which .doc files in Windows Directory is owned by this user ‘blackstar’. You can do this by issuing the following command:

find /path/to/Windows_Drive -user blackstar -name “*.doc”

Just replace ‘blackstar’ with your username to search on your system.

Direct the Output of ‘find’ to a File

You can save the results of your ‘find’ command to a text file which will allow you to examine the results in detail at some later time (or to create playlist of your songs). For this purpose a greater than (>) sign is used (referred to as “piping the command”).

Suppose you want to save the list of all the mp3 songs in your home directory to a text file (which you can later share with your friend), you can do this by:

find . -name "*.mp3" > mp3.txt

It will save the complete path to all of your mp3 songs in the file named mp3.txt.

Find, a Handy Command Line Tool

This article is basically directed towards new users of Linux which are not much familiar with command prompt. This is a small but comprehensive article about ‘GNU find’ . The man pages of ‘find’ list a huge number of options which are difficult to explain in detail in one small article. I tried to cover those option which are commonly used. Obviously, to know more about such a powerful tool , one has to visit its man pages again and again and spend a lot time with ‘find’ 🙂