BIPEDU

my ideas in action

Tag Archives: command-line

easy backup system with rsync – like Time Machine

Backup systems are good for recovering in case of accidental  lost data. But a more useful feature is the incremental backup where you have access to various snapshots in time like the Time Machine on Apple is doing. To do this in Linux (or any Unix  or alike ) systems is actually very easy.

For example we make a backup every day ( or every internal you want) . We need that  the amount of data transferred is small and not big. Imagine transferring few TB every day ! in case our important data is changing a little bit then we will backup only the modified parts. For this Rsync is the best tool. Everybody knows that. But there is a problem. How can we keep daily snapshots of the data without filling the disk ? For this we will use softlinks,  hardlinks and Rsync options.

So we have to create a script file like this:

#!/bin/bash
date=`date "+%Y-%m-%dT%H-%M-%S"`
rsync -aP --delete --log-file=/tmp/log_backup.log --exclude=lost+found --link-dest=/mnt/sdb2/Backups/current /mnt/sda1/ /mnt/sdb2/Backups/back-$date
rm -f /mnt/sdb2/Backups/current
ln -s /mnt/sdb2/Backups/back-$date /mnt/sdb2/Backups/current

So here I make first a “date” variable that will be used in the name of the backup folder to easily know when that backup/snapshot was made.

Then use the rsync with some parameters (see man rsync for more details):

-a = archive mode ( to send only changed parts)

-P = to give a progress info – (optional)

–delete = to delete the deleted files from backup in case they are removed from source

–log-file = to save the log into a file (optional)

–exclude = to exclude some folders/files from backup . This are relative to source path !!! do not use absolute path here !

–link-dest = link to the latest backup snapshot

/mnt/sda1 = source path (here I backup a whole drive)

/mnt/sdb2/Backups/back-$date  = destination folder , it will contain all the content from the source.

Then by using rm I remove the old link to the old backup ( the “current” link) and then I replace it with a new soft link to the newly created snapshot.

So now whenever I click on “current” I go in fact to the latest backup .

And because every time I make the backup the date is different the old snapshots will be kept. So for every day I will have a snapshot.

To automate this you have to create a cron job to execute the above script at the convenient time.

Example to run at 4:01AM every day:

1  4 * * * /path/to/script

Please notice that only the first time the full backup will take a long time since it will copy the full data. The second time you will run the script it will transfer only the changed files/bits.

Now on the destination folder you will see a “back-xxx” folder for every time you run the script. You can open/read the files from all this folders as it if they are completely independent files. In fact if you run df and du you will see something interesting.

For example if the backup is 600GB and the script is run every day you will see that the df will show the same 600GB used from disk space. But if you run “du -sh /* ”  you will see that each “back-xxx” folder is 600GB each. This is possible because there are only hardlinks to the same data copied. Do not worry, the disk is not full and you should trust the df results and not the du results.

user@box:/mnt/sdb2/Backups$ du  -sh ./*
623.8G    ./back-2014-02-24T17:47:12
623.8G    ./back-2014-02-24T21-46-41
623.8G    ./back-2014-02-25T17-05-02
623.8G    ./back-2014-02-25T18-45-34
0    ./current
user@box:/mnt/sdb2/Backups$ df /mnt/sdb2
Filesystem                Size      Used Available Use% Mounted on
/dev/sdb2                 2.7T    623.9G      1.9T  24% /mnt/sdb2

So the Time Machine is in fact only 3 lines of code in a script plus a cron job ! Easy and everybody can do it !

Adapt the script to your needs. Run it when you want with cron jobs.

At any point in time you can delete old backups ( for example backups older than few weeks). This can also be made with cron plus some scripts.

Advertisements

FTP backup with BASH script ( FTP scripting)

FTP protocol is used to transfer data between computers. The user has also a possibility to combine bash scripts with FTP to automate the backups of the files. This concept can be used for example to backup some files from a local machine to a remote server.
The way to do this is by making an executable script that is run from time to time by a cron job task. In this way the backup is made automatically in the background and do not require user intervention. But there is a problem. if you use FTP in command line then you have to type user name and password in clear text. So how to do it ?
The solution I suggest is like in the example below:

First step is to determine if the backup is needed . We check if the file was changed since last backup. For this we use a compare between the size of the file in this moment with the size of the file at the previous backup time. The previous size was saved in a file. To check this we use the “if compare” structure from BASH:

### check if the file was changed, and if YES then make FTP transfer, if not exit
 if [ `cat ~/.size_myfile.txt` -ne `du -s /path/to/file | cut -f1` ] ; then
 echo 'File changed !'
 fi
 ###########################

Then we define some parameters for FTP command. This definition can be made in a different ( hidden) file. For not critical situations ( home users) I recommend keeping the user details in the same file ( in the script) but to remove all the permissions for the other users ( use chmod command ). So the backup script should look like this:

-rwx------ myuser mygroup 32 june 12 14:52 backupFTP.script

Notice that only the “myuser” user has the right to read/write/execute the file

So for FTP command you need:

##############################
 HOST='ftp.myserver.com'
 USER='myusername'
 PASSWD='mypassword'
 FILE='/path/to/filesource filedestination'
ftp -n $HOST <<END_SCRIPT
 quote USER $USER
 quote PASS $PASSWD
 cd /path/to/destination
 put $FILE
 quit
 END_SCRIPT
 #############################

Since the source path and the destination path may be different you can use “cd /path/to/destination” for the file. The copied file can be also renamed as shown above ( see “filedestination“)

Notice that the commands between “END_SCRIPT” tags are executed inside FTP terminal. This are FTP commands and not BASH/Linux commands. You can put here whatever FTP commands you want based on your needs. For a full list of the FTP commands type “help” in the FTP terminal.

The 3rd step is to recalculate and save the new size of the file so that next time when backup script is run the size file is updated. For this we do:

################################
 ## recalculate the new size of the file, for next backup
 du -s /path/to/file | cut -f1 > ~/.size_myfile.txt
 env DISPLAY=:0 notify-send 'FTP backup done !' 'File : /path/to/file'
 ###############################

Optionally you can show a desktop notification that the backup was made. If you do not have a GUI then do not use it.

Next I show the full script in only one file:

#!/bin/bash
 ### check if the file was changed, and if YES then make FTP transfer, if not exit
 if [ `cat ~/.size_myfile.txt` -ne `du -s /path/to/file | cut -f1` ] ; then
#    echo 'File changed !'
 sleep 1
 #
 HOST='ftp.myserver.com'
 USER='myusername'
 PASSWD='mypassword'
 FILE='/path/to/filesource filedestination'
 ftp -n $HOST <<END_SCRIPT
 quote USER $USER
 quote PASS $PASSWD
 cd /path/to/destination
 put $FILE
 quit
 END_SCRIPT
 sleep 1
 ## recalculate the new size of the file, for next backup
 du -s /path/to/file | cut -f1 > ~/.size_myfile.txt
 env DISPLAY=:0 notify-send 'FTP backup done !' 'File : /path/to/file'
fi
 ###############

live USB linux with dd command

I use linux most of the time and from time to time I want to test a new distro. The easy way to try a new distro is to use the USB stick. The CD is outdated and the DVD is expensive.

There are many ways to make a live bootable USB but I want to show the easiest one. All you need is the ISO file and any linux machine. Any Unix OS should be fine but I did not tested.

So no special software is required.

First you have to download the ISO for the distro you want to try. For example Mint, Debian , Ubuntu, etc…etc…

Then open a terminal and use dd command to transfer the ISO image to the USB stick.

For this you need the root privileges. You can become root with su or you can use sudo in front of the command.

So in terminal type (first read this post till end !):

sudo dd if=/path/to/linuxdistro.iso of=/dev/sdb bs=4M ; sync

That’s it ! Now the ISO image is copied on the USB stick and you can boot from it.

Details about the command:

sudo = to get super user privileges, if not then you have to become root first with su command

dd = command to copy to disk

if=/path/file  = select the input file, in this case the ISO file.

of = the destination of the copy

/dev/sdb = your USB device. First check what is the name of your USB stick. You can plug the stick and check it with df command. Do not use any number after sdx since the dd must transfer the ISO file to the disk and not to the partition !

bs=4M = it means to copy up to 4MB at a time – it is optional but it optimize the transfer

sync = optional for the linux OS to flash the buffers

For more info about dd or sync commands please use man from terminal:

man dd

and / or:

man sync

easy way to upgrade Tiny Core Linux 4.x

The Tiny Core Linux is a distro that is very small in size and I personally use it for my home server.

The update/upgrade this distro is actually very easy but the Tiny Core website is chaotic in giving the information. Some pages are outdated ( from 2.x or 3.x versions)

So in my case I had version 4.5.5 and I wanted to upgrade to 4.7.6.

First you have to find what version you have.

Open terminal (any user) and type : version

 

Then the real upgrade begin.

The upgrade is made from 2 steps: first upgrade the TCL (Tiny Core Linux) core and then the extensions ( tcz)

 

So to upgrade the TCL itself you do not need to burn a new CD or use USB sticks , it is enough to download the latest iso file. In my case I downloaded “Core-current.iso” on another linux machine ( client of the TCL server) . Can be made also directly from TCL server but in my case I did not had a software that uncompress ISO files.

Then open the ISO file with a archive manager ( or a similar program that is able to extract files from ISO files). Inside there is a boot folder.

Then copy the content of the unpacked ISO to the TCL disk. In my case I copied from a client machine to the TCL server in a location where I have access to write. In my  TCL server I use  a USB stick as a boot partition “sdc1“. So I saved the files to a sda1 disk.

Now login as  “root” (on TCL server from terminal use “su root“).

Now check what files from the old TCL distro did you changed and which ones are identical with the files from new unpacked ISO file. In my case only /boot/syslinux/syslinux.cfg was customized by me. So I copied all the other files , one by one, and I overwrite the old files from TCL 4.5.5 with the new files (from ISO file).

Then optionally backup the personal settings , if necessary.

Then reboot the system.

After reboot login again as a normal user or as root and check the “version” command to see the new TCL distro version.

 

The second part is to upgrade the TCL extensions.

This can be upgraded independently from TCL core/kernel upgrade described above. Personally I prefer to upgrade them together.

So you have to login as root ( from terminal use “su root“). Then type “tce-update“. Follow the instructions on the screen ( you have to press ENTER ) and at the end reboot again.

 

And this is all you have to do : no CD/USB, only a copy and two reboots. All are done remote.

As an alternative to the ISO unpacking you can use the TCL mirrors that give the upgrade files already unpacked. In my case I was not sure which files I have to copy and I used ISO archive as a reference.

 

 

Screencast with FFMPEG with sound (Ubuntu 12.04 / Mint 13)

This is how to make a high quality screencast with sound from the soundcard.
Great if you want to record for example screencast tutorial or  a webpage that contains sound.
system: Ubuntu 12.04/ Mint 13

  • Record without sound, image size relative to dimensions of your screen (my case is 1600×900):

ffmpeg -f x11grab -s 1600×900 -r 25 -i :0.0 -sameq output.mkv

  • Record with sound, from your microphone:

ffmpeg -f alsa -i pulse -ab 192 -acodec pcm_s16le -f x11grab -s 1600×900 -r 25 -i :0.0 -sameq output.mkv

To record with sound, from your microphone you have to do also the following steps ( only first time):

* Install Pulse Audio Volume Control. (through synaptic package manager)
* Repeat the second command, recording starts.
* Go to Pulse Audio; go to the tab ‘Recording’, it shows ffmpeg recording the sound. Change the pull down menu to Monitor from Built in Analog Stereo.
Now it records the sound stream in stead of your micro. Set your microphone sound level to the desired values.
That is basically it.

After you save the file you will need to cut first few seconds and last few seconds to remove the parts where you start/stop the ffmpeg command.

This can be done with the below command.

Basically in the below example I extract from “output.mkv” the video starting from second 10 . The length of the output file (clip-output-file.mkv) is set to 120seconds.

ffmpeg -ss 10 -t 120 -i output.mkv -acodec copy -vcodec copy clip-output-file.mkv

print some columns from CSV file

If you have a CSV ( comma separated values file) and you need to print only some columns then you can use sed, awk or cut. Today I will show the “cut” command. Do not confuse it with “cat” !!

CUT is quite easy to use and is more simple than AWK or SED. For example if you have a file like this

$ cat datafile.csv

device1 device2 device2a device4b device5a device8 device9

1 56 8 99 5 41 36 8

22 5 99 89 56 56 2

1 0 2 5 9 63 5

As you can see the delimiter is not comma but space. If you want to print only columns 2,3 and 5 then “cut” is the best tool.

$ cut -d” ”  -f2,3,5 datafile.csv

device2 device2a device5a

56 8 5

5 99  56

0 2 9

-d” ” = means to use ” ” (space) as delimiter in the input file

-f2,3,5 = means to print only the fields 2, 3 and 5

So by default the output delimiter is space. If you want a specific delimiter, like tab, you can use this :

$ cut -d” ”  -f2,3,5 –output-delimiter=$’\t’  datafile.csv

device2         device2a        device5a

56                 8                       5

5                   99                     56

0                  2                        9

The syntax $’\t’ is special because cut do not accept TAB as “\t”

For another delimiter you can use the option like : –output-delimiter=”.:.”

$ cut -d” ”  -f2,3,5 –output-delimiter=$’\t’  datafile.csv

device2.:.device2a.:.device5a

56.:.8.:.5

5.:.99.:.56

0.:.2.:.9

Of course for all this there is a AWK or SED command but I think that for simple column selection a CUT command is easier.

More than that. The CUT command give also the possibility to select only some bytes or characters from the file. For example to print only the characters from 2 to 5 you have to use :

$ cut -c2-5 datafile.csv

evic

56

2 5

0

not very useful in my case but nice to have in case the input file format do not have a delimiter.

see more info with “man cut”

WD30EZRX Caviar Green 3TB, correct disk size and how to partition in Linux/Unix

I have a new HDD from Western Digital. It is a WD30EZRX Caviar Green. It has 3TB and the price is almost ok. The disk works fine for now so I will detail the way the partition can be done in Linux/Unix.

Why do we care about this ? Well.. because this 3TB is bigger than normal HDD on the market. The forums are full with people asking why do not see the full 3TB size . Well there are some issues with the way the data is addressed. It is nothing wrong with the disk but with the software used to see the partitions or to access the partitions.

In my case I used a old external HDD enclosure to test the drive. It was out of the box, so I was thinking to partition it and then to be screwed in the final rack.

Wrong ! Since my HDD enclosure ( cheap and old case with JMicron SATA-to-USB interface) is accessing the data in 32bit format (hex) the max visible disk size was 746MB. I searched the web and I found many guys complaining that they see the same 746MB size in the 3TB disk. But other people were saying that they see 2.1-2.1TB !?!

Well… after aprox 1 hour of reading the web, I discovered that the problem is do to software and not do to hardware.

Those of you that are using fdisk to see ( and/or partition) this disk will run in the same issue. So I used “parted” in command line. I did not tested with Gparted ( from live Ubuntu distro) but I assume that will work also. So I used the last Parted version and the disk size was ok 3.001TB.

WARNING  !!! : Parted writes the changes directly on the disk !!! (no undo option , and no cancel !! )

So in order to partition with parted ( command line version) you have to do the following steps: (assuming the disk is sdb , if not please change it to your proper value)

1: you have to change the partition table from msdos to gpt.

# parted /dev/sdb
GNU Parted 3.1
Using /dev/sdb
Welcome to GNU Parted! Type ‘help’ to view a list of commands.

(parted) print
Error: /dev/sdb: unrecognized disk label

(parted) mklabel gpt

(parted) print
Model: Unknown (unknown)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

2: partition the disk

# parted /dev/sdb

(parted) mkpart primary 0GB 3001GB

(parted) print
Model: Unknown (unknown)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  3001GB  3001GB               primary

3: then you must format the new partition to the ext2, ext3, ext4 or whatever you want. This step may take few minutes. Wait….

# mkfs.ext2 /dev/sdb1

4: mount the new created partition

# sudo mount /dev/sdb1 /mnt/somefolder

Probably there are other ( more easy or cleaver ) methods but this was working for me.

BTW: in my case I format it as ext2 and at the end I have approx 2.88TB and from this I can use max 2.7TB…

So it appears that this >100GB are lost in filesystem stuff. So please have this in mind for big HDD since it may create disappointment ..(“Look I have a 3TB disk …but I can use only 2.5TB !!”)

 

For more information on this issues please read :

http://www.unixgods.org/~tilo/linux_larger_2TB.html

http://www.thegeekstuff.com/2012/08/2tb-gtp-parted/

http://blogging.dragon.org.uk/index.php/mini-howtos/using-parted-to-add-and-manipulate-disk?page=2

http://www.gnu.org/software/parted/manual/

 

loop all files inside a folder

Another command line trick.

Let’s say that you just come back from vacation and you have few hundreds pictures in a folder. And you want to rename them from DSCxxxxx.JPG to something else like Vacation_2012_xxx.JPG.

You can do this with just a single line command:

for file in DSC*; do mv “$file” Vacation_2012_”${file#DSC}”; done

So basically this command is taking all files from current folder, that start with DSC string. If your files start with different string the replace DSC with yours (example “IMG”…).

Then each file is renamed (mv command) in to Vacation_2012_xxxxx.JPG . As you can see the string replacement is just adding the “Vacation_2012_” to a string from which it deleted “DSC”.

 

Now lets say that you have somewhere a web photo album ( like a PHP or Picasa, or similar). You want to upload all pictures there but they are too big (12MP, or 16MP). Well, a similar strategy works here, like this command:

for file in *.JPG; do convert $file -resize 1000 $file; done

So the same looping in the current folder, but now the “convert” command is used and all pictures are re-sized to  1000 pixel width. Please run this on a backup , since the originals will be replaced !

for more option please use :

man convert

 

hard passwords but easy to remember

One of the important problems of the modern computing is the security, privacy and the personal data that must be protected .

Usually we are using passwords to protect our data or  accounts ( email, web…).  But one of the main issues is that the passwords must be easy to remember for the owner but very hard to crack for the enemy. So if we choose a password that can be easily remembered, then there is a chance that other persons know it ( like object names, famous places, books quotes…etc).

If we use hard passwords, then it is a risk that we ( owner of the data) forget the password. The alternative is to write down the hard ( complex ) password on a paper and put the paper on a hard to find place. But this solution rise also the problem of the aces to that password. We may need this password in various moments , at random places in time, when we do not have immediate access to our little paper with passwords.

An alternative is to have a easy to remember password but to code it in such a way that the results is a very complex and hard to crack password.

For the people that have access to a computer this is an easy task.

Just an example : we generate the md5sum of a password.

$ echo “password” | md5sum

286755fad04869ca523320acce0dc6a4  –

The md5sum is unique to a input and cannot be cracked. This means that based on the result you cannot recover the input.

If we need only a certain number of characters, we can trim this string to a desired length (example 10 characters), like this:

$ echo “password” | md5sum | head -c 10 ; echo

286755fad0

If we want more, we can go further. As you notice in this string there are only numbers and letters ( small cap letters ). We can now pipe the result to another level of encoding. For example base64 encoding. This type of encoding is reversible ( the input can be found based on output data). The purpose here is to introduce more variation in the string.

$ echo “password” | md5sum | base64 | head -c 25 ; echo
Mjg2NzU1ZmFkMDQ4NjljYTUyM

Here we obtain the md5sum of the “password” word and the the result is send to base64 encoding. Then the length of the string is reduced to only 25 characters.

So now we have numbers and letters ( big and small cap).

So , as you can see this results are very hard to remember and crack. But for the owner that know to make this easy steps, the solution is easier.

Now lets suppose that the enemy find the “password” word. Well, whatever he tries, he cannot take the account.

The owner will never remember the Mjg2NzU1ZmFkMDQ4NjljYTUyM password, so even if the enemy force the owner to tell him his password , he cannot get it.

 

Of course, the secret here is that the “easy to remember password” is processed so that we obtain a hard password. Of course this is just an example.

You can even encode few times the result, mds5sum again few times, cut the length, or even add special characters in certain positions. So the possibilities here are infinite.

 

So please use hard password !

the dictionary attacks are very common in our days and the computing power is rising every day. ( now the cracking tools use  also GPU for a even faster speed).

Scrob cu pătrunjel verde – mâncarea burlacului

Am făcut rost de niște ouă proaspete. Îmi era o foame teribilă așa că am aplicat soluția simplă : scrob cu pătrunjel.

Se iau ouăle și se zbat până ajung spumă. ( de câteva ori cu furculița e de ajuns…depinde de optiunile fiecăruia)

Se ia ceva pătrunjel verde ( nu merge din ăla uscat ! )  și se toacă mărunt pe un fund de lemn.

Se amestecă cu ouăle bătute.

După gust, preferințe și opțiuni, se mai pot adăuga diverse ingrediente: bucățele de carne, ciuperci, etc…

Amestecul se pune într-o tigaie bine incinsă ( cu foarte puțin ulei). Se mai învârte ca să fie rumenirea uniformă.

Punem totul într-o farfurie.

Poftă bună !