BIPEDU

my ideas in action

Category Archives: scripts

use ffmpeg to edit videos

I want to show here how to produce a video and what are the steps to post-production.

All these are using free and open source software. No need to be expert in computers , only a little bit of courage.

You can use this recipe for homemade videos that you post online, for your vlog/blog or whatever.

I use ffmpeg and Inkscape that are available for Linux and Windows also.

As a general rule, if you have Linux/Unix, then please use the RAM since it is faster (read/write in /tmp folder always for all temporary files).

 

These are the steps :

1= record the video + audio

use a decent camera (preferable HD or 4K) with preferable audio . DSLR is OK also.

If your camera do not have good audio recording from builtin mic, then record the audio separate and clap in front of camera so that you can synchronize video/audio in post-production. Preferably use an external microphone ( ex. lavalier) as close as possible of the sound source. A cheap and good lav mic is Boya BM-M1 (see Amazon)

You can also record with a smartphone but be aware that the sample rate may be 44100 instead of 48000 as camera will record. This may produce audio that is not in sinc with video.

During recording please keep 5-10 seconds of silence , because you may need it later to do some noise reduction (see next step)

2= extract the audio separated and do some processing (if necessary)

Personally I use Audacity ( free and open source) .

I import audio and I do mainly two operations : Noise reduction and Compression. I use default settings from Audacity .. are ok for most cases. In case the audio sample rate is on 44100 then it is preferable to change it to 48000 to match the video.

Save the audio as WAV ( 16bit) in 48000.

3= then replace the video old audio with the new audio (new audio is with noise reduction)

3.1 = strip the audio from the video

ffmpeg -i /tmp/DSC_0009.MOV -an -c:v copy -c:a copy /tmp/p1.mov

3.2 = add new audio to the video

ffmpeg -i /tmp/p1.mov -i /tmp/DSC_0009.wav -vcodec copy -acodec copy /tmp/p2.mov

 

4= cut the video to the correct size

play first the new video and check the quality and note on paper the moments you want to save. For example : from 00:01:12 till 00:25:48 (hh:mm:ss format)

use example : cut from 20 sec till 9min and 36sec :

ffmpeg -i /tmp/p2.mov -vcodec copy -acodec copy -ss 00:00:20 -to 00:09:36 /tmp/p3.mov

5= (optional) add some logo/title/watermark

5.1 = Create the logo

With Inkscape (free and open source) create a image that has the same format as the video. Example if video is HD 1920*1080 then create a page that is 1920*1080. On this place your logo/watermark / artwork and preferable use alpha (transparency) to get more effect. Save the work and export it to PNG and keep the transparency.

5.2 = Overlay the PNG image to the video with ffmpeg

ffmpeg -i /tmp/p3.mov -loop 1 -i /tmp/TitleFile.png -filter_complex “[1:v]fade=in:st=0:d=0.1:alpha=1,fade=out:st=9:d=1:alpha=1[png];[0:v][png]overlay=x=0:enable=’between(t,0,10)'” -b:v 2000k -bufsize 2000k /tmp/p4.mov

Here I overlay the PNG image from second 0 ( st=0) in 0.1sec (d=0.1) and then fadeout starting at second 9 (st=9) for 1 second (d=1). The entire overlay is from seconds 0 to 10 ( see : between(t,0,10) ). Then I also force bitrate to be 2Mb/s (-b:v 2000k -bufsize 2000k) to avoid too much loss of quality. If not the ffmpeg will reduce the bitrate even more and it may be too low.

 

6= (optional) add a static picture at the final with credits/ links/Thanks or other announcements

6.1 = use Inkscape to create a closing image. Same as before, can use transparency (alpha) .

6.2= use ffmpeg to create and concatenate the image at the end of the video:

ffmpeg -i /tmp/p4.mov -loop 1 -framerate 30 -t 5 -i /tmp/Closing.png -f lavfi -t 0.1 -i anullsrc=channel_layout=stereo:sample_rate=48000 -filter_complex “[0:v][0:a][1:v][2:a]concat=n=2:v=1:a=1” -b:v 2000k -bufsize 2000k /tmp/final.mov

options :

framerate=30 : need to match the original framerate of your video ( if you shoot in 60FPS then use 60 here also)

t 5 : generate 5 seconds video that will show your Closing.png image

stereo : add stereo sound if youd video is stereo ( if not you can put mono)

sample_rate=48000 : use the same sample rate as your audio (other option is 44100)

 

7=(optional) post the video online ( Ex: Youtube, Vimeo……etc)

 

 

Advertisements

email me the result of a cronjob/script in Freenas

This is the simplest method to email the result of a command in Freenas.
For example if you run certain scripts with Cron you can use it also.

Personally I use this to get the SMART report about a HDD that may fail soon. So what I put in Cron this (all in one line) :

smartctl -a /dev/ada1 | /usr/bin/mail -s "MyFREENAS HDD /ada1 report " my.email@address.com

For user you can put “root” and any redirect should be off.

Of course to make the email work you have to configure the email server, ..etc… in the user config. Fill the settings here : System –> Email.

how to add/replace disk to FreeNAS mirror

I recently had to add new disks to my FreeNAS storage. I mounted the disks but I was not able to add them as mirror from GUI interface. But I found this Russian website with very simple and easy to follow tutorial. I copied here mainly for me as a reminder but may be also useful for others.

I hope the original author will not be upset.

The original post is here :
http://ukhov.ru/node/431


The high-level steps are:

Add the new disk to the system – (means connect the cables)
Partition the disk with gpart – (from FreeNAS terminal)
Attach the new partition to ZFS as a mirror

Create the GPT

Use the GUI to find the device ID of the new drive or use camcontrol.

# camcontrol devlist
at scbus2 target 0 lun 0 (ada0,pass0)
at scbus3 target 0 lun 0 (ada1,pass1)
at scbus4 target 0 lun 0 (ada2,pass2)
at scbus5 target 0 lun 0 (ada3,pass3)
at scbus7 target 0 lun 0 (da0,pass4)

let assume that our target is ada1. Create the GUID partition table for ada1.

# gpart create -s gpt ada1
ada1 created


Add the Swap Partition

Create a swap partition matching what FreeNAS created on the original drive. FreeNAS puts a swap partition on every data drive by default, stripes them together, and encrypts with a temporary key each boot. I’m not sure how that works when a drive fails, but it’s the recommended configuration.

# gpart add -b 128 -i 1 -t freebsd-swap -s 2G ada1
ada1p1 added

Add the Data Partition

Use the remaining space for the data partition.
# gpart add -i 2 -t freebsd-zfs ada1
ada1p2 added

Get the GPTID for the Partition

A device may change names depending on the connected port but the GPTID doesn’t change. FreeNAS uses the GPTID to track disks and so we want the rawuuid field of ada1p2.


# gpart list ada1
Geom name: ada1
scheme: GPT
1. Name: ada1p1
Mediasize: 2147483648 (2.0G)
...
rawuuid: 38d6835c-4794-11e4-b95b-08606e6e53d5
2. Name: ada1p2
Mediasize: 1998251364352 (1.8T)
...
rawuuid: 40380205-4794-11e4-b95b-08606e6e53d5

Attach to ZFS as mirror

Attach the partition using zpool which will begin the resilvering process. You will need the GPTID of the encrypted original disk parition.

# zpool attach
# zpool attach storage /dev/gptid/1c5238f9-5e2d-11e3-b7e0-08606e6e53d5 /dev/gptid/40380205-4794-11e4-b95b-08606e6e53d5

auto mount your NAS share

I have discovered a better way on mounting the NAS shares .

Previously I was using /etc/fstab but this was not very convenient.

So I found “autofs”. It is taking care of automounting the CD/DVD/ external shares (NFS/CIFS, etc) based on the usage. I mean that when you click on the folder then it will mount it , but when not using the mount folder then at a predefined timeout time it will umount it . This is a good feature for me since I wanted to unmount automatically when I do not use that share.

 

So after installing the autofs ( from repository) then you have to configure two files.

 

First is /etc/auto.master. You should put here the timeout setting and the mount directory.

If you have more mounting paths you can put different timeout settings.

My example below:

file:   /etc/auto.master

+auto.master
/mnt/nfsfreenas /etc/auto.misc --timeout 60 --ghost

The second file is /etc/auto.misc . This contain the mount settings ( somehow similar with /etc/fstab file)

My example file here:

file : /etc/auto.misc

bsddataset    -fstype=nfs,soft,sync,rsize=6000,wsize=6000    192.168.1.20:/mnt/data1/bsdshare

So what I have here is a Freenas Unix share on 192.168.1.20 ( bsdshare) that is NFS type.

You can put in this file the CIFS/Samba also or sshfs or other shares. CD/DVD disks work also.

at the end , after updating this two files with your setup please restart the autofs service with

service autofs restart

 

 

 

 

easy backup system with rsync – like Time Machine

Backup systems are good for recovering in case of accidental  lost data. But a more useful feature is the incremental backup where you have access to various snapshots in time like the Time Machine on Apple is doing. To do this in Linux (or any Unix  or alike ) systems is actually very easy.

For example we make a backup every day ( or every internal you want) . We need that  the amount of data transferred is small and not big. Imagine transferring few TB every day ! in case our important data is changing a little bit then we will backup only the modified parts. For this Rsync is the best tool. Everybody knows that. But there is a problem. How can we keep daily snapshots of the data without filling the disk ? For this we will use softlinks,  hardlinks and Rsync options.

So we have to create a script file like this:

#!/bin/bash
date=`date "+%Y-%m-%dT%H-%M-%S"`
rsync -aP --delete --log-file=/tmp/log_backup.log --exclude=lost+found --link-dest=/mnt/sdb2/Backups/current /mnt/sda1/ /mnt/sdb2/Backups/back-$date
rm -f /mnt/sdb2/Backups/current
ln -s /mnt/sdb2/Backups/back-$date /mnt/sdb2/Backups/current

So here I make first a “date” variable that will be used in the name of the backup folder to easily know when that backup/snapshot was made.

Then use the rsync with some parameters (see man rsync for more details):

-a = archive mode ( to send only changed parts)

-P = to give a progress info – (optional)

–delete = to delete the deleted files from backup in case they are removed from source

–log-file = to save the log into a file (optional)

–exclude = to exclude some folders/files from backup . This are relative to source path !!! do not use absolute path here !

–link-dest = link to the latest backup snapshot

/mnt/sda1 = source path (here I backup a whole drive)

/mnt/sdb2/Backups/back-$date  = destination folder , it will contain all the content from the source.

Then by using rm I remove the old link to the old backup ( the “current” link) and then I replace it with a new soft link to the newly created snapshot.

So now whenever I click on “current” I go in fact to the latest backup .

And because every time I make the backup the date is different the old snapshots will be kept. So for every day I will have a snapshot.

To automate this you have to create a cron job to execute the above script at the convenient time.

Example to run at 4:01AM every day:

1  4 * * * /path/to/script

Please notice that only the first time the full backup will take a long time since it will copy the full data. The second time you will run the script it will transfer only the changed files/bits.

Now on the destination folder you will see a “back-xxx” folder for every time you run the script. You can open/read the files from all this folders as it if they are completely independent files. In fact if you run df and du you will see something interesting.

For example if the backup is 600GB and the script is run every day you will see that the df will show the same 600GB used from disk space. But if you run “du -sh /* ”  you will see that each “back-xxx” folder is 600GB each. This is possible because there are only hardlinks to the same data copied. Do not worry, the disk is not full and you should trust the df results and not the du results.

user@box:/mnt/sdb2/Backups$ du  -sh ./*
623.8G    ./back-2014-02-24T17:47:12
623.8G    ./back-2014-02-24T21-46-41
623.8G    ./back-2014-02-25T17-05-02
623.8G    ./back-2014-02-25T18-45-34
0    ./current
user@box:/mnt/sdb2/Backups$ df /mnt/sdb2
Filesystem                Size      Used Available Use% Mounted on
/dev/sdb2                 2.7T    623.9G      1.9T  24% /mnt/sdb2

So the Time Machine is in fact only 3 lines of code in a script plus a cron job ! Easy and everybody can do it !

Adapt the script to your needs. Run it when you want with cron jobs.

At any point in time you can delete old backups ( for example backups older than few weeks). This can also be made with cron plus some scripts.

Tiny core linux no autologin

on Tiny core Linux distribution the tc user is automatically login at boot time.

If you want this not to happen then you have to change /opt/bootsync.sh file and to add this line :

echo “booting” > /etc/sysconfig/noautologin

and then backup with backup command. Of course all this must be run from root account.

install Debian on USB stick for a ATOM board based server

I have a Intel ATOM board D525MW and I use this for a small home server. Normally the Intel support for Linux is not great but the ATOM boards are special. They are forced (by Intel) to boot only from disks that have FAT bootable partition. So how to install Linux ( Debian Wheezy in my case) ? First I have to trick the ATOM board BIOS to think that the USB stick boots from FAT partition . To do this I have to create a small ( 10MB ) partition formatted with FAT16. I set this to be bootable. But the partition is empty and is never used. Then I create a second partition with EXT2 ( or ext3, ext4…) and I install my Debian on it. So the Debian system will be installed in to a second partition.
How to create a USB stick with this arrangement ? Answer : with fdisk.

Insert the USB stick into a USB port and from linux terminal type (you must be root !):

# fdisk /dev/sdb (or sdc or sdd or whatever is the device name in your case)

then delete (d) all the partitions and create (n) a new FAT16 partition with 10MB and set it bootable

Then write (w) the changes to the disk and unplug the USB stick.

Then plug it again so that the OS read the new partition.

Then create (n) a second partition in the available space. For this use a Linux compatible format like ext2, ext3 or ext4. You can create now any partition scheme you want.

Then write (w) the changes to the disk and unplug the USB stick.

Then plug it again.

Now you are ready to format the partitions.

Use mkfs.msdos to format the FAT16 partition

example of the command used :

# mkfs.msdos -v -F 16  -n LABEL /dev/sdb1

Use mkfs.ext2 to format the second partition with command:

# mkfs.ext2 -v /dev/sdb2

Then umount the USB stick and unplug and plug it again.

Now you are ready to install the Debian ( or whatever distro you want). As a reminder the special arrangement with FAT16 partition is only for Intel ATOM boards because the manufacturer did not solved yet this BIOS bug.

The Debian installation is using debootstrap method. In this method the Debian OS is installed directly on a USB stick. The is no Live install ,there is no CD/DVD involved, there is no ISO burning. You need just a Linux machine, a internet connection and a USB stick formatted like described above.

First create a empty folder to mount the USB stick (preferable on /tmp/usb ). I will assume here that the USB stick is sdb and the sdb1 is FAT16 and sdb2 is ext2 partition. I will assume that we install Debian Wheezy 32bit (i386) with GRUB and no GUI, standard system . The only server installed will be SSH but from there you can add whatever you want ( FTP, Samba, Web , …). All the commands below are run from root account.

# mount /dev/sdb2 /tmp/usb/
# debootstrap --arch=i386 wheezy /tmp/usb/ http://ftp.be.debian.org/debian

adapt the command to your needs. read “man debootstrap” if you are not sure. This step will take some time ( 20-30 minutes depending on your network speed)

Then configuration of the new Debian image.

# mount -t proc none /tmp/usb/proc
# mount -t sysfs none /tmp/usb/sys
# mount -o bind /proc /tmp/usb/proc
# mount -o bind /dev /tmp/usb/dev
# mount -o bind /sys /tmp/usb/sys
# LANG=C chroot /tmp/usb/ /bin/bash

Now you enter in chroot mode. The commands that you execute now are inside the new Debian Wheezy OS.

root@debian # mount devpts /dev/pts  -t devpts
root@debian # blkid

the purpose is to find UUID identifier of the dev/sdb2 partition because we will need later the UUID code.

To edit the configuration files you can use various methods: nano, vi, etc… ( personaly I prefere nano)

next edit the /etc/fstab file

root@debian # nano /etc/fstab

and add there the folowing :

# UNCONFIGURED FSTAB FOR BASE SYSTEM
proc /proc proc defaults 0 0
UUID=insert-here-your-UUID / ext2 defaults,noatime 0 1
#/dev/sda1 / ext3 defaults,noatime 0 1
tmpfs /tmp tmpfs defaults,noatime 0 0
tmpfs /var/tmp tmpfs defaults,noatime 0 0
tmpfs /var/run tmpfs defaults 0 0
tmpfs /var/log tmpfs defaults 0 0
tmpfs /var/lock tmpfs defaults 0 0

Then edit your network card config file: nano /etc/network/interfaces

auto lo
iface lo inet loopback
auto eth0
iface eth0 inet dhcp

Then configure your local timezone with

root@debian # dpkg-reconfigure tzdata

then edit you server network name ( the machine name in the local network)

root@debian # nano /etc/hostname

Then edit the sources.list for the apt-get/aptitude:

root@debian # nano /etc/apt/sources.list

add there the folowing (edit them to match your needs) :

deb http://ftp.be.debian.org/debian wheezy main
deb-src http://security.debian.org/ wheezy/updates main
deb http://secirity.debian.org/ wheezy/updates main

Then make refresh for the aptitude packages with:

root@debian # aptitude update

next install the locales :

root@debian # aptitude install locales
root@debian # dpkg-reconfigure locales

Next install console-data, linux image and grub:

root@debian # aptitude install console-data
root@debian # aptitude install linux-image-486
root@debian # aptitude install grub

next edit the grub config file like this:

root@debian # nano /etc/default/grub

and inside add something like this :

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
#GRUB_CMDLINE_LINUX_DEFAULT="verbose console=ttyS0,38400n8 reboot=bios"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=38400"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL=console
#GRUB_TERMINAL=serial

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_LINUX_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

Then open a new terminal in your Linux machine ( outside the chroot !) and add this in the /boot/grub/device.map :

(hd0) /dev/sdb

Then update the grub configuration (you are now back on the chroot environment) :

root@debian # grub-install /dev/sdb
root@debian # update-grub

Please check and edit the /boot/grub/device.map so that your USB stick is on hd0 and not hd1.

Then check if you have hd1 in the /boot/grub/grub.cfg. If yes then please replace it with hd0

Now change the root password for your new Debian Wheezy:

root@debian # passwd root

Next install the utility packages like sudo, ssh-server, standard Debian packages…or whatever you want to install :

root@debian # aptitude install rsyslog sudo
root@debian # tasksel install standard
root@debian # tasksel install ssh-server

now the installation is finished and you need to exit from chroot environment. Please be careful that if you do not use the correct exit method you can damage the newly installed OS.

So do :

root@debian # umount /dev/pts
root@debian # exit

So now you are not in chroot anymore and you are back at your linux terminal.

Now umount :

# umount /tmp/usb/proc/
# umount /tmp/usb/sys
# umount /tmp/usb/dev
# umount /tmp/usb
# umount /dev/sdb2

Now you are ready to unplug the USB stick and to use it on the ATOM board !

FTP backup with BASH script ( FTP scripting)

FTP protocol is used to transfer data between computers. The user has also a possibility to combine bash scripts with FTP to automate the backups of the files. This concept can be used for example to backup some files from a local machine to a remote server.
The way to do this is by making an executable script that is run from time to time by a cron job task. In this way the backup is made automatically in the background and do not require user intervention. But there is a problem. if you use FTP in command line then you have to type user name and password in clear text. So how to do it ?
The solution I suggest is like in the example below:

First step is to determine if the backup is needed . We check if the file was changed since last backup. For this we use a compare between the size of the file in this moment with the size of the file at the previous backup time. The previous size was saved in a file. To check this we use the “if compare” structure from BASH:

### check if the file was changed, and if YES then make FTP transfer, if not exit
 if [ `cat ~/.size_myfile.txt` -ne `du -s /path/to/file | cut -f1` ] ; then
 echo 'File changed !'
 fi
 ###########################

Then we define some parameters for FTP command. This definition can be made in a different ( hidden) file. For not critical situations ( home users) I recommend keeping the user details in the same file ( in the script) but to remove all the permissions for the other users ( use chmod command ). So the backup script should look like this:

-rwx------ myuser mygroup 32 june 12 14:52 backupFTP.script

Notice that only the “myuser” user has the right to read/write/execute the file

So for FTP command you need:

##############################
 HOST='ftp.myserver.com'
 USER='myusername'
 PASSWD='mypassword'
 FILE='/path/to/filesource filedestination'
ftp -n $HOST <<END_SCRIPT
 quote USER $USER
 quote PASS $PASSWD
 cd /path/to/destination
 put $FILE
 quit
 END_SCRIPT
 #############################

Since the source path and the destination path may be different you can use “cd /path/to/destination” for the file. The copied file can be also renamed as shown above ( see “filedestination“)

Notice that the commands between “END_SCRIPT” tags are executed inside FTP terminal. This are FTP commands and not BASH/Linux commands. You can put here whatever FTP commands you want based on your needs. For a full list of the FTP commands type “help” in the FTP terminal.

The 3rd step is to recalculate and save the new size of the file so that next time when backup script is run the size file is updated. For this we do:

################################
 ## recalculate the new size of the file, for next backup
 du -s /path/to/file | cut -f1 > ~/.size_myfile.txt
 env DISPLAY=:0 notify-send 'FTP backup done !' 'File : /path/to/file'
 ###############################

Optionally you can show a desktop notification that the backup was made. If you do not have a GUI then do not use it.

Next I show the full script in only one file:

#!/bin/bash
 ### check if the file was changed, and if YES then make FTP transfer, if not exit
 if [ `cat ~/.size_myfile.txt` -ne `du -s /path/to/file | cut -f1` ] ; then
#    echo 'File changed !'
 sleep 1
 #
 HOST='ftp.myserver.com'
 USER='myusername'
 PASSWD='mypassword'
 FILE='/path/to/filesource filedestination'
 ftp -n $HOST <<END_SCRIPT
 quote USER $USER
 quote PASS $PASSWD
 cd /path/to/destination
 put $FILE
 quit
 END_SCRIPT
 sleep 1
 ## recalculate the new size of the file, for next backup
 du -s /path/to/file | cut -f1 > ~/.size_myfile.txt
 env DISPLAY=:0 notify-send 'FTP backup done !' 'File : /path/to/file'
fi
 ###############

Screencast with FFMPEG with sound (Ubuntu 12.04 / Mint 13)

This is how to make a high quality screencast with sound from the soundcard.
Great if you want to record for example screencast tutorial or  a webpage that contains sound.
system: Ubuntu 12.04/ Mint 13

  • Record without sound, image size relative to dimensions of your screen (my case is 1600×900):

ffmpeg -f x11grab -s 1600×900 -r 25 -i :0.0 -sameq output.mkv

  • Record with sound, from your microphone:

ffmpeg -f alsa -i pulse -ab 192 -acodec pcm_s16le -f x11grab -s 1600×900 -r 25 -i :0.0 -sameq output.mkv

To record with sound, from your microphone you have to do also the following steps ( only first time):

* Install Pulse Audio Volume Control. (through synaptic package manager)
* Repeat the second command, recording starts.
* Go to Pulse Audio; go to the tab ‘Recording’, it shows ffmpeg recording the sound. Change the pull down menu to Monitor from Built in Analog Stereo.
Now it records the sound stream in stead of your micro. Set your microphone sound level to the desired values.
That is basically it.

After you save the file you will need to cut first few seconds and last few seconds to remove the parts where you start/stop the ffmpeg command.

This can be done with the below command.

Basically in the below example I extract from “output.mkv” the video starting from second 10 . The length of the output file (clip-output-file.mkv) is set to 120seconds.

ffmpeg -ss 10 -t 120 -i output.mkv -acodec copy -vcodec copy clip-output-file.mkv

print some columns from CSV file

If you have a CSV ( comma separated values file) and you need to print only some columns then you can use sed, awk or cut. Today I will show the “cut” command. Do not confuse it with “cat” !!

CUT is quite easy to use and is more simple than AWK or SED. For example if you have a file like this

$ cat datafile.csv

device1 device2 device2a device4b device5a device8 device9

1 56 8 99 5 41 36 8

22 5 99 89 56 56 2

1 0 2 5 9 63 5

As you can see the delimiter is not comma but space. If you want to print only columns 2,3 and 5 then “cut” is the best tool.

$ cut -d” ”  -f2,3,5 datafile.csv

device2 device2a device5a

56 8 5

5 99  56

0 2 9

-d” ” = means to use ” ” (space) as delimiter in the input file

-f2,3,5 = means to print only the fields 2, 3 and 5

So by default the output delimiter is space. If you want a specific delimiter, like tab, you can use this :

$ cut -d” ”  -f2,3,5 –output-delimiter=$’\t’  datafile.csv

device2         device2a        device5a

56                 8                       5

5                   99                     56

0                  2                        9

The syntax $’\t’ is special because cut do not accept TAB as “\t”

For another delimiter you can use the option like : –output-delimiter=”.:.”

$ cut -d” ”  -f2,3,5 –output-delimiter=$’\t’  datafile.csv

device2.:.device2a.:.device5a

56.:.8.:.5

5.:.99.:.56

0.:.2.:.9

Of course for all this there is a AWK or SED command but I think that for simple column selection a CUT command is easier.

More than that. The CUT command give also the possibility to select only some bytes or characters from the file. For example to print only the characters from 2 to 5 you have to use :

$ cut -c2-5 datafile.csv

evic

56

2 5

0

not very useful in my case but nice to have in case the input file format do not have a delimiter.

see more info with “man cut”