tail -f with highlighting

A little help with perl

http://sysadminnotebook.blogspot.com/2012/09/tail-f-with-highlighting.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+SysadminNotebook+%28Sysadmin+Notebook%29

Getting WAN IP on console or shell script.

There are few other guide in getting your current WAN IP on Linux console or fetching it within a shell script. But this one works for me.

1. To get the current WAN IP:

$> echo "$(wget http://automation.whatismyip.com/n09230945.asp -O - -o /dev/null)"

2. For LAN IP address/es:

$> ifconfig | grep 'inet addr:'| grep -v '127.0.0.1' | cut -d: -f2 | awk '{ print $1}'

 

 

Mount All Available CIFS Shares using GVFS

Here’s a bash script that will automatically mount all available SAMBA shares that has access permission for the current user.   The script composes of two part: (1) The script itself and (2) the user’s credentials.

But current problem is that the user must store the password to access the SAMBA shares in the keyring before the script runs smoothly. It requires the user to access the any of the permitted shares and store the password permanently to the keyring, then add the script to the Startup Applications(System Settings -> Startup Applications)

Autfile is a simple text file contain the following:

username = myuser
password = mypass
domain = localdomain
servername = localserver

Here’s the code:

#!/bin/bash
ARGS="$*"

BASEDIR="/home/rnartos/.scripts/uicsmb"
OPTYPE=""
CMDTYPE=""
FILTERARGS=`echo "$ARGS" | sed 's/ /#/g' | sed 's/-/ -/g'`
AUTHFILE="$BASEDIR/authfile"
NETBASENAME="WinShares"



if [ ! -f $AUTHFILE ]; then
   echo "ERROR: neither default or user-defined authfile not exist."
   exit 0
fi

for CMD in $FILTERARGS; do
   MCMD=`echo $CMD | sed 's/#/ /g'| awk '{print $1}' `

   #umount
   if [ "$MCMD" = "-u" ]; then
      OPTYPE=" -u "
   fi

   #help
   if [ "$CMD" = "-h"  ]; then
     echo "GFVS WinShares - rnartos"
     echo "$0 -u -a <authfile> -h"
     echo "-u umount current shares"
     echo "-a process authentication file"
     echo "-h this help message."
     exit 0
   fi

   #authfile
   if [ "$MCMD" = "-a"  ]; then
     PCMD=`echo $CMD | sed 's/#/ /g'| awk '{print $2}' `
     if [ "$PCMD" = ""  ]; then
        echo "ERROR:  parameters for -a <authfile>"
     else
         if [ -f $PCMD ]; then
             AUTHFILE="$PCMD"
         else
           echo "$PCMD authfile not exist."
           exit 0
         fi
     fi
   fi
done


SERVERNAME=`cat $AUTHFILE | grep servername | sed 's/=//g' | awk '{print $2}'`
DOMAINNAME=`cat $AUTHFILE | grep domain | sed 's/=//g' | awk '{print $2}'`
USERNAME=`cat $AUTHFILE | grep username | sed 's/=//g' | awk '{print $2}'`
PASSWORD=`cat $AUTHFILE | grep password | sed 's/=//g' | awk '{print $2}'`
CHECKMOUNT=`gvfs-mount -l | grep "smb:"`
GETIP=`nmblookup $SERVERNAME | grep -v "query" | awk '{print $1}'`
GETSHARES=`smbclient --debuglevel=0 -A $AUTHFILE -L $SERVERNAME -I $GETIP -g  2>/dev/null | grep "Disk" | sed 's/|/ /g' | awk '{print $2}'`
WINSHARENAME="$HOME/$NETBASENAME/$DOMAINNAME/$SERVERNAME"

if [ ! -d $WINSHARENAME ];  then
   mkdir -p $WINSHARENAME
fi


for NAMES in $GETSHARES; do

   CHECKSHARE=`smbclient //$SERVERNAME/$NAMES -A $AUTHFILE -c '' 2> /dev/null | grep 'DENIED'`

   if [ "x$CHECKSHARE" = "x" ]; then
     if [ "$OPTYPE" = "" ]; then
         if [ "$CHECKMOUNT" = ""  ]; then
           CMDTYPE="mounting"
         fi
     else
        CMDTYPE="umounting"
     fi
     if [ "$CMDTYPE" = "" ]; then
        echo "Shares Already mounted."
        exit 0
     else
        echo "$CMDTYPE $NAMES"
	if [ "$OPTYPE" = "" ]; then
           gvfs-mount $OPTYPE smb://$SERVERNAME/$NAMES 2> /dev/null
           GVFSNAME=`echo $NAMES | tr '[:upper:]' '[:lower:]'`
           ln -sf "$HOME/.gvfs/$GVFSNAME on $SERVERNAME" $WINSHARENAME/$NAMES
        else
           gvfs-mount $OPTYPE smb://$SERVERNAME/$NAMES 2> /dev/null
        fi
     fi
   fi
done

exit 0 

Novell iManager workstation on Ubuntu 11.04

Currently unning a Novell eDirectory  LDAP service and wonder how to use iManager on your Ubuntu workstation.

Here’s how:

1. Download the latest iManager Workstation for Linux in Novell Download Center

2. Extract the package (for me usually in $HOME/Desktop)

3. Install the necessary packages and libraries for alien (alien, libstdc++5, gcc), installing alien package also includes rpm in the requirements.

4.  Prepare NICI for installation.

$> cd  $HOME/Desktop/imanager
$> cd  NICI/linux
$> sudo alien -d --scripts nici.i386.rpm
$> sudo dpkg -i nici_2.7.3-1.01_i386.deb

5. Modify the current  iManager.sh in the bin folder ($HOME/Desktop/imanager/bin) and remove the word  ”function” in every functions of the script.

example:

function CHECK_PERMISSIONS()
{
 echo "Test permissions" > $IMANAGER_BIN_NATIVE_DIR/perm.txt
 ....
}

should be

CHECK_PERMISSIONS() 
{ 
echo "Test permissions" > $IMANAGER_BIN_NATIVE_DIR/perm.txt 
.... 
}

5.  In the STARTMANAGER() function, remove the CHECK_NICI;

6.  Save the script and do  some test run.

$> cd $HOME/Desktop/imanager/bin
$> ./iManager.sh

7. Done.

Ubuntu 11.04 Shutdown and Restart Problem with CIFS

UPDATE(07/29/2011): The GRUB thing didn’t work at all, back to basic trapping signal via upstart scripts in /etc/init/dbus.conf.  I tried it before using /etc/init/network-manager.conf but on Ubuntu 10.10, it’s not working anymore. Here’s another test and works for me.

1. Modify the current /etc/init/dbus.conf.

$> sudo vi /etc/init/dbus.conf

2. Add a pre-stop script, which looks like this:

pre-stop script
       trap "TERM signal" TERM
      /bin/umount -a -t cifs -l -f
      trap - TERM
end script 

3. Save the script and have a test.

4. Done.

UPDATE(07/28/2011): Found a better alternative via GRUB, link here. Mainly you just need to modify the  /etc/default/grub. Here’s the details:

1. Edit the /etc/default/grub

$> sudo vi /etc/default/grub

2. Add “reboot=pci” on the GRUB_CMDLINE_LINUX line, it should look something like this:

... GRUB_CMDLINE_LINUX="reboot=pci" ...

3. Update the grub

$> sudo update-grub

 

4. Done

—————

It’s an ancient bug (here), which still exist in Ubuntu 11.04, or even in other distros. When you got a mounted samba shares before the shutdown or restart process, the machine waits for around 10 minutes before it complete the operation. Very troublesome that’s why I tried various workaround but none of them  works with Ubuntu 11.04. Not even the /etc/rc6.d/K*  or the upstart /etc/init configurations, nor the old python script I posted before (here’s the link).

After few considerations, I made a desperate workaround, creating a script that triggers before the /sbin/shutdown, /sbin/reboot and /sbin/restart commands.

1. Rename the current shutdown, reboot and restart commands in /sbin.

#> mv /sbin/shutdown /sbin/shutdown2
#> mv /sbin/reboot /sbin/reboot2
#> mv /sbin/restart /sbin/restart2 

2. Then create scripts with names of the previous commands in /sbin, which contains the following:

#!/bin/sh
umount -t cifs -a -f -l
/sbin/shutdown2 $@
exit 0 

3. Make similar script for reboot and restart command which also points to /sbin/reboot2 and /sbin/restart2.

4.  Until the dbus implementation of stop on deconfiguring-networking comes to Ubuntu 11.04, which I think working with
Ubuntu 11.10 oneiric. I think this is the least workaround that works for me.

5. Done

DropBox Like using Lsyncd

I have several notebooks that is connected to our File Server via OpenVPN, the problem is sometimes the connection from and to the server is slow and problematic, cannot browse files or takes to long to upload or download things, and the idea of DropBox like application or tool will eventually fix things around those problems.

I found this old link “How to build your own dropbox clone ” (link here) and gives me an idea on howtos.

The key tool is lsyncd (http://lsyncd.googlecode.com)

Lsyncd watches a local directory trees event monitor interface (inotify). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes. By default this is rsync. Lsyncd is thus a light-weight live mirror solution that is comparatively easy to install not requiring new filesystems or blockdevices and does not hamper local filesystem performance.

Here’s  the details:

1.  From you current Ubuntu Desktop, install rsync and other necessary packages.

$> apt-get install rsync libxml2-dev build-essential lua5.1

2. Then download the lsyncd souce code and compile.

$> wget http://lsyncd.googlecode.com/files/lsyncd-2.0.4.tar.gz
$> tar xvfz  lsyncd-2.0.4.tar.gz
$> cd lsyncd-2.0.4 $> sudo -s
#> ./configure 
#> make; make install

3. Then create a configuration file in your $HOME/.config/lsyncd path called lsyncd.lua

$> mkdir -p  ~/.config/lsyncd
$> vi ~/.config/lsyncd/lsyncd.lua

4.  The content as follows:

sync{default.rsyncssh, source="/home/username/mybox", host="server.localdomain", targetdir="mybox/", rsyncOps="-ltus"}

5.  Save the file, and before the sync test, make sure you have a passwordless ssh session to the given host, if not, follow this steps:

On your Desktop/client:

$> ssh-keygen -N '' -f  ~/.ssh/id_dsa

On paraphrase prompt, just press ENTER, then the following (note: REMOTE_SERVER is the ssh server host)

$>  cat ~/.ssh/id_rsa.pub | ssh REMOTE_SERVER 'cat - >> ~/.ssh/authorized_keys2' $> ssh REMOTE_SERVER 'chmod 700 .ssh' 

Then have a test

$> ssh REMOTE_SERVER

6. When all are successfully done, create your preferred name for the sync folder, mine.. I just called it “mybox”. It should have the path as follows:

/home/username/mybox
server.mydomain.local:~/mybox

7.  All are set and ready to go, test the lsyncd.

$> lsyncd -nodaemon ~/.config/lsycnd/lsycnd.lua

8. Populate the local folder and watch the remote ssh server for changes.

9. Done.

Note: This is currently a definitely Linux workaround, haven’t tried it in other OS (using cygwin on Windows).

Document Converter for Multiple Files

I’ve found a python script that uses OpenOffice.org’s UNO library. The script can convert from Open Document formats (.odt, .ods, .odp) to Microsoft Office Formats (.doc, .xls, .ppt) or to comma separated value (.csv), a PDF or even to a text file (.txt).

It works great, but still I need some enhancement to make it more usable in my current situation. I need to convert  multiple files located in different folders inside a common path. So I created a bash script that can convert them faster and more efficient than doing it manually.

The script requires a running OpenOffice.org as service on port 8100 and python 2.4 or 2.6.  The script automatically create a headless OpenOffice.org service and terminate it afterwards.

You can get the python script here: DocumentConvert.py.

Here’s my script:

#!/bin/bash
# Document Converter Main
# To convert various document to different types (.doc, .xls, .odt, .ods, .ppt, .odp, .pdf)


#START OPENOFFICE HEADLESS MODE
DOCUMENTCONVERT=$HOME/.scripts/documentconverter/DocumentConverter.py
/usr/bin/soffice -headless -nologo -nofirststartwizard -accept="socket,host=127.0.0.1,port=8100;urp" & > /dev/null
sleep 2
GET_PID_SOFFICE=$(ps aux | grep "soffice.bin" | grep -v "grep" | grep "8100" | awk '{print $2}')

TEMP_FILE=/tmp/doc_source.txt
TARGET_DIR=$(zenity --file-selection --directory --title="Select Target Directory")

DOCTYPE=$(zenity --entry  --text="$RETURNVAL Enter Source File Extension" --title="Souce Document Type" --width=250)
SOURCE_DOCTYPE=$(echo "$DOCTYPE" | sed 's/[^a-zA-Z0-9]//g')

DOCTYPE=$(zenity --entry  --text="$RETURNVAL Enter Target File Extension" --title="Target Document Type" --width=250)
TARGET_DOCTYPE=$(echo "$DOCTYPE" | sed 's/[^a-zA-Z0-9]//g')

find "$TARGET_DIR" -iname "*.$SOURCE_DOCTYPE" > $TEMP_FILE

cat $TEMP_FILE | \
while read FILE; do
 FILENAME=$(basename "$FILE")
 GETFILENAME=${FILENAME%.*}
 PATHNAME="$( readlink -f "$( dirname "$FILE" )" )"
 python $DOCUMENTCONVERT "$PATHNAME/$FILENAME"  "$PATHNAME/$GETFILENAME.$TARGET_DOCTYPE"
done


kill -9 $GET_PID_SOFFICE
exit 0

 

 

Automatically shutdown KVM Virtual Machines

Currently, there’s no built-in tool to properly shutdown VMs  when the KVM host does. So after couple of hour “googling” it I got some idea from this link, made some changes to make simplier. Here’s my version:

#!/bin/bash

LIST_VM=`virsh list | grep running | awk '{print $2}'`
TIMEOUT=90
DATE=`date -R`
LOGFILE="/var/log/shutdownkvm.log"

if [ "x$activevm" =  "x" ]
then
 exit 0
fi

for activevm in $LIST_VM
do
 PIDNO=`ps ax | grep $activevm | grep kvm | cut -c 1-6 | head -n1`
 echo "$DATE : Shutdown : $activevm : $PIDNO" >> $LOGFILE
 virsh shutdown $activevm > /dev/null
 COUNT=0
 while [ "$COUNT" -lt "$TIMEOUT" ]
 do
ps --pid $PIDNO > /dev/null
if [ "$?" -eq "1" ]
then
COUNT=110
else
sleep 5
COUNT=$(($COUNT+5))
fi
done
if [ $COUNT -lt 110 ]
then
echo "$DATE : $activevm not successful force shutdown" >> $LOGFILE
virsh destroy $activevm > /dev/null
fi
done

1. Save the code in /etc/init.d/shutdownvm
2. Then make it an executable file

   chmod 755 /etc/init.d/shutdownvm

3. Create links to both rc0.d and rc6.d

   cd /etc/rc0.d ln -s ../init.d/shutdownvm K18shutdownvm cd /etc/rc6.d ln -s ../init.d/shutdownvm K18shutdownvm

4. Done.

Automatically unmount CIFS before Network-Manager Ubuntu 10.04

Even though my previous article already fixed my problem (esp. on Ubuntu 10.04), browsing the net last night and found this script which is way much easier and more effective than my previous:

How to use the script:

1. Save it in your personal script folder (mine usually $HOME/bin), for now I named it umountcifs.sh

2. Make the script executable

$> chmod +x $HOME/bin/umountcifs.sh

3. Add it in your Gnome’s startup application. (System > Preferences > Startup Applications)

4. Modify the file /etc/sudoers  or you can issue

$> sudo visudo

5. Add the following line or append it line with

ALL ALL=(ALL) NOPASSWD: /etc/init.d/umountnfs.sh

6. Done.

Here’s the scripts:

#!/usr/bin/env python

#Author: Seamus Phelan

#This program runs a custom command/script just before gnome shuts
#down.  This is done the same way that gedit does it (listening for
#the 'save-yourself' event).  This is different to placing scipts
#in /etc/rc#.d/ as the script will be run before gnome exits.
#If the custom script/command fails with a non-zero return code, a
#popup dialog box will appear offering the chance to cancel logout
#
#Usage: 1 - change the command in the 'subprocess.call' in
#           function 'session_save_yourself' below to be what ever
#           you want to run at logout.
#       2 - Run this program at every gnome login (add via menu System
#           -> Preferences -> Session)
#
#

import sys
import subprocess
import datetime

import gnome
import gnome.ui
import gtk

class Namespace: pass
ns = Namespace()
ns.dialog = None

def main():
    prog = gnome.init ("gnome_save_yourself", "1.0", gnome.libgnome_module_info_get(), sys.argv, [])
    client = gnome.ui.master_client()
    #set up call back for when 'logout'/'Shutdown' button pressed
    client.connect("save-yourself", session_save_yourself)
    client.connect("shutdown-cancelled", shutdown_cancelled)

def session_save_yourself( *args):
    #Unmount those CIFS shares!
    retcode = subprocess.call("sudo /etc/init.d/umountnfs.sh", shell=True)
    if retcode != 0:
        #command failed
        show_error_dialog()
    return True

def shutdown_cancelled( *args):
    if ns.dialog != None:
        ns.dialog.destroy()
    return True

def show_error_dialog():
    ns.dialog = gtk.Dialog("There was a problem running your pre-shutdown script",
                           None,
                           gtk.DIALOG_MODAL | gtk.DIALOG_DESTROY_WITH_PARENT,
                           ("There was a problem running your pre-shutdown script - continue logout", gtk.RESPONSE_ACCEPT))
    if ns.test_mode == True:
        response = ns.dialog.run()
        ns.dialog.destroy()
    else:
        #when in shutdown mode gnome will only allow you to open a window using master_client().save_any_dialog()
        #It also adds the 'Cancel logout' button
        gnome.ui.master_client().save_any_dialog(ns.dialog)

#Find out if we are in test mode???
if len(sys.argv) >=2 and sys.argv[1] == "test":
    ns.test_mode = True
else:
    ns.test_mode = False

if ns.test_mode == True:
    main()
    session_save_yourself()
else:
    main()
    gtk.main()

Toggle USB Flash Disk access on/off on Ubuntu

In a series of test I’ve made on the built-in access permission control regarding the USB Flash Drives in Ubuntu, in all of my test it fails. The “Access External Storage Devices” control in Users & Groups Management, for me is not working (or I haven’t discovered how it really works). Then I just came up with this script:

#!/bin/bash
# Toggle on/off for USB Storage
# by http://hardc0l2e.wordpress.com
# KERNEL=="sd*[!0-9]|sr*", ENV{ID_SERIAL}!="?*", SUBSYSTEMS=="usb", IMPORT{program}="usb_id --export %p"
# ------------------------------

TARGET_NAME="/etc/udev/rules.d/10-disable-usb.rules"
PATTERN='SUBSYSTEMS=="usb"'
CHECK_ENABLED=`grep "$PATTERN" /etc/udev/rules.d/* | grep "OPTIONS"`
ENV_CHECK="no"

if [ "$ENV_CHECK" = "yes" ]; then
 if [ "x$CHECK_ENABLED" = "x" ]; then
 echo 'KERNEL=="sd*[!0-9]|sr*", ENV{ID_SERIAL}!="?*", SUBSYSTEMS=="usb", IMPORT{program}="usb_id --export %p", OPTIONS+="ignore_device"' > $TARGET_NAME
 fi
else
 if [ "x$CHECK_ENABLED" != "x" ]; then
 echo 'KERNEL=="sd*[!0-9]|sr*", ENV{ID_SERIAL}!="?*", SUBSYSTEMS=="usb", IMPORT{program}="usb_id --export %p"' > $TARGET_NAME
 fi
fi

To toggle ON/OFF just replace the variable ENV_CHECK=”yes” to disable and ENV_CHECK=”no” to enable USB Flash Disk access.

Done.