====== General linux tips and tricks ======


===== Limit the CPU usage of an application (process) - cpulimit =====

=== Installation: ===

Download last stable version of [[http://downloads.sourceforge.net/cpulimit/cpulimit-1.1.tar.gz|cpulimit]]
Then extract the source and compile with make:

    tar zxf cpulimit-xxx.tar.gz
    cd cpulimit-xxx
    make

Executable file name is cpulimit. You may want to copy it in /usr/bin.

Usage:
Limit the process 'bigloop' by executable name to 40% CPU:

    cpulimit --exe bigloop --limit 40
    cpulimit --exe /usr/local/bin/bigloop --limit 40

Limit a process by PID to 55% CPU:

    cpulimit --pid 2960 --limit 55 

<note tip>
If your machine has one processor you can limit the percentage from 0% to 100%, which means that if you set for example 50%, your process cannot use more than 500 ms of cpu time for each second. But if your machine has four processors, percentage may vary from 0% to 400%, so setting the limit to 200% means to use no more than half of the available power. In any case, the percentage is the same of what you see when you run top. 
</note>

===== Move a running process into a screen session =====
[[http://isteve.bofh.cz/~isteve/knowledgebase/articles/screenify.html]]:
<code bash>
#!/bin/sh
# Copyright Timo Lindfors 2004
usage () {
    echo usage: $0 pid
    exit 1
}
TCGETS=0x5401
TCSETS=0x5402
SIZEOF_STRUCT_TERMIOS=60
O_RDWR=2
((FLAGS=O_RDWR))

PID=$1
if [ x`which gdb` == x ]; then
    echo gdb not found in PATH. Please apt-get install gdb
    exit
fi
if [ x$PID == x ]; then
    usage;
fi
if [ x$2 != x ]; then
    usage;
fi
MYPID=$$
MYFD0=`readlink /proc/$MYPID/fd/0`
MYFD1=`readlink /proc/$MYPID/fd/1`
MYFD2=`readlink /proc/$MYPID/fd/2`
EXE=`readlink /proc/$PID/exe`
if [ x$EXE == x ]; then
    echo $0: $PID: no such pid
    exit 1
fi

BATCHFILE=`mktemp /tmp/gdb.XXXXXXXXXXXX`
cat >$BATCHFILE <<EOF
file $EXE
attach $PID
call malloc($SIZEOF_STRUCT_TERMIOS)
call malloc($SIZEOF_STRUCT_TERMIOS)
call malloc($SIZEOF_STRUCT_TERMIOS)
call ioctl(0, $TCGETS, \$1)
call ioctl(1, $TCGETS, \$2)
call ioctl(2, $TCGETS, \$3)
call close(0)
call close(1)
call close(2)
call open("$MYFD0", $FLAGS)
call open("$MYFD1", $FLAGS)
call open("$MYFD2", $FLAGS)
call ioctl(0, $TCSETS, \$1)
call ioctl(1, $TCSETS, \$2)
call ioctl(2, $TCSETS, \$3)
call free(\$1)
call free(\$2)
call free(\$3)
detach
EOF
gdb -batch -x $BATCHFILE >/dev/null 2>&1 </dev/null
rm $BATCHFILE

cat <<EOF
Process $PID should now be talking to this pty. Refresh the screen
(e.g. CTRL+L) and have fun!
EOF

exec tail -f --pid=$PID /proc/$PID/stat
</code>

===== Force a clean reboot when the system freezes =====

If you manage to freeze your system in such a way that even ''Ctrl+Alt+Del'' doesn’t work anymore, the mouse is stuck on the screen any none of the key combinations will work, don’t think of the reset button just yet.

Hold down the **Right Alt** and **SysRq** keys and press this sequence:

**R E I S U B**

<code>
R: Switch the keyboard from raw mode to XLATE mode
E: Send the SIGTERM signal to all processes except init
I: Send the SIGKILL signal to all processes except init
S: Sync all mounted filesystems
U: Remount all mounted filesystems in read-only mode
B: Immediately reboot the system, without unmounting partitions or syncing
</code>

This will cleanly unmount drives, terminate processes and nicely reboot your machine.

<note important>**If you suspect memory corruption killed your kernel, be careful with Sync option. It may flush corrupted disk buffers over your filesystem and cause more headache after reboot.**</note>

**also for more detailed info see:** [[http://en.wikipedia.org/wiki/Magic_SysRq_key]]

===== suid, sgid, sticky bit, permissions =====
     * **4000 (chmod u+s)** is suid; for files execute as owning user (often root).
     * **2000 (chmod g+s)** is sgid; for files execute as owning group (often root); for directories the group on newly created files will be set to the directory’s group rather than the creator’s group. Typically used for shared directories.
     * //suid and sgid are ignored on scripts, due to the security risk//
     * **1000 (chmod +t)** is sticky bit (”save text image”); for files it used to be ‘pin in memory’ but is now ignored; for directories only root, file owner and directory owner can delete a file (even if non-owners have directory write permissions). Typically used for /tmp. ''—t——''
     * capital letters when doing ls -al usually means the permissions have been set incorrectly eg -''r-S——'' SUID is set, but owner execute is not set. However ''-rw——T'' means no update of “last modified time”; usually used for swap files (not very common nowadays - swap is usually a partition).

===== Four Tweaks for Using Linux with Solid State Drives =====
[[http://en.wikipedia.org/wiki/Solid-state_drive|SSDs (solid state drives)]] are great. They’re shock resistant, consume less power, produce less heat, and have very fast seek times. If you have a computer with an SSD, such as an Eee PC, there are some tweaks you can make to increase performance and extend the life of the disk.

   * The simplest tweak is to mount volumes using the noatime option. By default Linux will write the last accessed time attribute to files. This can reduce the life of your SSD by causing a lot of writes. **The noatime mount option turns this off**.

   sudo gedit /etc/fstab

Ubuntu uses the relatime option by default. For your SSD partitions (formatted as ext3), replace relatime with noatime in fstab. Reboot for the changes to take effect.

   * Using a ramdisk instead of the SSD to store temporary files will speed things up, but will cost you a few megabytes of RAM.

   sudo gedit /etc/fstab

Add this line to fstab to mount /tmp (temporary files) as tmpfs (temporary file system):
   tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0

Reboot for the changes to take effect. Running df, you should see a new line with /tmp mounted on tmpfs:
   tmpfs 513472 30320 483152 6% /tmp

   * Firefox puts its cache in your home partition. By moving this cache in RAM you can speed up Firefox and reduce disk writes. Complete the previous tweak to mount /tmp in RAM, and you can put the cache there as well.
   * Open about:config in Firefox. Right click in an open area and create a new string value called browser.cache.disk.parent_directory. Set the value to /tmp.
   * An I/O scheduler decides which applications get to write to the disk when. Because SSDs are so different than a spinning hard drive, not all I/O schedulers work well with SSDs.
The default I/O scheduler in Linux is cfq, completely fair queuing. cfq is works well on hard disks, but I’ve found it to cause problems on my Eee PC’s SSD. While writing a large file to disk, any other application which tries to write hang until the other write finishes.
The I/O scheduler can be changed on a per-drive basis without rebooting. Run this command to get the current scheduler for a disk and the alternative options:

   cat /sys/block/sda/queue/scheduler

You’ll probably have four options, the one in brackets is currently being used by the disk specified in the previous command:

   noop anticipatory deadline [cfq]

Two of these are better suited to SSD drives: noop and deadline. Using one of these in the same situation, the application will still hang but only for a few seconds instead of until the disk is free again. Not great, but much better than cfq.

Here’s how to change the I/O scheduler of a disk to deadline:
   echo deadline > /sys/block/sda/queue/scheduler

<note>**The above command needs to be run as root. If you use sudo, make sure you escape the > with a backslash (\), otherwise output of sudo command instead of echo will go to /sys/block/sda/queue/scheduler, and of course it won't be as the root user**.</note>

You can replace sda with the disk you want to change, and deadline with any of the available schedulers. This change is temporary and will be reset when you reboot.

If you’re using the deadline scheduler, there’s another option you can change for the SSD. This command is also temporary and also is a per-disk option:
   echo 1 > /sys/block/sda/queue/iosched/fifo_batch

You can apply the scheduler you want to all your drives by adding a boot parameter in GRUB. The menu.lst file is regenerated whenever the kernel is updated, which would wipe out your change. Instead of this way, I added commands to rc.local to do the same thing.


Open rc.local:
   sudo gedit /etc/rc.local

Put any lines you add before the exit 0. I added six lines for my Eee PC, three to change sda (small SSD), sdb (large SSD), and sdc (SD card) to deadline, and three to get the fifo_batch option on each:
   echo deadline > /sys/block/sda/queue/scheduler
   echo deadline > /sys/block/sdb/queue/scheduler
   echo deadline > /sys/block/sdc/queue/scheduler
   echo 1 > /sys/block/sda/queue/iosched/fifo_batch
   echo 1 > /sys/block/sdb/queue/iosched/fifo_batch
   echo 1 > /sys/block/sdc/queue/iosched/fifo_batch

Reboot to run the new rc.local file.

[update] Commenter dondad has pointed out that it’s possible to add boot parameters to menu.lst that won’t be wiped out by an upgrade. Open menu.lst (Remember to make a backup of this file before you edit it):
      sudo gedit /boot/grub/menu.lst

      The kopt line gives the default parameters to boot Linux with. Mine looks like this:
      # kopt=root=UUID=6722605f-677c-4d22-b9ea-e1fb0c7470ee ro

      Don’t uncomment this line. Just add any extra parameters you would like. To change the I/O scheduler, use the elevator option:
      elevator=deadline

      Append that to the end of the kopt line. Save and close menu.lst. Then you need to run update-grub to apply your change to the whole menu:
      sudo update-grub[end update] 

Want to know how fast your SSD or other storage device is? Using hdparm you can test the read performance of your disk:
sudo hdparm -t /dev/sda

The 4 GB SSD on my Eee PC 901 gets about 33 MB/s. My desktop PC’s hard drive gets about 78 MB/s. (What hdparm doesn’t show is that the seek time for an SSD is much, much lower than a hard disk.)

Have any other suggestions for SSDs, or disagree with any of these? Leave a comment to let me know.



===== Reducing the frequency of disk-writes =====

I found that my Linux-system was way to noisy for my bedroom: almost every 5 seconds it was writing to disk. I found 2 solutions:
  - use noflushd, but I didn't like that: it also spins down the harddisk which reduces the lifetime of your harddisk
  - fiddle around with mount-options and kernel-parameters

For the last option, I did 2 things: in /etc/fstab I added '**noatime**' to the options of the mounted partition:

   /dev/hda2       /       ext3    defaults,noatime  1 1

and I set some parameters which make disk-reads occur as soon as (reasonably) possible and disk-writes only when all buffers are at their limits:

   echo "90 500 0 0 600000 600000 95 20 0" > /proc/sys/vm/bdflush
   /sbin/elvtune /dev/hda -r 2048 -w 131072

<note warning>**Warning: don't use these settings if your PC is not reliable and crashes often or when power-outtages occur frequently!**</note>

===== SpamAssassin 'sa_learn'/updatedb/backups make system unresponsive =====

Not only can you renice the cpu priority, one can also renice the I/O priority. For this to work you need to use the 'cfq' I/O scheduler and the ionice utility.
Enabling the cfq I/O scheduler:

   echo cfq > /sys/block/hda/queue/scheduler

Then create the following script (/usr/local/bin/LP):

<code bash|f /usr/local/bin/LP>
#!/bin/sh
/bin/nice -n 40 /usr/bin/ionice -c 3 $*
</code>

Now when you want to run some disk-intensive process while not suffering too much, but 'LP ' in front of the command. E.g. 'LP updatedb'.





===== How to clear (and keep empty) the pagecache, dentries and inodes =====

<note tip>
As this is a non-destructive operation, and dirty objects are not freeable, the user **should run ''"sync"'' first in order to make sure all cached objects are freed**. If you are writing data at the time you run these commands, you’ll actually be dumping the data out of the filesystem cache before it reaches the disk, which could lead to very bad things. \\

//This tunable was added in 2.6.16.// </note>

   echo 3 > /proc/sys/vm/drop_caches

Afterwards, echo '0' to this file.

**Detailed explanation**

  echo # > /proc/sys/vm/drop_caches

| **#** | **description** |
| 1 | To free pagecache |
| 2 | To free dentries and inodes |
| 3 | To free pagecache, dentries and inodes |

===== Linux memory management =====

Memory overcommit can be configured via two parameters:

   loreto:/tmp # sysctl -a|grep overcommit
   vm.overcommit_ratio = 50
   vm.overcommit_memory = 0

From Red Hat manual:

    * **''overcommit_memory''** — Configures the conditions under which a large memory request is accepted or denied. The following three modes are available:
       * **0** — The kernel performs heuristic memory over commit handling by estimating the amount of memory available and failing requests that are blatantly invalid. Unfortunately, since memory is allocated using a heuristic rather than a precise algorithm, this setting can sometimes allow available memory on the system to be overloaded. This is the default setting.
       * **1** — The kernel performs no memory over commit handling. Under this setting, the potential for memory overload is increased, but so is performance for memory intensive tasks (such as those executed by some scientific software).
       * **2** — The kernel fails requests for memory that add up to all of swap plus the percent of physical RAM specified in ''/proc/sys/vm/overcommit_ratio''. This setting is best for those who desire less risk of memory overcommitment.

//**Note** This setting is only recommended for systems with swap areas larger than physical memory.//

    * **''overcommit_ratio''** — Specifies the percentage of physical RAM considered when ''/proc/sys/vm/overcommit_memory'' is set to 2. The default value is 50.


===== /proc/sys/vm/vfs_cache_pressure =====

   * Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects.
   * At the default value of vfs_cache_pressure = 100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to  agecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing  fs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes.
   * //In short, values less than 100 won’t reduce the caches very much as all. Values over 100 will signal to the kernel that you want to clear out the caches at a higher priority. I found that no matter what value you use, the kernel clears the caches at a slow rate. I’ve been using a value of 10000 on the server I talked about earlier in the article, and it has kept the caches down to a reasonable level.//





===== swappiness (/proc/sys/vm/swappiness) =====

   * **swappiness** is a parameter which sets the kernel's balance between reclaiming pages from the page cache and swapping process memory. //The default value is 60//.
   * If you want kernel to swap out more process memory and thus cache more file contents increase the value. Otherwise, if you would like kernel to swap less decrease it.
   * More info [[http://lwn.net/Articles/83588/|2.6 swapping behavior]]

Example:
   * ''swappiness=0'' tells the kernel to avoid swapping processes out of physical memory for as long as possible
   * ''swappiness=100'' tells the kernel to aggressively swap processes out of physical memory and move them to swap cache
   * A value of ''swappiness=10'' is recommended, but feel free to experiment.


===== Getting rid of that obsolete delay after password authentication =====

In Debian, add ''**nodelay**'' to the line with **pam_unix.so** in **/etc/pam.d/common-auth**.


===== Reducing disk-access (and thus saving power) =====

Mount filesystems with parameters:

    * noatime - do not update inode when a file got accessed or use relatime when it still must be updated somewhat
    * commit=300 - do not flush buffers to disk every 5 seconds (like ext3 does default) but every 5 minutes - only do this when your system never crashes and is on a UPS or a laptop-battery
    * async 

Enable laptop-mode with this command:

   echo 5 > /proc/sys/vm/laptop_mode

Expire dirty buffers only after a day (same warning applies):

   echo 864000 > /proc/sys/vm/dirty_expire_centisecs


**See also: [[http://www.lesswatts.org/tips/|Tips & Tricks - Saving power with linux]]**

===== Reduce disk I/O for small reads using memory =====

from [[http://rackerhacker.com/2008/08/07/reduce-disk-io-for-small-reads-using-memory/|Racker Hacker by major]]

Many applications that are used on a standard server perform quite a few of small writes to the disk (like MySQL or Apache). These writes can pile up and limit the performance of your applications. If you have kernel 2.6.9 or later, you can adjust how these small writes are handled to allow for better performance.

There’s two main kernel variables to know:

**vm.dirty_ratio** - The highest % of your memory that can be used to hold dirty data. If you set this to a low value, the kernel will flush small writes to the disk more often. Higher values allow the small writes to stack up in memory. They’ll go to the disk in bigger chunks.

**vm.dirty_background_ratio** - The lowest % of your memory where pdflush is told to stop when it is writing dirty data. You’ll want to keep this set as low as possible.

These might confuse you. In short, when your memory begins filling with little pieces of data that needs to be written to the disk, it will keep filling until it reaches the dirty_ratio. At that point, pdflush will start up, and it will write data until it reduces the dirty data to the value set by dirty_background_ratio.

Stock 2.6.9 kernels have a dirty_background_ratio of 10% and a dirty_ratio of 40%. Some distributions tweak these defaults to something different, so you may want to review the settings on your system. On a system with heavy disk I/O, you can increase the dirty_ratio and reduce the dirty_background_ratio. A little experimentation may be necessary to find the perfect setting for your server.

If you want to play with the variables, just use your standard echo:

   echo 5 > /proc/sys/vm/dirty_background_ratio
   echo 60 > /proc/sys/vm/dirty_ratio

Once you’ve found the right setting, you can set it permanently by adding lines to your /etc/sysctl.conf:

   vm.dirty_background_ratio = 5
   vm.dirty_ratio = 60

If you have a reliable server with a good RAID card and power supply, you could set the dirty_ratio to 100 and the dirty_background_ratio to 1. This was recommended by a buddy of mine who runs quite a few servers running virtual machines.

===== convert files from ISO to UTF-8 ====

<code> 
for x in `find . -name '*.htm'` ; do iconv -f ISO-8859-2 -t UTF-8 $x > "$x.utf8"; rm $x; mv "$x.utf8" $x; done
</code>


===== How do I force users to change their passwords upon first login? =====

1.) Firstly, lock the account to prevent the user from using the login until the change has been made:

   # usermod -L <username>

2.) Change the password expiration date to 0 to ensure the user changes the password during the next login:

   # chage -d 0 <username>

3.) To unlock the account after the change do the following:

   # usermod -U <username>

=== space report ===
for i in `ls -1 /home`;do du -sh /home/$i|mail $i -a "From: \"Admininistrator\" <root@aufbix.org>" -s "Poraba prostora `date +%d.%m.%y`";done\\