Deploying Jenkins Continuous Integration (CI) Server on Ubuntu LTS

For the past few weeks I have been helping Internet-in-a-Box(IIAB) team develop their Continuous Integration (CI) infrastructure setup using Vagrant, VirtualBox and Jenkins CI server.

Running Jenkins server from .war file might be convenient. But it is recommended to deploy GNU/Linux Continuous Integration (CI) server using these instructions.

# Add Debian package repository of Jenkins GPG key:
$ wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key | sudo apt-key add -

# Then add the following entry to your/etc/apt/sources.list:
$  echo deb http://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list

# Update your local package index:
$ sudo apt-get update

# Install Jenkis server:
$ sudo apt-get install jenkins

# Start Jenkins service and verify its status:
$ sudo systemctl start jenkins
$ sudo systemctl status jenkins

# Open http://localhost:8080 in your browser and copy paste in the secret password.
# Follow the on screen instructions to create Jenkins server admin account
$ sudo cat /var/lib/jenkins/secrets/initialAdminPassword

# Open the ports 8080 through your firewall:
$ sudo ufw allow 8080
$ sudo ufw status

Adding a monit service to monitor the Jenkins server is a good idea. Monit ensures that Jenkins server is restarted automatically when it crashes.




# Create a Jenkins monitoring file with following lines with:
$ sudo nano /etc/monit/conf.d/jenkins
check process jenkins with pidfile /var/run/jenkins/jenkins.pid
    start program = "/etc/init.d/jenkins start"
    stop program  = "/etc/init.d/jenkins stop"

Also you'll need to configure mail server to send Jenkins build reports.

Free SSL Certificate from Mozilla Let's Encrypt project

Last week Mozilla Let's Encrypt project announced the launch of its free, automated and open certificate authority. I had been waiting for this news for a long time. I quickly deployed Let's Encrypt on my staging server to learn how this technology works. The deployment process is painless and very straight forward. The certificates needs to renewed every 3 months, the Let's Encrypt client does this automatically. Thank you Jerome and Ryan for all your help!.

At the time of writing this blog post, Let's Encrypt client was not available for Ngnix server. So I am using the manual method to obtain the SSL certificate here. Please read the latest docs for additional information.

# Check out the let's encrypt source code
$ git clone https://github.com/letsencrypt/letsencrypt

# Stop the Nginx server, we need the client to bind to port 80.
$ sudo service nginx stop

# Start the let's encrypt client and follow the instructions on screen. You need to provide an email address.
$ sudo ./letsencrypt-auto --server https://acme-v01.api.letsencrypt.org/directory certonly --domains staging.example.org
IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/staging.example.org/fullchain.pem. Your
   cert will expire on 2016-01-02. To obtain a new version of the
   certificate in the future, simply run Let's Encrypt again.
 - If like Let's Encrypt, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

# Edit the Nginix config to point to generated certificates.
$ sudo nano /etc/nginx/sites-enabled/staging.example.org

 listen 443 ssl;
        server_name staging.example.org;
        ssl_certificate /etc/letsencrypt/live/staging.example.org/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/staging.example.org/privkey.pem;

# Restart the Nginx server
$ sudo service nginx start

Please don't forget to test your server using an comprehensive SSL server test such as SSLLabs.

Server Monitoring with Monit Tutorial

As you sit down for a lovely Christmas dinner with family and friends, your phone rings. The frantic client is on the line shouting that the server just went down. Don't you wish you had a helpful program that automatically monitors and restarts the servers automatically.

Monit is an open source process supervision program that monitoring your server processes and restarts failed services. It is easy to setup and relatively painless to use.

Here is a quick tutorial to configure Monit to monitor Apache and MySQL servers.

$ sudo apt-get install monit 

# Monit Configuration settings
$ sudo nano /etc/monit/monitrc 
  set daemon 60 
  set logfile /var/log/monit.log
  set idfile /var/lib/monit/id
  set statefile /var/lib/monit/state
  set eventqueue
  set httpd port 2812 and
   use address localhost
   allow localhost
   include /etc/monit/conf.d/*

# Create Apache2 monitoring file 
$ sudo nano /etc/monit/conf.d/apache2 
check process apache with pidfile /run/apache2.pid
    start program = "/etc/init.d/apache2 start" 
    stop program  = "/etc/init.d/apache2 stop"


# Create  MySQL monitoring file
$ sudo nano /etc/monit/conf.d/mysql 
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
    start program = "/etc/init.d/mysql start"
    stop program = "/etc/init.d/mysql stop"

# Check monit configuration for syntax errors 
$ sudo monit -t 


# Start Monit daemon
sudo service monit start 

# Check Monit status 
$ sudo monit status
The Monit daemon 5.3.2 uptime: 4m 

Process 'mysqld'
  status                            Running
  monitoring status                 Monitored
  pid                               2676
  parent pid                        1
  uptime                            4d 8h 1m 
  children                          0
  memory kilobytes                  256380
  memory kilobytes total            256380
  memory percent                    12.5%
  memory percent total              12.5%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  data collected                    Wed, 10 Dec 2014 08:43:13

Process 'apache'
  status                            Running
  monitoring status                 Monitored
  pid                               4230
  parent pid                        1
  uptime                            4d 8h 1m 
  children                          27
  memory kilobytes                  11556
  memory kilobytes total            391484
  memory percent                    0.5%
  memory percent total              19.1%
  cpu percent                       0.0%
  cpu percent total                 0.0%
  data collected                    Wed, 10 Dec 2014 08:43:13

System 'system_foobar.org'
  status                            Running
  monitoring status                 Monitored
  load average                      [0.00] [0.01] [0.05]
  cpu                               0.0%us 0.0%sy 0.0%wa
  memory usage                      666736 kB [32.6%]
  swap usage                        140 kB [0.0%]
  data collected                    Wed, 10 Dec 2014 08:43:1

That's all folks! Now you have a wonderful Christmas!

Managing Configuration files with etckeeper

Managing your /etc configuration files using version control is a good practice. Trust me, someday this will save your skin.

In the post I'll share setup notes for using etckeeper with git version control. You can install etckeeper and git using system package manager.

# Initialize etckeeper
sudo etckeeper init 

# Commit all your /etc/ configurations into git
sudo etckeeper commit '...enable etckeeper...'

# Check the commit history of file 
$ sudo git log /etc/passwd
commit 322b63ede6cf3073a8f48a883b49d5b3b60fdfb9
Author: arky <arky@localhost.localdomain>
Date:   Wed Mar 5 07:11:47 2014 +0000

    ...enable etckeeper...

There is very little etckeeper documentation out there. You can learn more from this dated Ubuntu Server 10.04 etckeeper wiki page.

Using Archivemail to pruning Mailman archives

The mailman server I maintain for a local non-profit is running out of disk space.Time for some spring cleaning. Decided to archive all mailing-list archives older than 2 years. Grabbed the nifty Archivemail python program and installed with 'python setup.py'.

The archivemail is a tool for archiving and compressing old email in mailboxes. It moves messages older than the specified number of days to a separate mbox format mailbox that is compressed with gzip.

Don't forget to back up mailman mbox files before you start pruning mailman archives.

# Prune mbox file
$ cd /var/lib/mailman/archives/private
$ archivemail --days=550 name-of-list/list.mbox 

# Wipe and rebuild archive
$ cd /usr/lib/mailman
$ bin/arch --wipe 

Postfix Log Entry Summarizer

Everyday I manage a Postfix mail server that handles emails from 20 mailing-lists. On days like this when there lot of email traffic, I keep an eye on the activity using pflogsumm perl script.

pflogsumm -e --problems_first -d today /var/log/maillog | pager

pflogsumm.pl is designed to provide an overview of postfix activity, with just enough detail to give the administrator a "heads up" for potential trouble spots.
http://jimsun.linxnet.com/postfix_contrib.html

Setting Timezone On Debian

Oops something went wrong with the last package upgrade on my Debian Desktop. This morning I observed that my system date had reverted back to UTC. Its not a big issue but let me take this opportunity to show how to change the timezone on your GNU/Debian Linux system.

The /usr/bin/tzselect tool from the libc6 package allows to set the proper timezone interactively. As I live in India (Asia) it is Indian Standard Time (IST) for me so I have to set TZ environmental variable to "Asia/Calcutta".

 $ /usr/bin/tzselect  Please identify a location so that time zone rules can be set correctly. Please select a continent or ocean.  1) Africa  2) Americas  3) Antarctica  4) Arctic Ocean  5) Asia  6) Atlantic Ocean  7) Australia  8) Europe  9) Indian Ocean 10) Pacific Ocean 11) none - I want to specify the time zone using the Posix TZ format. #? 5 Please select a country.  1) Afghanistan           18) Israel                35) Palestine  2) Armenia               19) Japan                 36) Philippines  3) Azerbaijan            20) Jordan                37) Qatar  4) Bahrain               21) Kazakhstan            38) Russia  5) Bangladesh            22) Korea (North)         39) Saudi Arabia  6) Bhutan                23) Korea (South)         40) Singapore  7) Brunei                24) Kuwait                41) Sri Lanka  8) Cambodia              25) Kyrgyzstan            42) Syria  9) China                 26) Laos                  43) Taiwan 10) Cyprus                27) Lebanon               44) Tajikistan 11) East Timor            28) Macau                 45) Thailand 12) Georgia               29) Malaysia              46) Turkmenistan 13) Hong Kong             30) Mongolia              47) United Arab Emirates 14) India                 31) Myanmar (Burma)       48) Uzbekistan 15) Indonesia             32) Nepal                 49) Vietnam 16) Iran                  33) Oman                  50) Yemen 17) Iraq                  34) Pakistan #? 14 The following information has been given:          India  Therefore TZ='Asia/Calcutta' will be used. Local time is now:      Tue Nov 27 02:31:35 IST 2007. Universal Time is now:  Mon Nov 26 21:01:35 UTC 2007. Is the above information OK? 1) Yes 2) No #? 1 You can make this change permanent for yourself by appending the line         TZ='Asia/Calcutta'; export TZ to the file '.profile' in your home directory; then log out and log in again.  Here is that TZ value again, this time on standard output so that you can use the /usr/bin/tzselect command in shell scripts: Asia/Calcutta  

After you set the TZ='Asia/Calcutta'; export TZ line in your /etc/profile or ~/.profile file. Type source ~/.profile and run 'date' to check if the changes has been reflected properly.

Load Balancing Apache Servers on Debian with HAProxy/Keepalived

A fine article shows you how to set up a two-node load balancer in an active/passive configuration with HAProxy and keepalived on Debian Etch.

Not only does the load balancer distribute the requests to the two backend Apache servers, it also checks the health of the backend servers. If one of them is down, all requests will automatically be redirected to the remaining backend server. In addition to that, the two load balancer nodes monitor each other using keepalived, and if the master fails, the slave becomes the master, which means the users will not notice any disruption of the service. HAProxy is session-aware, which means you can use it with any web application that makes use of sessions (such as forums, shopping carts, etc.)

From the HAProxy web site: "HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. Its mode of operation makes its integration into existing architectures very easy and riskless, while still offering the possibility not to expose fragile web servers to the Net." http://www.howtoforge.com/haproxy_loadbalancer_debian_etch

Booting GNU / Debian Linux in Verbose Mode

Nobody with a sane mind watch their computer boot up, its not very interesting to watch messages of kernel being loaded and services started flashing by.

If you read my last nights post on understanding the GNU / Debian Linux boot process then you have a fairly good idea of the importance of learning about the GNU / Debian Linux booting process. This knowledge will aid you in troubleshooting Linux boot problems.

The GNU/Debian Linux by default boots in a quiet mode with very few messages printing to the computer console (as you can see in the video). Let's make the computer be more verbose by turning on the VERBOSE variable in /etc/default/rcS file .

# Set VERBOSE to "no" if you would like a more quiet bootup. VERBOSE=yes

Save the file and reboot the machine. Watch and learn what goes on when the machine boots up. What next ? if you are done understanding how it works its time to hack the system find ways to speedup the boot process and try to free some system resources by getting rid of unused services and kernel modules. Happy Hacking !!!

Mounting a Windows partition automatically at boot time

If you have setup a Microsoft Windows and Linux dual boot system you may want to mount the windows partition automatically when you boot into GNU/Linux. As usual I don't suggest using any special scripts or packages for this. We will use the standard *nix files system table (fstab for short) file which stores all the entries about all known filesystems on the system and place an entry for our partition. Lets get started,Just place the following line into your /etc/fstab (filesystem table) file and you are done. /dev/hdaX /mnt/C vfat defaults,auto 0 0

• /dev/hdaX if you are using a IDE harddrive, X is the number of the partition (use fdisk -l to list your partition info)
• /mnt/C the mount point, where the windows partition is attached
• vfat - vfat for 98x or ntfs for NTFS partition
• auto - Mount the partition at boot time

Other possible parameters you can use uid/gid , showexec, umask etc.. the fstab manual page (man fstab) has detailed information about all the options. Also if you want your windows partition to check for errors with fsck.vfat at boot time check the last entry in the line from 0 0 to 0 1 .

Bittorrent complains unable to setrlimit not allowed to raise maximum limit

Ever wondered why Bittorrent complains that it is unable to setrlimit not allowed to raise maximum limit while starting. Most people would ignore this harmless warning but I just had to know why. First I posted a support request on Bittorrent Support site , I am still waiting for any response from them. Meanwhile here is the error message if you haven’t seen it before. $ bittorrent-console >>> unable to setrlimit not allowed to raise maximum limit In plain speak, its the python interpreters way of saying “look I need more system resources”. So,lets try to do just that. In gnu/Linux the Pluggable Authentication Modules (PAM) are used to place a cap on system resources. The /etc/security/limit.conf is the configuration file where the limits are, raise the limits and the warning will go away. Here is a example of the /etc/security/limits.conf configuration. * soft core 0 * hard rss 10000 @student hard nproc 20 @faculty soft nproc 20 @faculty hard nproc 50 ftp hard nproc 0 @student – maxlogins 4 More information on limits.conf configuration syntax is available here Happy Torrenting !!!!! Resources

• PAM page on wikipedia.org
• Linux-Pam Page
• Linux-Pam The System Administrators Guide
• Linux-Pam The Module Writers Guide
• Linux-Pam The Application Developers Guide