HOWTO: install graylog2 on CentOS 5 with RVM + Passenger

Graylog2 is an open-source self-hosted centralized log management tool. Think of it as a do-it-yourself version of loggly.com, or perhaps a simpler alternative to Splunk. Logs are stored in a MongoDB database.

I won’t go into too much detail, so if you want more info check out graylog2.org

Graylog2 consists of two components: graylog2-server, a Java process which receives logs and writes them to a MongoDB database. graylog2-web-interface, a Ruby on Rails app that seems happiest with modern versions of Ruby and Rails. CentOS 5, however, ships with an old version of Ruby (1.8.x) which did not play nicely with all of the gems that graylog2 wanted.

If the server is going to be dedicated to only running graylog2, we could probably overwrite the installed Ruby with a newer version. However, the server we wanted to install it on was already running a complex Rails app under mod_passenger which we did not want to risk breaking.

I decided to see if RVM – Ruby Version Manager – would allow me to setup an isolated Ruby environment just for graylog2 and not disturb the other Ruby apps on the machine. I also wanted to setup an isolated instance of Passenger-standalone for graylog2 then configure apache to listen on port 80 and forwarding requests with mod_proxy.

Everything worked even better than I expected. In the Ruby world, with so much rapid change, it can be challenging to put together a compatible set of gems for multiple apps running on the same machine. RVM worked so well I will consider using it to provide isolated ruby environments for other apps in the future. I love RVM!

Here is how I put it all together:

Overview:

  • Install graylog2-server in /opt/graylog2-server
  • Install graylog2-web-interface in /opt/graylog2-web-interface
  • graylog2-web-interface runs as unprivileged ‘graylog’ user
  • RVM installed to provide a Ruby 1.9.2 and Rails 3.x environment for graylog2-web-interface
  • Passenger-standalone installed in the Ruby 1.9.2 environment, running graylog2-web-interface on http://127.0.0.1:3001
  • Apache listens on tcp/80 and forwards to Passenger-standalone with mod_proxy

MongoDB install

  1. install mongodb from EPELyum repo
    # yum install mongodb mongodb-server
    # mkdir /var/lib/mongodb
    # chown -R mongodb:mongodb /var/lib/mongodb/
  2. enable mongod to start on boot
    # chkconfig mongod on
  3. start mongod (logfile: /var/log/mongodb/mongodb.log)
    # service mongod start
  4. run mongo, create an admin user in the admin collection and and a graylog user in a new graylog2 collection
    # mongo
         use admin
         db.addUser('admin', 'admin-mongo-passwd')
         db.auth('admin', 'admin-mongo-passwd')
         use graylog2
         db.addUser('grayloguser', 'grayloguser-mongo-passwd')

graylog2-server install

based on: https://github.com/Graylog2/graylog2-server/wiki/Installing

  1. download latest graylog2-server tarball: https://github.com/Graylog2/graylog2-server/downloads
  2. Install into /opt/graylog2-server:
    # cd /opt
    # tar xfvz ~/graylog2-server-0.9.5p1.tar.gz
    # ln -s /opt/graylog2-server-0.9.5p1 /opt/graylog2-server
  3. configure /etc/graylog2.conf
    # cp graylog2.conf.example /etc/graylog2.conf
     
    edit /etc/graylog2.conf, set monogdb_user, mongodb_password
  4. create /etc/init.d/graylog2-server
    #!/bin/sh
    #
    # graylog2-server:   graylog2 message collector
    #
    # chkconfig: - 98 02
    # description:  This daemon listens for syslog and GELF messages and stores them in mongodb
    #
     
    CMD=$1
    NOHUP=`which nohup`
    JAVA_HOME=/usr/java/latest
    JAVA_CMD=$JAVA_HOME/bin/java
     
    GRAYLOG2_SERVER_HOME=/opt/graylog2-server
     
    start() {
        echo "Starting graylog2-server ..."
        $NOHUP $JAVA_CMD -jar $GRAYLOG2_SERVER_HOME/graylog2-server.jar > /var/log/graylog2.log 2>&1 &
    }
     
    stop() {
            PID=`cat /tmp/graylog2.pid`
        echo "Stopping graylog2-server ($PID) ..."
            kill $PID
    }
     
    restart() {
        echo "Restarting graylog2-server ..."
            stop
            start
    }
     
    case "$CMD" in
        start)
            start
            ;;
        stop)
            stop
            ;;
        restart)
            restart
            ;;
        *)
            echo "Usage $0 {start|stop|restart}"
            RETVAL=1
    esac
  5. create /etc/logrotate.d/graylog2-server
    /var/log/graylog2.log {
           daily
           rotate 90
           copytruncate
           delaycompress
           compress
           notifempty
           missingok
    }
  6. register graylog2-server init script
    # chmod +x /etc/init.d/graylog2-server
    # chkconfig --add graylog2-server
    # chkconfig graylog2-server on
  7. start graylog2-server
    # service graylog2-server start
    # ps aux | grep graylog2
        root     23197  2.3  0.1 1196204 18428 pts/0   Sl   12:37   0:00 java -jar ../graylog2-server.jar

graylog2-web-interface install

based on: https://github.com/Graylog2/graylog2-web-interface/wiki

  1. Install RVM and prepare a new Ruby 1.9.2 environment
    # bash < <(curl -B http://rvm.beginrescueend.com/install/rvm)
    # rvm install 1.9.2
  2. create ‘graylog’ user
    # adduser -m graylog
  3. download latest graylog2-web-interface tarball: https://github.com/Graylog2/graylog2-web-interface/downloads
    # mkdir /opt
    # cd /opt
    # tar xfvz ~/graylog2-web-interface-0.9.5.tar.gz
    # ln -s /opt/graylog2-web-interface-0.9.5  /opt/graylog2-web-interface
    # chown -R graylog /opt/graylog2-web-interface-0.9.5
  4. Install gems necessary to run graylog2 using Bundler in the ruby 1.9.2 environment
    # rvm use 1.9.2
    # cd /opt/graylog2-web-interface
    # gem install bundler
    # bundle install
  5. Configure graylog2-web-interface
    • Edit the various config files in /opt/graylog2-web-interface to suit your environment:
      • email.yml
      • general.yml
      • mongoid.yml (make sure to use the user/pass you setup earlier)
  6. Start graylog2-web-interface, create first user
    # rvm use 1.9.2
    # cd /opt/graylog2-web-interface
    # RAILS_ENV=production script/rails server

    Connect to http://127.0.0.1:3000. If everything is working, graylog2 will ask you to create the first user. Shutdown graylog2 (ctrl-C) after you create the first user.

  7. Install and configure passenger-standalone
    # yum -y install curl-devel
    # rvm use 1.9.2
    # gem install passenger
    # gem install file-tail
     
    # cd /opt/graylog2-web-interface
    # mkdir tmp log
    # chmod -R 777 tmp log
    # passenger start

    Passenger should download, compile, and build everything it needs. If passenger tries to serve requests, ctrl-C to quit.

  8. Create /etc/init.d/graylog2-web-interface
    #!/bin/bash
    #
    # graylog2-web-interface:   graylog2 web interface
    #
    # chkconfig: - 98 02
    # description:  Starts graylog2-web-interface using passenger-standalone. \
    #       Uses RVM to use switch to a specific ruby version.
    #
     
    # config
    USER=graylog
    APP_DIR=/opt/graylog2-web-interface
    RVM_RUBY=1.9.2
    ADDR=127.0.0.1
    PORT=3000
    ENVIRONMENT=production
    LOG_FILE=/var/log/graylog2-web-interface.log
     
    # --
     
    CMD_START="cd $APP_DIR; rvm use $RVM_RUBY; passenger start -d \
                        -a $ADDR \
                        -p $PORT \
                        -e $ENVIRONMENT \
                        --user $USER"
    CMD_STOP="cd $APP_DIR; rvm use $RVM_RUBY; passenger stop -p $PORT"
     
    CMD_STATUS="cd $APP_DIR; rvm use $RVM_RUBY; passenger status -p $PORT"
     
    . /lib/lsb/init-functions
    case "$1" in
      start)
        echo "Starting graylog2-web-interface"
        su - $USER -c "$CMD_START"
        ;;
      stop)
        echo "Stopping graylog2-web-interface"
        su - $USER -c "$CMD_STOP"
        ;;
      status)
       su - $USER -c "$CMD_STATUS"
       ;;
      *)
        echo "Usage: $0 start|stop|status" &gt;&amp;2
        exit 3
        ;;
    esac
  9. Create /etc/logrotate.d/graylog2-web-interface
    /opt/graylog2-web-interface/log/*log
           size=256M
           rotate 90
           copytruncate
           delaycompress
           compress
           notifempty
           missingok
    }
  10. Register graylog2-web-interface service
    # chmod +x /etc/init.d/graylog2-web-interface
    # chkconfig --add graylog2-web-interface
    # chkconfig graylog2-web-interface on
  11. Start graylog2-web-interface
    # service graylog2-web-interface start
  12. Configure apache to forward requests to passenger
      1. Create /etc/httpd/conf.d/graylog2.conf
    NameVirtualHost *:80
     
    	ServerName log.mydomain.com
    	ServerAlias graylog2.mydomain.com
    	ProxyPreserveHost On
    	ProxyPass        / http://127.0.0.1:3000/
    	ProxyPassReverse / http://127.0.0.1:3000/
     
            CustomLog /var/log/httpd/log.mydomain.com-access_log common
    1. Restart apache
      # service httpd configtest
      # service httpd restart

Startup scripts for running multiple Passenger-standalone instances:

If you need a method for setting up multiple Passenger-standalone Rails apps, checkout these scripts. They’re very well done: http://moeffju.net/blog/passenger-standalone-initscript

Alternatives to Apache

rippleAdder posted example configs on the graylog2 mailing list for running graylog2-web-interface with other webservers than Apache, namely unicorn and nginx. These are likely going to be faster than apache, although I haven’t benchmarked any webservers with graylog2 yet. You can find his sample configs here

Posted Wednesday, April 13th, 2011 under centos, howto, linux, software.
  • tom

    These instructions are very handy, but bundler does not install on CentOS as the gems from EPEL is too old

    ERROR: Error installing bundler:
    bundler requires RubyGems version >= 1.3.6

    EPEL is 1.3.1

    I am looking into a way round this.

    • jmiller

      I also ran into this issue and that is why I choose to use RVM to install a parallel ruby-1.9.2 environment.

  • Steve

    Can’t figure out how to get apache to forward requests to passenger.
    Can you give an example of your : /etc/httpd/conf.d/azdf-graylog2.conf

  • jmiller

    Steve – I added an example /etc/httpd/conf.d/graylog2.conf file to the article. Thanks for pointing out the omission, I had intended to include this in the original post.

  • Steve

    Thanks, that did it.
    Could you provide an example of how a remote server syslog.conf should be configured to send to port 514 on the graylog server?
    I tried creating /etc/graylog2.d/rules/graylog2.drl rules file to test, did a
    # nc -w0 -u 192.168.1.10 514 <<< "TEST from remote to graylog"
    then get "Could not handle GELF client: Missing GELF message parameters. version, short_message and host are required."
    My graylog2.conf has "use_gelf=true" and "gelf_listen_port=12201"

  • Steve

    Never mind… Got remote servers sending syslog messages fine now.

  • http://twitter.com/rasputnik Dick Davies

    Feel your pain on the CentOS ruby front; I cheat and run the UI as a JRuby war file on tomcat, believe it or not it’s much simpler :)

    Curious though; if you’re going down the native Ruby route, why run passenger out of process (i.e. not mod_passenger in apache itself)?

    • jmiller

      In this case, I already had mod_passenger running another rails app on this server and the gems were not compatible between it and graylog2, thus I ran graylog2 in a standalone/out-of-process passenger+nginx instance. In the future I will likely use a simpler rack webserver like thin, unicorn, or rainbows! instead, because I find the whole passenger setup more trouble than it’s worth.

  • http://arnie.com arnie

    Jmiller, Steve, how do you start logging syslog from remote servers into graylog2?

    Sorry if its obvius I cant seem to find how to do it.

    Thanks in advance.

  • miguel_

    Hi
    great article. I’ve one question in the script number one (graylog-server) in the start function you placed the commads to start but i see strange characters

    [...]
    $NOHUP $JAVA_CMD -jar $GRAYLOG2_SERVER_HOME/graylog2-server.jar > /var/log/graylog2.log 2>&1 &
    [...]

    Is possible that post the correct sentence.

    thanks

  • Esente

    @Steve:

    Can you explain how you did that?

  • Ron

    I set up graylog2 according to these instructions. Thanks by the way, everything worked perfectly on my fedora 15 box. Now, however, I can only reach the web interface if my web browser is running on the same machine. (I’ve double-checked and my firewall is not in the way, it’s completely disabled). I’ve been reading through the passenger & nginx docs (never used either of these before) but I can’t figure out why I can’t access the web interface from other machines. Thoughts?

  • Ron

    Nevermind, I re-read the instructions, and figured out that I needed to front it with apache. Imagine, reading the instructions. ;-)

  • Steve

    There is an error in the /etc/init.d/graylog2-server script.
    Just replace the line 18 with that:

    $NOHUP $JAVA_CMD -jar $GRAYLOG2_SERVER_HOME/graylog2-server.jar > /var/log/graylog2.log 2>&1 &