groovy-statsdclient

Posted June 29, 2011, under groovy, monitoring, software
No comments yet

In an attempt to learn some Groovy and Gradle I wrote an implementation of a Statsd client in groovy.  It’s similar to other statsd clients in other languages and supports the typical increment(), decrement(), and timing() methods.

It’s not available on Maven Central or via Grape at this time, which will be a future learning exercise. In the meantime you can download a pre-built .jar or the source code from github:

For more background on Statsd, check out this blog article from Etsy:  Measure Anything, Measure Everything

 

Leave the first comment ▶

#monitoringsucks – but it doesn’t have to

Posted June 6, 2011, under devOps, monitoring, software, testing
No comments yet

On May 26, 2011, John Vincent, aka @lusis, started a conversation with a simple tweet that led to an hour-plus chat on IRC freenode between some very smart folks.  The IRC channel was supposed to go away after the chat was over but instead it has survived and is attracting more and more attention.

The genesis of it all is the general staleness of the open source monitoring landscape.  There have been numerous and rapid improvements in operations tools in the last 10 years but when it comes to monitoring we’re still pretty much stuck on the same tools we’ve been using since the late-90s.  It would be understandable if these tools were “as good as it gets”, but they’re not, and there’s definitely room for improvement and sharing of ideas and experiences.  If nothing else, at least there’s a central place for people to discuss monitoring and share ideas.  There’s hardly a day that goes by where someone doesn’t stop into ##monitoringsucks on IRC and announce a new tool or concept they’re hacking away on.

Since the initial chat, a set of github.com repos has popped up to collect links to various tools and blog posts on the topic of monitoring:

  • irc-logs – IRC log of the initial chat that started it all on 5/26/2011 is available here
  • tools-repo – a great resource for many monitoring tools
  • blog-posts – a growing list of interesting monitoring related blog posts

If you’re interested in helping make the art of monitoring better, consider stopping by IRC Freenode or contributing to the github repos.  I’m hoping to learn quite a bit out of this little spark that John created.

Leave the first comment ▶

Alternative skins for Skype 5 on Mac OSX

Posted May 24, 2011, under macosx, software
No comments yet

The consensus at this point is that Skype 5 on Mac OSX is ugly and wastes a ton of screen space for no good reason (all of this is true, btw.) But, you don’t have to live with the ugliness.  It’s not widely known, but Skype 5 supports skins or “chatstyles”, which are constructed entirely in HTML/CSS/Javascript, so they’re easy to write and hack.

Skype’s official page for information on ChatStyles:  http://macthemes.skype.com/start

Here are a few of the alternate ChatStyles available on github:

  • PanamericanaMini – this is probably the first alternate ChatStyle.  It’s very similar to the default Panamericana theme, but with some whitespace removed.
  • Brief – nice theme.  Check out the screenshots.
  • My fork of Brief on github – I like the Brief chatstyle, but I made some small modifications to further reduce extra whitespace and I modified colors slightly.  Make sure you switch to the ‘personal-changes’ branch.
  • Simples – heavily influenced by the Renkoo style for Adium.   Screenshot.
  • StyleShift – reduced whitespace, every image URL is expanded in a linked thumbnail, custom emoticons.

More ChatStyles are available in Skype’s ChatStyle Gallery.  They’re also running a competition, so you can go vote for your favorite.

Leave the first comment ▶

Charlie Sheen plugin for the Hudson/Jenkins CI Server

Posted April 26, 2011, under software, testing
No comments yet

I built a Charlie Sheen “Persona” plugin for our Hudson continuous integration server at work, which is now available on my github page.  If you’re familiar with the Chuck Norris plugin, then you know what this is all about.

Add a little bit of Sheen-ius to your Hudson/Jenkins server!!  Tiger blood is included.  #winning

Installation instructions are in the README.

Leave the first comment ▶

Kanbanops – new mailing list

Posted April 24, 2011, under devOps, kanban
No comments yet

I am a fan of using Kanban boards to organize and visualize the work coming into an infrastructure team, but most of the Kanban resources on the internet are geared towards software development teams.

Recently, a new mailing list was started for the sole purpose of sharing Kanban experiences in operations teams. The list is appropriately named kanbanops and you can sign up, browse the archives, etc, over at yahoo groups:

Leave the first comment ▶

Collectd-Graphite plugin. Bringing together two great tools

Posted April 14, 2011, under software
No comments yet

Collectd is a powerful tool for gathering metrics using its wide range of plugins, such as cpu, disk, load, memory, etc.  But there is a lack of good frontend tools for visualizing the data collectd produces.

Graphite is an amazingly powerful tool from Orbitz for visualizing metrics, but there is a lack of tools for gathering host-level stats and sending into graphite.

It would be great if we could leverage the strengths of both tools.

So, I wrote a plugin for collectd that will send data into a graphite instance – collectd-graphite. This plugin runs inside the collectd process using the collectd-perl interface. This makes it different than Jordan Sissel’s collectd-to-graphite tool which runs in a separate process using node.js. The plugin-based approach reduces the number of moving parts we need to worry about (ie: less stuff to monitor, fail, restart, etc.) Jordan’s tool is good, btw, but I wanted something different.

Grab the plugin on github:  https://github.com/joemiller/collectd-graphite

Continue Reading ▶

HOWTO: install graylog2 on CentOS 5 with RVM + Passenger

Posted April 13, 2011, under centos, howto, linux, software
15 comments

Graylog2 is an open-source self-hosted centralized log management tool. Think of it as a do-it-yourself version of loggly.com, or perhaps a simpler alternative to Splunk. Logs are stored in a MongoDB database.

I won’t go into too much detail, so if you want more info check out graylog2.org

Graylog2 consists of two components: graylog2-server, a Java process which receives logs and writes them to a MongoDB database. graylog2-web-interface, a Ruby on Rails app that seems happiest with modern versions of Ruby and Rails. CentOS 5, however, ships with an old version of Ruby (1.8.x) which did not play nicely with all of the gems that graylog2 wanted.

If the server is going to be dedicated to only running graylog2, we could probably overwrite the installed Ruby with a newer version. However, the server we wanted to install it on was already running a complex Rails app under mod_passenger which we did not want to risk breaking.

I decided to see if RVM – Ruby Version Manager – would allow me to setup an isolated Ruby environment just for graylog2 and not disturb the other Ruby apps on the machine. I also wanted to setup an isolated instance of Passenger-standalone for graylog2 then configure apache to listen on port 80 and forwarding requests with mod_proxy.

Everything worked even better than I expected. In the Ruby world, with so much rapid change, it can be challenging to put together a compatible set of gems for multiple apps running on the same machine. RVM worked so well I will consider using it to provide isolated ruby environments for other apps in the future. I love RVM!

Here is how I put it all together:

Continue Reading ▶

Client support for Amazon S3 multipart uploads (files >5GB)

Posted February 18, 2011, under AWS, internet services, macosx, software
4 comments

I recently had a need to upload a few large (11GB) files to S3 for offsite archive purposes. Amazon announced support for objects >5GB back in November 2010, so I had assumed that most S3 clients and libraries had support for it already.  I was wrong.  It turns out that in order to support large files, you need to use the new “multipart upload” feature.  What I found was that many S3 clients do not yet support multipart uploads.

In case this helps save anyone else a few hours of time, here is a summary of what I found regarding multipart upload support in the various S3 clients:

As of Feb 18, 2011:

To solve my immediate need I used S3 Browser on Windows 7.  It was the only free client I found that supported multipart uploads.  I mostly use a Mac, however, so if someone knows of a free S3 client for Mac please let me know.

I did not look at which S3 libraries support multipart uploads yet.  I have heard that my favorite python library – boto – supports it already.  And I have seen some reports that RightScale’s right_aws ruby library may support it on some platforms.  I’m not sure about the others.  Post in the comments if you want to confirm support for multipart uploads in your favorite S3 client or library.

Update (3/10/2010): Cyberduck 4.x was released this week and includes support for multipart uploads and 5TB uploads.  A Windows version is also available now.  Pretty cool, good job Cyberduck.

Update (4/18/2012): s3cmd 1.1.0-beta2 is available and supports multipart uploads along with many other new S3 and Cloudfront features. Grab the beta or wait for 1.1.0 final to be released

4 comments so far, add yours ▶

Amazon Cloudfront Private Streaming CLI Tools

Posted January 31, 2011, under AWS, internet services, software
No comments yet

Back in late 2010 I had a need to create a private Cloudfront streaming distribution but there was no simple way to do this. Amazon’s web-based AWS Management Console did not support this and I could not find any simple CLI tools for doing this.

Luckily, RightScale’s right_aws ruby gem (>= 2.0.0) provides support for the Cloudfront private streaming API.

These tools should make it simple for any linux admin or developer to get started with creating a private streaming distribution on Amazon.

The code and detailed documentation is available on github:
https://github.com/joemiller/aws-cf-private-streaming-tools

To read more, and see example usage, click past the break!
Continue Reading ▶

dns_compare.py – testing DNS servers

Posted December 22, 2010, under dns, testing
No comments yet

Sometimes it might be nice to be able to test a DNS server’s output, such as with a continuous monitoring system, or as a validation tool when migrating to a different DNS server (or service.)

$ dns_compare.py -z example.com --file example.com.zone --server 10.1.1.1
....X...X..done
Results:
Matches:      9
Mis-matches:  2

To see more examples or download the code, goto: https://github.com/joemiller/dns_compare

Leave the first comment ▶
◀ Older Entries
Newer Entries ▶