Just another open source experience

I spend life time in the OpenNMS project and I love to work in free software and the workflows around it. We moved with our project from SourceForge to GitHub a few years ago and I think it was the right decision. There are now some established workflows in this ecosystem and they tear down borders between different open source projects and here is an example of it.

I develop with IntelliJ IDEA and spend currently some time working in documentation of OpenNMS. We have migrated from Docbook XML format to AsciiDoc and started with the help of a few brave community volunteers a new documentation environment. I’ve found a plugin for IDEA which renders AsciiDoc and gave it a try to have a better workflow working on documentation and navigating through source code in just one program.

Unfortunately I run in an error message. The project is hosted on GitHub and I’ve opened an issue with the version and the behavior. They responded my question just a few hours later.

They where really helpful and asked how it was installed to avoid basic mistakes. The issue still existed and they asked me if its possible to provide the .adoc files to reproduce the error. This was quite easy, the documentation I work with is also public on GitHub, so I just sent them a link to the files. The guys figured out, the problem occurred whenever I use the source code formatting from AsciiDoc, so it was easy to create a small test file to reproduce the issue. Beside the fact finding the reason of the problem, they created also a new release of the plugin and sent me a link in the public comments and asked if I can verify the solution. I’ve installed the new plugin and ran unfortunately in a different error, I’ve provided a screenshot and the detailed error message.

Just a few hours later they figured out, there was a reference in AsciidoctorJ a dependency of the IDEA plugin which pointed to a already fixed issue in JRuby dealing with white spaces in a file path. They updated to the latest version of JRuby and provided another version of the IDEA plugin just a few hours later which fixed my problem.

Please let imagine the procedure with a similar issue in a closed source software where the issue is in a dependency of a dependency.
This issue was fixed in 5 days, they provided two releases of the plugin and they responded to my comments in maximum a day!
I paid nothing – just the time providing information of my issue – which I had anyways even I had to pay for the software and the support. There was not one moment blaming, finger pointing and complete transparency about three different open source projects to identify and fix the problem.

Thank you for this open source experience

– or my friend Q would sing: ♪ Everything is awesome … ♪

Flattr the authorShare on Google+Tweet about this on TwitterShare on RedditDigg thisShare on StumbleUponShare on Tumblr

Investigate file descriptor issues running OpenNMS

If you run a centralized monitoring system in large environment you can run in some issues regarding file descriptor limits. Linux gives you very detailed information in the kernel control and information center in /proc. The soft and hard limits have effect for file and network sockets, which can end up in a too many files open exception in OpenNMS.

The default values for soft and hard limits can be checked with

ulimit -a
ulimit -a -H

The value is per user and each new process inherits these limits. OpenNMS changes the hard limit during the start with the default init script and changes with

ulimit -n 20480

from normally 4096 to 20480. You can change this value by adding the following line to your /etc/opennms/opennms.conf

MAXIMUM_FILE_DESCRIPTORS=40960

If you start OpenNMS you can see the limits for the OpenNMS JVM with

cat /proc/$(cat /var/run/opennms.pid)/limits

You can see how much file descriptors OpenNMS has allocated with:

ls -l /proc/$(cat /var/run/opennms.pid)/fd | wc -l

If you use lsof with the process id of OpenNMS you will see a larger number than in /proc/pid/fd

lsof -p $(cat /var/run/opennms.pid) | wc -l

The reason is memory mapped .so files are listed and don’t count for the configured limits and are listed with lsof.

lsof | grep $(cat /var/run/opennms.pid) | wc -l

If you want to see how many filesystem handles OpenNMS uses, you can run:

cat /proc/sys/fs/file-nr
4128	0	262144

you can see three values:

number of allocated file handles: 4128
number of used file handles:      0
maximum number of file handles:   262144
Flattr the authorShare on Google+Tweet about this on TwitterShare on RedditDigg thisShare on StumbleUponShare on Tumblr

[Release] – OpenNMS PRIS 1.1.0

We have release OpenNMS Provisioning Integration Server 1.1.0. Cause we had quite a big change in the code structure. We have now a clearer abstracted API for new sources and every data source we can integrate with is now a separate maven module. It makes it much easier dealing with technology specific dependencies. The documentation is now completely moved to AsciiDoc. This documentation is now also deployed in the archive, so if you have PRIS running in HTTP mode you can have access to the docs locally by going to http://your-ip:8000/documentation. We have introduced a new file and folder structure for requisition configuration and scripts, so please have a look at the migration instructions in the release notes.

The release of the project is done with Github and the OpenNMS issue tracker. Here can you find information about it:

gl&hf

Flattr the authorShare on Google+Tweet about this on TwitterShare on RedditDigg thisShare on StumbleUponShare on Tumblr

Something about OpenNMS and documentation

To help the OpenNMS project I spent a year together with Alexander Finger and Klaus Thielking-Riechert writing an OpenNMS book. We started with a development version of for 1.8 and tried to find a mixture between consistent slow changing and new concepts in 1.8. It was kind a complicated thing and from my point of view not perfectly done. Nevertheless I’ve thought about something like: “It’s about time writing a second edition covering the topics in coming 1.14?” The problem I’ve with books, it doesn’t allow contribution and hasn’t a really long life cycle. So I’ve decided instead of spending a year again writing a book, I want something which is more helpful to the project itself.

During my visits on different conferences like FroSCON, Linuxtage in Chemnitz and LinuxTag in Berlin, I tried to get in talks and met people working in the community area. Two things I’ve quickly recognized, one was the Open Stack project, which did a really cool job communicating about how to contribute in their project. The second one was a talk from Mikey Ariel a technical writer and documentarian at Red Hat, Inc. on LinuxTag in Berlin.

To get some feedback, I started an open discussion about documentation in the OpenNMS project with attendees at OUCE 2014 in Southampton. I was very happy and we had a very constructive conversation and people seems interested to help the project here. The status quo of our documentation is currently MediaWiki based collection of unstructured sometimes category tagged pages. On one side it makes it easy for people creating content and there is quite a lot of good stuff in the Wiki, but on the other side it is hard to find things. You have always exactly to know what term you have to search for and makes it hard to get an idea what OpenNMS is capable of. To address this issue I decided to spend the week at DevJam on OpenNMS documentation and my friend Markus von Rüden helped me a lot. He is a developer from hell and helped me a lot evaluating technologies and gave important input how to establish a process in an agile environment.

A few weeks later I got in contact with a guy worked at Juniper with OpenNMS and he asked to help us with this topic. I was amazed and used his positive feedback as trigger to get a documentation meeting set up. We used the opennms-discuss and move now this topic to opennms-docs mailing list. I’ve suggested a Google Hangout where we could explain a little about the experience we had what we have figured out during DevJam.

We have basically divided the documentation in two parts:

  • Wiki based documentation for tutorials and integration of OpenNMS with external devices, applications etc.
  • Source code based documentation which gives a complete reference how to configure things which is tightly based on a version of OpenNMS

For each of this components we created a Jira component to track issues against:

We started with a simple doable task like, document all existing monitors in OpenNMS with default, values and possible configuration parameters. It is our first little project we use to establish and model the process and also building the “How to contribute to documentation guide” which is currently a work in progress. The guys helping at the docs suggested to have a bi-weekley meeting and so I’ll drop a doodle at the opennms-docs mailing list to get a next appointment set up. If you are interested in join, the opennms-docs is a low traffic mailing list and this is the place to be in the loop if you are interested on this topic.

Thanks for reading and I’m excited about the help you guys who want to help

Have a nice day.

Flattr the authorShare on Google+Tweet about this on TwitterShare on RedditDigg thisShare on StumbleUponShare on Tumblr

!(I’m a DevOps-Guy)

 I started in 1998 in IT-Services and I did all the funny stuff – was involved building new networks for companies, migrated software from commercial vendors from version A to B to C to D and I have wasted too much lifetime on broken RAID 5s, backups and restores.

If you want to learn how to operate a computer network – this is a good place. You will be always called for the complicated problems nobody can solve with a simple Google search. You get also a good feeling, which solution is maintainable over a longer period of time. Fortunately people developing software experienced enough pain and started thinking about how to make code maintainable. Good developers realized years ago, people spend more time maintaining software than writing it. In my point of view an underestimated acceptance criteria in an “Agile” process ;) Since I spend more and more time in the OpenNMS project, I get much better an idea what the meaning about testing, documenting, Continuous Integration and Test Driven programming is. What people realize is a nameless magical connectivity between the guys who develop Things™ and the ones who bring them to production and the poor souls which have to keep it productive. In the late 90s there wasn’t a name for it. To make a good job Ops-Guys and the Dev-Guys have to work together – which wasn’t/isn’t a default behavior by nature.

With the vision of “Software Defined ${thing}” and tools like Docker, Vagrant, Chef and Puppet, Operators started using similar tools like developers and the term DevOps popped up.

Just the reason you treat your infrastructure like code and you commit your bash-Script or Chef cookbook with IntelliJ IDEA to a git repository don’t make your OpsJob automatically a Dev Job ;) Long story short: In my opinion and experience with operators and developers, DevOps isn’t a job title – it is a culture in your company where your developers and operators establish processes and work together.

Flattr the authorShare on Google+Tweet about this on TwitterShare on RedditDigg thisShare on StumbleUponShare on Tumblr

[Release] – OpenNMS emergency security release

m4s0n501

This is an emergency security release which fixes a bug that could expose arbitrary filesystem data to logged-in users through the Web UI. Upgrading is strongly recommended.

For details on the issue, see: NMS-6580

Also, a reminder: as of OpenNMS 1.12.7 and 1.13.2, the way OpenNMS starts Jetty has changed, so if you have configured OpenNMS to use AJP, or used other esoteric tweaks to the Jetty configuration in opennms.properties, you will need to adapt them to jetty.xml instead. For details, see: Whats New in OpenNMS 1.12

Thanks

Flattr the authorShare on Google+Tweet about this on TwitterShare on RedditDigg thisShare on StumbleUponShare on Tumblr

OpenNMS Dev-Jam 2014

We are right in the middle of our 5th season of the year – DEVJAM. It is always a lot of fun spending one week with people from all over the world on the campus of the University of Minnesota. I’ve put on my personal agenda for this week, spending time with the folks working on the way how we want to document our project. I try also to gather the project ideas and where they contribute the code to the project. Currently i try to get some updates from the progress through the opennms.eu page. We have the following topics on the list:

You find the current work of progress in the Github public code repository and the wiki pages get updated. If you want more direct contact. Please feel free to join the IRC channel on irc.freenode.net and join #opennms.

Happy hacking …

Flattr the authorShare on Google+Tweet about this on TwitterShare on RedditDigg thisShare on StumbleUponShare on Tumblr

[Release] – OpenNMS 1.12.7

Welcome to a new OpenNMS stable release with code name Bob Marley aka 1.12.7 in our repositories. The release fixes various bugs and has a few nice enhancements. It addresses mostly bugs displaying stuff in the WebUI if you have large sets of information. A null pointer exception bug for the ones how forward OpenNMS Events via Syslog to external systems using the Syslog Northbound is fixed.

Eventd and Vacuumd fixed a bug ignoring the reloadDaemonConfig event. An issue with large provisioning groups and WebUI crashes is fixed and you can create now Notifications with more than 4000 characters. A security cross site scripting bug in the WebUI is fixed. For the ones who run OpenNMS on Debian based systems, the dependencies to postgres are fixed to version 9.3.

The full list of bugs and enhancements can be found in the Relase Notes.

Happy updating …

Flattr the authorShare on Google+Tweet about this on TwitterShare on RedditDigg thisShare on StumbleUponShare on Tumblr

OUCE 2014 – Review

Ulf 2.0 and David

Ulf 2.0 and David

We have finished our 6th OpenNMS User Conference Europe. It was the first time we made a conference outside of Germany. We started with the first conference in 2009 when I worked at NETHINKS and begun with OpenNMS professional services. For the 3rd and 4th conference we decided to move the venue closer to the people organizing the conference and the IT center in Fulda was a really good place. In 2013 NETHINKS and the OpenNMS Foundation Europe worked together to move the organization of the conference to the community at the University of Applied Science in Fulda.

In 2014 we had our first conference out of Germany at the University of Southampton – United Kingdom, Great Britain, England, the Empire … whatever. We had attendees from 12 countries. The OpenNMS Group, Inc. showed the latest development sponsored to the project by commercial companies like the new Semantic Topology User Interface and the Operators Panel.

OpenNMS and OCS Inventory

OpenNMS and OCS Inventory

Additionally had a visit from three nice guys from OCS Inventory with an talk about OCS Inventory and practical workshop.

The crowd around OpenNMS is really amazing and we finished with an open discussion about the OpenNMS project from a community perspective. We came also around documentation where have really some issues about. So I decided to give a short overview how other projects handle documentation. I’ve spent the last conferences with sitting in “How to contribute” talks from OpenStack and ElasticSearch and they do really a great job when it comes around the community.

Sorry if you missed this great opportunity having the possibility talking with a lot of great people. We got all slides and they are available in our conference system. As soon we received the videos we will upload them to our OpenNMS Youtube Channel.

Thanks to all attendees for the great time

Flattr the authorShare on Google+Tweet about this on TwitterShare on RedditDigg thisShare on StumbleUponShare on Tumblr