Not dead yet

I know this is taking a long time, but here’s what I’m up to with Weathermap now:

  • Cacti 1.0 uses a sort-of old-fashioned “download chunks of HTML” approach to its new UI. That meant rewriting the Weathermap Cacti plugin to work in this (undocumented) style, too.
  • I also have wanted to rewrite it anyway for some time into a more object-oriented format which separates the actual actions on the database from the display code. This makes it possible to write automated tests for the action code, and for someone to potentially reuse it a lot more easily in another NMS integration.
  • I was halfway through doing all that, when I realised that if I was going to have to rewrite it anyway, I’d rather do it in a manner than benefits everyone, including non-Cacti users. Particularly as I am personally not really much of a Cacti user nowadays, but also I’ve wanted to make it easier for folks like the LibreNMS project to integrate weathermap for some time now.
  • So: new plan. The user-interface parts of the Cacti plugin will be entirely javascript client-side code. The remaining plugin parts will just form an API that the client-side calls. This means I can ignore the Cacti AJAXy redesign stuff, largely. I’ll also be trying to ensure that the client-side doesn’t need Cacti-specific data. At that point, someone from $OTHER_NMS team can embed the client javascript as-is, and implement a simple JSON API, and they too have integrated Weathermap, including map management. To prove this works, I’m planning to make a standalone web app too, providing a web-based, database-backed manager for people using non-integrated NMSes.
  • The ‘user’ plugin is about 70% done (map cycling remains). The ‘management’ plugin is started but early days. This work is also a lot more fun than trying to untangle someone else’s intent though, so it’s progressing well. The client side of things will be written with Inferno and Redux.

Once again, sorry for the long delay, but I wanted this to be right. I also wanted it to be fun to do – this is my hobby project, without much external contribution[*], so having it feel like a job I hate doing for the last year has been no fun at all. The end result should be worth it.

Also, this is the path the data-picker, and then editor will take in a future release, and quite possibly weathermap itself after that – I already did a little bit of experimentation with that.

*¬†A huge thank you to those contributors though! It is still my goal to build a development group around the project if possible. For me personally, that still means simplifying, modernising, clarifying the internals, to make it easier for others to get started and contribute. I do still see this as the ‘dark ages’ before a brave new world of open source weathermap – it’s just taking longer than I hoped! ūüôā

Site and project updates

First of all, apologies for the slow progress, but the “obstacles”* in my life that have been slowing things down are clearing up. The website has a temporary new look, after the previous theme I’ve been using seems to have dramatically changed between updates. This one happens to also match better with the other new change: a forum!

It’s rather empty at the moment. This is supposed to be the worst way to launch a forum, but I can’t see a way around that. I really wanted to have somewhere for people to talk about their weathermap issues that wasn’t necessarily dependent on Cacti. I know lots of Weathermap users are not Cacti user. I also wanted somewhere better than Github Issues to discuss and dissect problems, before they become specific issues on Github. Also, somewhere to try and grow some development community. I’ve been having good discussions about this with some folks recently, and there some great early work by Martin over there about getting going with development stuff.

So progress should speed up a bit for what I’ve decided to call Weathermap 1.0.0 which will have Cacti 1.x (and Cacti 0.8.8) compatibility. Thanks to those who are hitting me with bug reports and even the odd pull request!

And I’m looking forward to the next release after that being hopefully from a group…

* Please don’t tell my new wife I said that!

Cacti 1.0 and Weathermap

Cacti 1.0 was released a few days ago, and with it, a new plugin API. Weathermap (even from github) doesn’t currently work with Cacti 1.0. Unfortunately, at least from a first look, it also seems that it’s not possible to easily have a plugin that supports both Cacti 0.8.x and Cacti 1.x. The Cacti Group’s own plugins have made a clean break, with new versions only supporting 1.0. I think it will be a while before everyone moves to 1.0 (personally I avoid version x.0 of anything, especially with such big internal changes), and that it’s unfair that to get new Weathermap features (or fixes!), you’d need to upgrade Cacti too. So I want to continue supporting 0.8.x with new versions also, at least until Weathermap is also settled down again. With PDO, and PHP 7 support, and the changes related to code improvement, I’d hope that even if I decide to stop supporting Cacti 0.8.x, someone else could continue, after that point.

After a very busy new year, I’m back to thinking about Weathermap again, and will be finishing off the changes for PDO and PHP 7.0 soon. Once that version is released, then I’ll look at Cacti 1.0. It’s likely¬†that Weathermap will be broken into multiple downloads: a ‘core’ with the actual map functionality and probably the CLI, and then a series of extra downloads to provide the editor, Cacti 0.8.x plugin, Cacti 1.x plugin, standalone web app, and LibreNMS plugin. You would download the core plus whichever of the other features you actually need. To get both Cacti versions working, something like this is necessary, and users of other systems don’t need any of that code cluttering up their installation.

There is (as far as I know) one remaining set of failing tests before the next release – all related to colourising icons, and transparency.

5000 downloads of 0.98, and progress on 0.98a

Weathermap 0.98 hit 5000 downloads yesterday… woo!

Also, work is going well on the database changes for 0.98a. The actual database stuff has all been replaced and about 70% tested. Some of the other changes have highlighted some issues though, so I’m working on those now. The solution has been to pull in some of the refactored code from last year, which is tricky in places – it is all interdependent, so you have to try and break bits off cleanly. So now I am working through, weaving the two parts back together again.

Anyway, the end result will be some nicer code that most people won’t care about, and PHP 7 working, which they will! I don’t plan on this taking a lot longer, spare time permitting.

Making Weathermap “really” open source

Although Weathermap has always been available under the GPL, you could reasonably argue¬†that it’s not really open source. The reason it’s GPLed is simply that it was originally written in perl, based off of GRNET’s perl Weathermap, way back at the end of 2004. When I got tired of explaining how CPAN works (to install the perl GD module and dependencies) and switched to PHP, I kept the same license.

Despite being GPLed for 10 years, there have been very few code contributions. I can see why – it’s a fairly big tangled mess of PHP, considering the relatively simple job it does. It was my first PHP application from scratch too, and has lived long enough to be partly PHP 4 and partly PHP 5 style. Even if you do make the effort to read it all, you’ll find it has a few “god objects” which make modification harder than it needs to be. So effectively, it’s a personal project that you could re-use if you really wanted to.

Also the number of users – network engineers, mostly – who can and want to code is pretty small. I don’t want to discourage the few that remain!

Having worked a little on other “real”¬† open source projects, I really want to change this. For example, when I had an issue with beets, I was able to submit patches, get useful feedback from the project lead, and eventually get my change into the next release. It felt good, was collaborative, and solved my problem permanently too. That’s the kind of experience I want for Weathermap.

The first steps towards this have already happened – moving from subversion to github gets us the infrastructure to do Pull Requests, which is very powerful. There are the beginnings of unit tests, and more on the way as new code is added, to hopefully get to the stage that any PRs can be tested automatically using TravisCI. The code quality is (slowly) improving, and the ‘master’ branch on github has the results of some refactoring already. It may be that only some of that work makes it into a release, but the goal is to get to more and smaller classes, much reduced global variable usage and documented classes!

I have been using CodeClimate to highlight areas to work on, mostly for removing complexity and duplication so far. For example, processing of data source targets in the refactored version is a lot cleaner. Also, as UI code is moved into classes, it makes that easier to test too, which is a big gap currently Рthe image generation side of things has hundreds of tests. Similarly, there are Vagrant configuration files to automatically produce test environments on 10 or so difference Linux distros.

At the moment, things are still moving around, but once the next couple of releases are complete, I’ll be working on bringing the ‘new’ version and the 0.97-maintenance branches back together, and hopefully have a working version with as much of the new code as possible. Parts (especially the Node class) are still too big and clunky, but there is some useful work already done.

With that in place, I hope we can have a really open source network visualisation tool!

September 2016 – Current issues

tire-fire-1200

Just a quick note about known current issues with Weathermap 0.98:

  1. It doesn’t work with PHP 7.0¬†–¬†In PHP 7.0, the ancient mysql_xxxxx() functions that WM uses for database access are no longer available. The rest of the world moved on to PDO some time ago, but I didn’t bother because Cacti didn’t. PHP 7.0 is the default in some distros now (Ubuntu 16 at least) though, which makes it a bigger problem. I’m in the process (about 70% through) of fixing this by replacing all the database access code. This has some other benefits for the future, like the possibility of supporting other databases, and better database security (I’m moving to prepared statements everywhere too). I’m also separating out the actual database stuff from the Cacti plugin, so writing a plugin for something else should be easier too. The editor and command-line tool don’t use a database currently, so those are not affected.
  2. (some of) It doesn’t work with PHP 5.3 – Well, this is embarrassing! In fixing a deprecated code issue for PHP 5.6+, I added a problem for PHP 5.3. Luckily it’s a one-line fix, and only affects the command-line ‘weathermap’ tool. If you get PHP Parse errors with 0.98 and PHP 5.3, here is the solution¬†until 0.98a comes out.

Photo Credit: Wisconsin Department of Natural Resources

Links between maps

Introduction

Links between maps, and links to other sites are often requested as ‘new features’, even though they are already possible. Here are the things you need to know:

  • Any node or link can have a URL associated with it, using the INFOURL keyword. Usually, this is a link to a Cacti graph or something similar, but it doesn’t have to be.
  • A node doesn’t have to represent a router or device either.
  • For Cacti in particular, the ‘random’ ID in the weathermap name doesn’t ever change, and is safe to use in (HTML) links.

Simple version

So, putting all that together:

NODE link-to-other-map
    POSITION 300 300
    LABEL Other Map
    INFOURL http://my-cacti-server/plugins/weathermap/weathermap-cacti-plugin.php?action=viewmap&id=4df1356edd3366055c5f

Will make a text link to your other map, using Cacti’s authentication to make sure you are allowed to see it. If you don’t use Cacti, this would just be the URL for the other map as you see it in the browser.

Fancier version

How about having a thumbnail of the other map that you can click on?

As long as both maps are on the same Cacti server, Weathermap already generates a thumbnail to use in the web interface, which you can use too.

Some things you need to think about though:¬†Weathermap updates¬†the maps in the order they are listed in the management page, including the thumbnail updates. So if you want to be sure of getting the latest thumbnail, your master top-level map should be the last one in the list. You can put it in its own group though, so it isn’t all the way down a page of maps, and Weathermap remembers which group you last clicked on, so that shouldn’t be a problem either.

NODE link-to-other-map
    POSITION 300 300
    LABEL Other Map
    ICON 64 64 output/4df1356edd3366055c5f.thumb.png
    INFOURL http://my-cacti-server/plugins/weathermap/weathermap-cacti-plugin.php?action=viewmap&id=4df1356edd3366055c5f

So now we are using the thumbnail image as the icon for our node, and also telling Weathermap to scale it down so that it fits in a 64×64 pixel box.

Final Notes

The only real limitations are that you can’t use a URL for an ICON (this is possible with PHP, but not enabled on all installations. Also, try to avoid using the icon scaling to scale full-size images down to thumbnails, if you are concerned about performance (like updating a set of maps once per minute).

New Site

To get the blog functionality, I’ve switched from a simple static template to WordPress. This currently has limited content – just the text from the previous static page and the manual pages as before. In particular I want to add some images to the the front page – a graphical tool with no pictures isn’t so good! I hope it will all appear in the coming days.

Survey results: this blog!

I recently posted a survey for Weathermap users, asking how they use it, which features they use, and which features they would value most for future versions. One thing that was surprising was requests for features that already exist! So that implies that some better documentation, or tutorial information would also be good.

With that in mind, I’ve converted the static network-weathermap.com site to a blog, so that I can add tutorials here. I’ll try to think of ways to improve the reference documentation too.

I’ll also share some more of the results, once I have more, and have had time to process them.