Skin lesions classification: SparkR in DSX with IBM Watson visual recognition (9th WBME)

Written by fsmunoz on 14 April 2017 Categories: datascience Tags: ,

I have recently had the pleasure of paricipaing in the 9th Workshop on Biomedical Engineering that occurred in the Faculty of Sciences of the University of Lisbon; it is always refreshing to attend such events since one is exposed to a lot of extremely interesting research that is easy to miss when focusing on the day-to-day tasks.

My participation was twofold: a presentation concerning the use of IBM Watson in the bio-medical and health-care domains and a practical workshop to showcase how to use a concrete Watson API in a specific use-case, something which could be used to show what’s involved in using Watson and thus demystifying a bit the perceived complexity of using cognitive APIs. I ended up using IBM Watson Visual Recognition and Data Science Experience to build a step-by-step interactive tutorial on how to obtain and clean an existing curated skin lesions database to build a model that tries to identify malignant lesions:

Read on for the details.


No Comments on Skin lesions classification: SparkR in DSX with IBM Watson visual recognition (9th WBME)

Using MQTT in Common Lisp with ABCL and the Eclipse Paho client

Written by fsmunoz on 24 June 2013 Categories: Lisp, M2M

I have recently written some posts about using Clojure to take advantage of Java libraries. The idea of using a Lisp language that can take advantage of existing Java infrastructure is something that can however be achieved in different ways, Clojure being but one of them.

On the Scheme side I know at least GNU Kawa and SISC, and on the Common Lisp side Armed Bear Common Lisp (ABCL). Since I lean towards CL more than Scheme I have made some experiences with ABCL (including some comparisons with Clojure in terms of syntax which I will likely post about since the original content, sent to the ABCL mailing list as a pastebin link, is no longer accessible).

This is the simple MQTT example, which I previously wrote in Clojure, written in Common Lisp and using ABCL Java interop.

The use of ABCL Java facilities is of course essential to use the the Java libraries (see Hans Hübner’s post about using ABCL to parse Excel files using Apache POI for another example of the usefulness of this, and much better code-wise than my own code), so those parts aren’t portable to other CL implementations, but the rest of the code is. This means that by using ABCL one can use CL – which is a standartised language with multiple implementations – for the code, and that even with non-portable parts (which can be isolated) the bulk of the code is implementation independent.

Additionally (and at least equally important) it allows the use of Common Lisp wherever a JVM exists, which in itself is an advantage to those who know the language and do not want to program  in Clojure for whatever reason. There are several Java interop options built into ABCL – the example above uses  JSS 3, but there are also the included primitives and even Rich Hickey’s pre-Clojure framework JFLI.

No Comments on Using MQTT in Common Lisp with ABCL and the Eclipse Paho client

Publishing messages with MQTT, using Clojure and the Eclipse Paho client

Written by fsmunoz on 23 June 2013 Categories: Clojure, Lisp, M2M

As alluded in my last post I have found that MQTT (MQ Telemetry Protocol) is quite simple to use, and especially so when compared to most other messaging protocols. In particular it’s a great fit for embedded devices or whenever resources are limited, since it’s quite lightweight. From the MQTT page:


MQTT is a machine-to-machine (M2M)/”Internet of Things” connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport. It is useful for connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium. For example, it has been used in sensors communicating to a broker via satellite link, over occasional dial-up connections with healthcare providers, and in a range of home automation and small device scenarios. It is also ideal for mobile applications because of its small size, low power usage, minimised data packets, and efficient distribution of information to one or many receivers


On top of that I can see it being used when the needs themselves are straightforward, which is often the case: just publish a payload to a topic somewhere and avoid worrying with keeping up connection state, error handling, etc. Additionally it has been proposed as an OASIS standard which is good news for those looking for a messaging (and M2M in particular)

There are many implementation options to choose from  (see the MQTT software page for more details), this example code is based on the Eclipse Paho implementation. See the full code on the github gist, the following is a snippet:

Usage is trivial, here’s an example:

$ java -cp ~/.m2/repository/org/clojure/clojure/1.5.1/clojure-1.5.1.jar:/tmp/dummy/org.eclipse.paho.client.mqttv3.jar clojure.main mqtt.clj
Connected to tcp://
*** PUBLISHING TO TOPIC cljtest ***

… and seeing the publishing using the Mosquitto MQTT client to subscribe to the topic:

$ mosquitto_sub -h -t "cljtest" -v
cljtest But at least, out of my bitterness at what I'll never be, There's the quick calligraphy of these lines, The broken archway to the Impossible.

Yet again Clojure presents itself as a good way to take advantage of the Java libs, even when one doesn’t use most of other Clojure’s strengths.

Fernando Pessoa


PS: For the curious the standard message is from Fernando Pessoa‘s “Tobacco Shop“, part of his English Poems. Well worth exploring.

1 Comment on Publishing messages with MQTT, using Clojure and the Eclipse Paho client

Sending messages via Websphere MQ, a simple Clojure example

Written by fsmunoz on  Categories: Clojure, Lisp, M2M

I have been exploring different ways to integrate information (in the context of SmarterCities work) with a central repository that can be used to correlate it. The IBM Intelligent Operations Center comes with both Websphere MQ and Webpshere Message Broker, so this is the most obvious integration point I use for testing some ideas.

Since I’ve started using Clojure for some experiences – namely because it’s a good way to use the JVM and the Java libraries while still remaining in Lisp land – I went looking for ways to send messages.

After a while I decided to use the Websphere MQ classes for JMS to produce the example bellow (follow the links to the full gist):

A simple HTTP POST would actually suffice but I wanted to use some kind of messaging mechanism. I explored JMS – which on the surface sounded like the most obvious choice – but after a while I must confess becoming a bit lost on the number of operations, JNDI config files and above all necessary libs needed: it seems that JMS is the “standard Java messaging”, but relies on external providers for some of the work. What this meant was that it was not so “batteries included” as expected, while at the same time being a bit byzantine for my modest needs.

Defining a Message Broker workflow

A MQInput node listens to a specific queue, then passes it out to a FileOutput which appends the message in a file.

The Message Broker Toolkit makes it almost excindingly easy to pick this message and define a workflow; here I simply created a MQInput that listens to a node and passes it up to a File node that saves it to a temp file.

Much more complex routing and transformations are possible: nodes allow for immediate processing of, say, XML, JSON, MIME, which makes it trivial to get the information needed without any code at all.

I will follow this post with my preferred solution for M2M and light-weight messaging as a whole: MQTT, which is really easy to use. I didn’t even imagined how easy before venturing amidst the world of messaging in the context of service buses.

1 Comment on Sending messages via Websphere MQ, a simple Clojure example

Oracle Data Change Notification in Clojure

Written by fsmunoz on 15 June 2013 Categories: General, Lisp

I have been analysing a work-related thread related to how to integrate database “events”  – i.e. any information that reflects interesting actions or changes and that are currently in a RDBMS – with the IBM IOC. There are many ways to do it but each has different requirements and benefits and this is important when, say, in a situation where the database access is somewhat restricted, the ability to deploy specific software solutions limited (e.g. replication software like IIDR) or there are other more mundane constraints (like time and availability of resources).

Paper dolls with DCN events written on them doing a Dance Macabre

Nothing escapes me. No one escapes me. Not even database change notifications.


Enough of this though. In the particular case of Oracle databases there is a feature called Oracle Database Change Notification (aka “Oracle Continuous Query Notification” in more recent releases) that initially seemed helpful to me. The documentation seemed reasonably simple, even if I do not touch database stuff in ages and was never particularly good with it in terms of programming. Examples abound on the net, but mainly (as expected) in Java, which is quite expected since the features is expose through the JDBC driver.

Regardless of the actual applicability to any real project at hand I became curious about it since it seemed a generally useful feature. Since it uses JDBC I decided to use Clojure to keep it Lisp-y and cut down development time to a minimum. I will likely later on try the same with ABCL, which I really like, but for now Clojure seems to be easier to use when doing translations from Java, and in this case it worked quite well.

The result is in my Gitbhub repository oracledcn-clojure and is merely a small command-line tool that connects to an Oracle database via JDBC, registers for notification on whatever tables are affected by a query and then sends a notification to standard output each time there is a change. Note that this is client-side only so any actual changes (updates, inserts, etc.) will have to be done “manually”. This is trivial to add to the code but I wanted to focus on the notification aspect more than anything else. Do read the documentation since there is some small but fundamental setup needed because of the ojdbc6.jar file. The way that it is set up is likely to change since I’m using a feature that is not to be used in Real Projects (TM), but since this isn’t one I’m not terribly worried. I’m not particularly good with Clojure but this demo is to provide some quick information for those who are looking around for implementations and want to have an idea on how it works.

The demo itself is simple but could be easily extended to do more complex event transformation and routing: instead of printing the event it could analyse it, detect rows and tables, trigger additional queries and even transform the result into XML and submit it to a message queue, all within Clojure. This depends a lot of the business goal and specific technical environment: my business goal for this demo was to build wee paper puppets, write the event notification results on them and watch them do a Dance Macabre. I can imagine that there are other, less obvious applications for this technology and your business goals can be wildly different.

Overall notes and observations:

  • Make sure that there are no network obstacles: I spend most of the time trying to “fix” a problem that didn’t exist: I wasn’t getting any notification back, but a “select * from USER_CHANGE_NOTIFICATION_REGS;” was returning a correct entry. I changed the code, updated JDBC drivers, went back and forth and in the end I should have just began with “telnet myhost myport” from the database server. I was bitten by this because I assumed that a KVM image using NAT in my laptop would not have a problem communicating, but since this is is based on callbacks (and not in an established connection) firewall rules in both sides were blocking it.
  • The “select * from USER_CHANGE_NOTIFICATION_REGS;” query is a good way to see if things are working, the code itself checks for the registration ID but this query can be used directly on the database
  • I have found conflicting requirements for use of this feature; most documentation only mentions “grant change notification to foo;”, other sources indicate “GRANT EXECUTE ON DBMS_CHANGE_NOTIFICATION TO foo;” as also being required. YMMV.
  • The ojdc6.jar should, from what I’ve read, ideally be the same version, mainly because there are sometimes bugs in the authentication component. That being said this feature should work (as per the documentation at least) using a thin JDBC driver from 11g and an Oracle Database 10gR2.
  • DBeaver seems to be a good database manager if you’re not sure on what to use.

I like how Clojure feels but I can’t seem to wrap my mind around it; for a Lisp it has significant differences in terms of “paradigm” but also command names and standard libraries that leave me fumbling around. The easy interop with Java also has the side-effect of making one “lean back” into Java methods, which is most of the time not the most elegant approach. That being said leiningen works very well, it’s easy to get a lot of packages and it’s also very easy to distribute the final work: I can send a überjar for someone to click in and all they need as dependency ia a JVM.

Some important lessons in developing the code (which is extremely simple and only took about 2 hours of my own time because I’m not a programmer and I needed to get back to speed with some infrastructure details) are actually more related to the “social” aspect than to any technical details. A good example is how my lack of knowledge of the Clojure development toolkit was quickly surpassed by going to #clojure (NB: this is not an “hashtag” or whatever people call it, it’s a real IRC channel) and getting help directly from technomancy. This seemed something common to me but then I stopped and analysed it a bit more: leiningen is not exactly a small project, it’s *the* Clojure de facto’s configuration and dependency manager and used throughout the world in hundreds of projects. That some small doubts related to “how to add odbc6.jar to the classpath” could be dealt with the author is something that people coming from more “Enterprise” backgrounds would find unexpected to say the least – I know, because I talk to them.

This, in turn, reminded me something that Nic Ferrier (another #emacs habitué, of elnode and Teamchat fame to name a few) referred some time ago: using software that is maintained and developed by those on your close “social vicinity” is not only emotionally logical but also a practical benefit, and especially so when we are talking about communities which are IMO less artificial and more organic. Also, #clojure was very helpful, got some good answers to obvious questions, which is not always the case in programming communities.

Finally, the whole process just made me more sure that IRC continues to be one of the best “social networks” for people to actually build relationships from scratch. I never met the people in #emacs but I know several of them better that many people I work with. Other networks are fine for announcements and other more unidirectional patterns, but to this day I still think that IRC could be use to build quite an interesting social network, with links from the /whois referring back to source repositories, etc.

Well, I like the idea at least.

No Comments on Oracle Data Change Notification in Clojure