Tag Archives: troubleshooting

Troubleshooting lifecycle

I seem to have hit on a troubleshooting pattern for trying to get new services up and running: this time Rabbitmq.

Most of the application stuff I have been working with recently has been  disaster (and I know I’m of the opinion that most software – even, particularly my own – is rubbish) having to abandon development with Rails and Django because of basic stuff that just doesn’t work (or fails silently), and I’ve following the message queue posts on https://techtutorialsx.com/ and rather than sign up for a CloudMQTT account I thought I’d install rabbitmq locally. It can’t be that hard.

The web pages for Rabbitmq aren’t terribly inviting and I immediately suspect that there will be some winging it.

I installed the packages and start the service and try to use the rabbitmqctl command to see how things are going

# rabbitmqctl status
Status of node rabbit@fnunbob ...
Error: unable to connect to node rabbit@fnunbob: nodedown

DIAGNOSTICS
===========

attempted to contact: [rabbit@fnunbob]

rabbit@fnunbob:
 * connected to epmd (port 4369) on fnunbob
 * epmd reports: node 'rabbit' not running at all
 no other nodes on fnunbob
 * suggestion: start the node

current node details:
- node name: 'rabbitmq-cli-97@fnunbob'
- home dir: /root
- cookie hash: 63+eNTCkMJ1cMwrAcJ88rg==

Now, first off, ‘* suggestion: start the node’. What’s all that about! I thought I had started the node; there’s nothing I could find (easily) on the website to suggest anything else. Perhaps provide a clue on how to start the node!

Okay, so let’s try starting the node:

# rabbitmqctl start_app
Starting node rabbit@fnunbob ...
Error: unable to connect to node rabbit@fnunbob: nodedown

DIAGNOSTICS
===========

attempted to contact: [rabbit@fnunbob]

rabbit@fnunbob:
 * connected to epmd (port 4369) on fnunbob
 * epmd reports node 'rabbit' running on port 25672
 * TCP connection succeeded but Erlang distribution failed

* Authentication failed (rejected by the remote node), please check the Erlang cookie


current node details:
- node name: 'rabbitmq-cli-75@fnunbob'
- home dir: /root
- cookie hash: 63+eNTCkMJ1cMwrAcJ88rg==

Huh? I’m beginning to think that Rabbitmq is yet another piece of crapsoftware that just doesn’t work out the box: netstat and ps show plenty pof rbbitmq processes running. But, I’ll try some troubleshooting to figure out what I’ve done wrong.

Now, my troubleshooting pattern is to search (Google – expect Facebook adverts for message queue services in a few days) for the application and problem and ignore the links to the application vendor and start with the first StackOverflow page.

This leads me to http://stackoverflow.com/questions/10347751/rabbitmq-refusing-to-start and even the link is promising. And sure enough, it mentions the rabbitmq-server command, so we give it a go.

# rabbitmq-server
RabbitMQ 3.6.9. Copyright (C) 2007-2016 Pivotal Software, Inc.
 ## ## Licensed under the MPL. See http://www.rabbitmq.com/
 ## ##
 ########## Logs: /var/log/rabbitmq/rabbit@fnunbob.log
 ###### ## /var/log/rabbitmq/rabbit@fnunbob-sasl.log
 ##########
 Starting broker...
 completed with 0 plugins.

Maybe something’s happening, maybe not: CTRL-C; man rabbitmq-server. There’s a ‘-detached option@. Ah, okay, let’s try that.

# rabbitmq-server -detached
Warning: PID file not written; -detached was passed.

So, let’s see if that makes a difference.

# rabbitmqctl status
Status of node rabbit@fnunbob ...
[{pid,13168},
 {running_applications,
 [{rabbit,"RabbitMQ","3.6.9"},
 {ranch,"Socket acceptor pool for TCP protocols.","1.3.0"},
 {ssl,"Erlang/OTP SSL application","8.2"},
 {public_key,"Public key infrastructure","1.4.1"},
 {asn1,"The Erlang ASN1 compiler version 5.0","5.0"},
 {rabbit_common,
...

That’s more like it.

The important thing here is that StackOverflow is more useful for working with applications than the application documentation itself. Because there are so many thing that go wrong with modern software and they won’t likely be mentioned in teh official docs, SO catches all the efforts to fix things.

P.S. This is one of the most annoying things about Puppet. They must have a deal with Google to make sure only official documentation is returned, it links to the most recent puppet version (and enterprise to boot) and selecting from the version dropdown takes you to a 404 page: Just show the SO pages where people have fixed stuff and be done with it.