This year got off a good start; I changed jobs and I am almost half way my target of jogging 100km! Speaking of jogging, it's been really hard to make serious progress even for a seasoned runner like me.

Part of it is because of my work schedule which allows me to report slightly late but also leave late sometimes. But 30% done is not bad.

Now when I am seated in front of my computer, containers, automation and concurrency are on my mind. The company I work with is actually moving in this direction, so that's a plus for me.

Containers and Orchestration: Docker + Kubernetes

How are developers shipping applications in 2018? How do you cope up with the pressure to make changes to code while not breaking things. How to do you collaborate with other developers in a way that their development environment is exactly the same as yours. And how do you autoscale your app and make it portable across various operating systems and cloud providers?

No doubt, containerization has been making waves in the developer community worldwide. Docker in particular has received a lot of media coverage and love from developers as the ultimate choice of containerizing their applications.

Kubernetes on the other hand has emerged as the winner for container management and orchestration. Several cloud providers including Google cloud, AWS, DigitalOcean now have support for Kubernetes.

So Docker and Kubernetes can make a great way of shipping and managing containerized apps at scale. Beyond the buzzwords, I have been playing with these tools myself to ascertain for sure if they solve my needs. Clearly the needs of Google or Amazon or Facebook can't be the same as those of a startup in Africa and so their choice of tools can't be the same. This is why I am learning about these technologies.

So far, I have managed to dockerize a part-time project of mine. It's simple monitoring app written in Python Django and Golang(later on this) and my goal was to dockerize it and deploy it using Kubernetes. There's still a lot I have to learn.

So far what I know is that you need high speed internet to pull, build and deploy containers. A docker container can be anywhere between 300-800MB. This can be extremely frustrating and costly if your internet speeds are less than 2Mbps, the average in Uganda.

[email protected] /h/o/w/d/docker# docker images 
REPOSITORY                                                TAG                 IMAGE ID            CREATED             SIZE
192.168.99.1:5000/sitemonkey_django                       latest              02ccd97de7d1        2 weeks ago         742MB
sitemonkey_django                                         latest              02ccd97de7d1        2 weeks ago         742MB
weeks ago         742MB
192.168.99.1:5000/sitemonkeygo                            latest              9a53581d018c        3 weeks ago         804MB
sitemonkeygo                                              1.0                 9a53581d018c        3 weeks ago         804MB
python                                                    3.6.5-jessie        11aa3556fb90        3 weeks ago         691MB
phpmyadmin/phpmyadmin                                     latest              4bdc31ab2ded        5 weeks ago         164MB
rabbitmq                                                  3                   64e7c1bc2efa        7 weeks ago         125MB
mysql                                                     latest              a8a59477268d        8 weeks ago         445MB
richarvey/nginx-php-fpm                                   latest              1bb16fc4c08f        2 months ago        303MB

Also the way you develop containerized apps can't be the same approach you use for conventional apps. Particularly you have to think about shared resources, persistence or state and services ahead of time. This is because your App will be running on a number of instances represented by a number of containers or Pods. What used to be fairly static resource like the file system or IP addresses now dynamic, so you can't refer to certain source by path on the file system on the server's hard drive or server IP address. You have to use Docker volumes, Service discovery on Kubernetes. These tools take care of mapping and allocating dynamic resources on your behalf.

There's a lot to learn here so much so that its a discipline of its own called DevOps.

Configuration management: Vagrant + Ansible

Until now, I have had to manually provision, deploy and configure a Virtual Machine(VM) or Virtual Private Server(VPS). I hated this manual process, so I have created some scripts like this one that automates LAMP stack installation. And If had to change something, I had to manually SSH into a box and edit configuration files. This is fine if you are managing a handful of servers, but if there are in the tens of hundreds, this becomes a mess.

I have got time to look into configuration management tools. Between Ansible, Salt, Chef and Puppet, I have decided to concentrate on Ansible for one reason; it's agentless. All it needs is python and openssh installed on remote hosts both of which come pre-installed on most Linux boxes. Chef, Puppet, Salt all need agents installed on remote hosts and a master servers that does the orchestration. So I have put those on hold.

Vagrant isn't configuration management tool but rather, it's used to automate provisioning and deployment of Virtual machines and I believe more recently containers. It works with existing hypervisors such as KVM and Virtualbox.

So I use Vagrant to provision and configure VMs, then Ansible kicks in to do the installation and configuration of system and application packages. You have to know where one stops and the other takes over.

We are currently using this combo with the Uganda dev team to build projects. It helps us have consistent development environments,so there are no instances of "it works on my machine, but it fails on yours".

But of course, I can reuse Ansible playbooks to configure even cloud VPNs from Linode, DigitalOcean, Google computer, AWS, Rackspace and others.

Concurrent programming and message brokers: Golang + RabbitMQ

Now on the code side, for a while I have been writing non-asynchronous code. Concurrency is headache; the developer has to deal with race conditions, deadlocks, livelocks, starvation among other issues. You could spend the whole day or weeks debugging concurrent code.

But Golang has made things a lot easier. Unlike other programming languages, Golang was designed with Concurrency in mind. Thanks concurrency abstractions such as goroutines and channels concurrency programming in Go is breeze. I have been reading this book Concurrency in Go: Tools and Techniques for Developers by Katherine Cox-Buday which I highly recommend if you want to dig in.

Now not all programs should be written using concurrency programming. But if your app routinely creates or accesses expensive resources such as making network or file system calls, you could make it run faster if written asynchronously.

Here's a code snippet from my Go side-project. I create channel variables to store results of a couple of sites which I pick from a db, then in a goroutine loop/range over the rows pushing results to respective channels. The results stored in the channels are used by other goroutines in the code.

   sitesDown := make(chan map[string]string, 5)
	sitesUp := make(chan map[string]string, 5)
	go func() {
		defer close(sitesDown)
		defer close(sitesUp)
		for {
			sites := getSites()
			for _, site := range sites {
				site_id := site.Id
				site_name := site.WebsiteName
				url := site.WebsiteUrl
				site_status := checkSite(url)
				if site_status {
					sitesUp <- map[string]string{"site_id": strconv.Itoa(site_id), "site_name": site_name}
				} else {
					sitesDown <- map[string]string{"site_id": strconv.Itoa(site_id), "site_name": site_name}
				}

			}

		}
	}()

While this checking runs asynchronously, alert messages are pushed to a RabbitMQ queue where an message alerts consumer are waiting to notify a user that their website is down. This is done via email and SMS of course using Africa's Talking API for the latter :) Another logs consumer listening in writes the incident to a log file.

Alerts producer

func publishAlert(site_name string, site_status string, alert_msg string, email string) {
	conn, err := amqp.Dial("amqp://guest:[email protected]:5672/")
	failOnError(err, "Failed to connect to RabbitMQ")
	defer conn.Close()

	ch, err := conn.Channel()
	failOnError(err, "Failed to open a channel")
	defer ch.Close()

	err = ch.ExchangeDeclare(
		"site_monitor", // name
		"direct",       // type
		true,           // durable
		false,          // auto-deleted
		false,          // internal
		false,          // no-wait
		nil,            // arguments
	)
	failOnError(err, "Failed to declare an exchange")

	t := time.Now().Format(time.RFC3339)
	body, err := json.Marshal(map[string]string{"site_name": site_name, "site_status": site_status,
		"email": email, "timestamp": t})
	err = ch.Publish(
		"site_monitor",
		"sites_status_key",
		false,
		false,
		amqp.Publishing{
			DeliveryMode: amqp.Persistent,
			ContentType:  "application/json",
			Body:         body,
		})

	failOnError(err, "Failed to publish a message")
}

Alerts consumer

func SendAlerts() {
	fmt.Println("running eventhandler")
	conn, err := amqp.Dial("amqp://guest:[email protected]:5672/")
	failOnError(err, "Failed to connect to RabbitMQ")
	defer conn.Close()

	ch, err := conn.Channel()
	failOnError(err, "Failed to open a channel")
	defer ch.Close()

	err = ch.ExchangeDeclare(
		"site_monitor", // name
		"direct",       // type
		true,           // durable
		false,          // auto-deleted
		false,          // internal
		false,          // no-wait
		nil,            // arguments
	)
	failOnError(err, "Failed to declare an exchange")

	q, err := ch.QueueDeclare(
		"alert_queue", // name
		true,          // durable
		false,         // delete when unused
		false,         // exclusive
		false,         // no-wait
		nil,           // arguments
	)
	failOnError(err, "Failed to declare a queue")

	err = ch.QueueBind(
		q.Name,
		"sites_status_key",
		"site_monitor",
		false,
		nil,
	)

	failOnError(err, "Failed to bind queue with exchange")

	msgs, err := ch.Consume(
		q.Name,           // queue
		"alert_consumer", // consumer
		false,            // auto-ack
		false,            // exclusive
		false,            // no-local
		false,            // no-wait
		nil,              // args
	)
	failOnError(err, "Failed to register a consumer")
	forever := make(chan bool)
	go func() {
		fmt.Printf("event go func running in the background")
		for d := range msgs {
			log.Printf("Received a message: %s", d.Body)
			d.Ack(true)
			var data map[string]string
			err := json.Unmarshal(d.Body, &data)
			if err != nil {
				fmt.Println("error has occured", err)
			}
			log_msg := fmt.Sprintf("site %s is %s ", data["site_name"], data["site_status"])
			fmt.Printf(log_msg)
			site_name := data["site_name"]
			status := data["site_status"]
			email := data["email"]
			t := data["timestamp"]
			alert_msg := fmt.Sprintf("Site %s is back UP at %s", site_name, t)
			email_subject := fmt.Sprintf("%s is back UP!", site_name)
			if status == "down" {
				alert_msg = fmt.Sprintf("Site %s has gone down at %s", site_name, t)
				email_subject = fmt.Sprintf("Site %s is DOWN!", site_name)
			}
			fmt.Printf("Send an email notification with msg %s ", alert_msg)
			emailSend(email, email_subject, alert_msg)
		}
	}()
	log.Printf(" [*] Waiting for event messages. To exit press CTRL+C")
	<-forever

}

Because of concurrency and message queueing, the app works reasonably faster than if it were written the conventional way. That is the alerting and logging subroutines for instance don't have to wait for checking module to finish running those http requests. And the checking subroutine doesn't have to wait for the alerts module to send the message first before it performs another check. Perfect!

So that's what I am doing now. I'll hopefully post more detailed articles on these individual subjects in the following weeks.

Image: Pixabay.com