This year got off a good start; I changed jobs and I am almost half way my target of jogging 100km! Speaking of jogging, it's been really hard to make serious progress even for a seasoned runner like me.
Part of it is because of my work schedule which allows me to report slightly late but also leave late sometimes. But 30% done is not bad.
Now when I am seated in front of my computer, containers, automation and concurrency are on my mind. The company I work with is actually moving in this direction, so that's a plus for me.
Containers and Orchestration: Docker + Kubernetes
How are developers shipping applications in 2018? How do you cope up with the pressure to make changes to code while not breaking things. How to do you collaborate with other developers in a way that their development environment is exactly the same as yours. And how do you autoscale your app and make it portable across various operating systems and cloud providers?
No doubt, containerization has been making waves in the developer community worldwide. Docker in particular has received a lot of media coverage and love from developers as the ultimate choice of containerizing their applications.
Kubernetes on the other hand has emerged as the winner for container management and orchestration. Several cloud providers including Google cloud, AWS, DigitalOcean now have support for Kubernetes.
So Docker and Kubernetes can make a great way of shipping and managing containerized apps at scale. Beyond the buzzwords, I have been playing with these tools myself to ascertain for sure if they solve my needs. Clearly the needs of Google or Amazon or Facebook can't be the same as those of a startup in Africa and so their choice of tools can't be the same. This is why I am learning about these technologies.
So far, I have managed to dockerize a part-time project of mine. It's simple monitoring app written in Python Django and Golang(later on this) and my goal was to dockerize it and deploy it using Kubernetes. There's still a lot I have to learn.
So far what I know is that you need high speed internet to pull, build and deploy containers. A docker container can be anywhere between 300-800MB. This can be extremely frustrating and costly if your internet speeds are less than 2Mbps, the average in Uganda.
[email protected] /h/o/w/d/docker# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
192.168.99.1:5000/sitemonkey_django latest 02ccd97de7d1 2 weeks ago 742MB
sitemonkey_django latest 02ccd97de7d1 2 weeks ago 742MB
weeks ago 742MB
192.168.99.1:5000/sitemonkeygo latest 9a53581d018c 3 weeks ago 804MB
sitemonkeygo 1.0 9a53581d018c 3 weeks ago 804MB
python 3.6.5-jessie 11aa3556fb90 3 weeks ago 691MB
phpmyadmin/phpmyadmin latest 4bdc31ab2ded 5 weeks ago 164MB
rabbitmq 3 64e7c1bc2efa 7 weeks ago 125MB
mysql latest a8a59477268d 8 weeks ago 445MB
richarvey/nginx-php-fpm latest 1bb16fc4c08f 2 months ago 303MB
Also the way you develop containerized apps can't be the same approach you use for conventional apps. Particularly you have to think about shared resources, persistence or state and services ahead of time. This is because your App will be running on a number of instances represented by a number of containers or Pods. What used to be fairly static resource like the file system or IP addresses now dynamic, so you can't refer to certain source by path on the file system on the server's hard drive or server IP address. You have to use Docker volumes, Service discovery on Kubernetes. These tools take care of mapping and allocating dynamic resources on your behalf.
There's a lot to learn here so much so that its a discipline of its own called DevOps.
Configuration management: Vagrant + Ansible
Until now, I have had to manually provision, deploy and configure a Virtual Machine(VM) or Virtual Private Server(VPS). I hated this manual process, so I have created some scripts like this one that automates LAMP stack installation. And If had to change something, I had to manually SSH into a box and edit configuration files. This is fine if you are managing a handful of servers, but if there are in the tens of hundreds, this becomes a mess.
I have got time to look into configuration management tools. Between Ansible, Salt, Chef and Puppet, I have decided to concentrate on Ansible for one reason; it's agentless. All it needs is python and openssh installed on remote hosts both of which come pre-installed on most Linux boxes. Chef, Puppet, Salt all need agents installed on remote hosts and a master servers that does the orchestration. So I have put those on hold.
Vagrant isn't configuration management tool but rather, it's used to automate provisioning and deployment of Virtual machines and I believe more recently containers. It works with existing hypervisors such as KVM and Virtualbox.
So I use Vagrant to provision and configure VMs, then Ansible kicks in to do the installation and configuration of system and application packages. You have to know where one stops and the other takes over.
We are currently using this combo with the Uganda dev team to build projects. It helps us have consistent development environments,so there are no instances of "it works on my machine, but it fails on yours".
But of course, I can reuse Ansible playbooks to configure even cloud VPNs from Linode, DigitalOcean, Google computer, AWS, Rackspace and others.
Concurrent programming and message brokers: Golang + RabbitMQ
Now on the code side, for a while I have been writing non-asynchronous code. Concurrency is headache; the developer has to deal with race conditions, deadlocks, livelocks, starvation among other issues. You could spend the whole day or weeks debugging concurrent code.
But Golang has made things a lot easier. Unlike other programming languages, Golang was designed with Concurrency in mind. Thanks concurrency abstractions such as goroutines and channels concurrency programming in Go is breeze. I have been reading this book Concurrency in Go: Tools and Techniques for Developers by Katherine Cox-Buday which I highly recommend if you want to dig in.
Now not all programs should be written using concurrency programming. But if your app routinely creates or accesses expensive resources such as making network or file system calls, you could make it run faster if written asynchronously.
Here's a code snippet from my Go side-project. I create channel variables to store results of a couple of sites which I pick from a db, then in a goroutine loop/range over the rows pushing results to respective channels. The results stored in the channels are used by other goroutines in the code.
sitesDown := make(chan map[string]string, 5)
sitesUp := make(chan map[string]string, 5)
go func() {
defer close(sitesDown)
defer close(sitesUp)
for {
sites := getSites()
for _, site := range sites {
site_id := site.Id
site_name := site.WebsiteName
url := site.WebsiteUrl
site_status := checkSite(url)
if site_status {
sitesUp <- map[string]string{"site_id": strconv.Itoa(site_id), "site_name": site_name}
} else {
sitesDown <- map[string]string{"site_id": strconv.Itoa(site_id), "site_name": site_name}
}
}
}
}()
While this checking runs asynchronously, alert messages are pushed to a RabbitMQ queue where an message alerts consumer are waiting to notify a user that their website is down. This is done via email and SMS of course using Africa's Talking API for the latter :) Another logs consumer listening in writes the incident to a log file.
Alerts producer
func publishAlert(site_name string, site_status string, alert_msg string, email string) {
conn, err := amqp.Dial("amqp://guest:[email protected]:5672/")
failOnError(err, "Failed to connect to RabbitMQ")
defer conn.Close()
ch, err := conn.Channel()
failOnError(err, "Failed to open a channel")
defer ch.Close()
err = ch.ExchangeDeclare(
"site_monitor", // name
"direct", // type
true, // durable
false, // auto-deleted
false, // internal
false, // no-wait
nil, // arguments
)
failOnError(err, "Failed to declare an exchange")
t := time.Now().Format(time.RFC3339)
body, err := json.Marshal(map[string]string{"site_name": site_name, "site_status": site_status,
"email": email, "timestamp": t})
err = ch.Publish(
"site_monitor",
"sites_status_key",
false,
false,
amqp.Publishing{
DeliveryMode: amqp.Persistent,
ContentType: "application/json",
Body: body,
})
failOnError(err, "Failed to publish a message")
}
Alerts consumer
func SendAlerts() {
fmt.Println("running eventhandler")
conn, err := amqp.Dial("amqp://guest:[email protected]:5672/")
failOnError(err, "Failed to connect to RabbitMQ")
defer conn.Close()
ch, err := conn.Channel()
failOnError(err, "Failed to open a channel")
defer ch.Close()
err = ch.ExchangeDeclare(
"site_monitor", // name
"direct", // type
true, // durable
false, // auto-deleted
false, // internal
false, // no-wait
nil, // arguments
)
failOnError(err, "Failed to declare an exchange")
q, err := ch.QueueDeclare(
"alert_queue", // name
true, // durable
false, // delete when unused
false, // exclusive
false, // no-wait
nil, // arguments
)
failOnError(err, "Failed to declare a queue")
err = ch.QueueBind(
q.Name,
"sites_status_key",
"site_monitor",
false,
nil,
)
failOnError(err, "Failed to bind queue with exchange")
msgs, err := ch.Consume(
q.Name, // queue
"alert_consumer", // consumer
false, // auto-ack
false, // exclusive
false, // no-local
false, // no-wait
nil, // args
)
failOnError(err, "Failed to register a consumer")
forever := make(chan bool)
go func() {
fmt.Printf("event go func running in the background")
for d := range msgs {
log.Printf("Received a message: %s", d.Body)
d.Ack(true)
var data map[string]string
err := json.Unmarshal(d.Body, &data)
if err != nil {
fmt.Println("error has occured", err)
}
log_msg := fmt.Sprintf("site %s is %s ", data["site_name"], data["site_status"])
fmt.Printf(log_msg)
site_name := data["site_name"]
status := data["site_status"]
email := data["email"]
t := data["timestamp"]
alert_msg := fmt.Sprintf("Site %s is back UP at %s", site_name, t)
email_subject := fmt.Sprintf("%s is back UP!", site_name)
if status == "down" {
alert_msg = fmt.Sprintf("Site %s has gone down at %s", site_name, t)
email_subject = fmt.Sprintf("Site %s is DOWN!", site_name)
}
fmt.Printf("Send an email notification with msg %s ", alert_msg)
emailSend(email, email_subject, alert_msg)
}
}()
log.Printf(" [*] Waiting for event messages. To exit press CTRL+C")
<-forever
}
Because of concurrency and message queueing, the app works reasonably faster than if it were written the conventional way. That is the alerting and logging subroutines for instance don't have to wait for checking module to finish running those http requests. And the checking subroutine doesn't have to wait for the alerts module to send the message first before it performs another check. Perfect!
So that's what I am doing now. I'll hopefully post more detailed articles on these individual subjects in the following weeks.
Image: Pixabay.com