relaxdiego

KubeVirt on Autoscaling Baremetal Nodes

Sun, 20 Aug 2023 00:00:00 +0000

^{Co-Authored by Jonathan Kinred and Paul Myjavec}

For this article, we’ll learn about the process of setting up KubeVirt with Cluster Autoscaler on EKS. In addition, we’ll be using bare metal nodes to host KubeVirt VMs.

Required Base Knowledge

This article will talk about how to make various software systems work together but introducing each one in detail is outside of its scope. Thus, you must already:

Know how to administer a Kubernetes cluster;
Be familiar with AWS, specifically IAM and EKS; and
Have some experience with KubeVirt.

Companion Code

All the code used in this article may also be found at github.com/relaxdiego/kubevirt-cas-baremetal.

A Case for Monorepos

Sun, 02 Jul 2023 00:00:00 +0000

For some time now, we’ve been living in the era of microservices. An era where a strict separation of concerns is the norm. Where it is difficult to circumvent the rule by way of putting each microservice in its own repository and execution environment. This has, in many ways, permitted us to be more thoughtful about our system designs, allowing us to avoid the “big ball of mud” problem to a significant extent. However, this era of isolation has also presented a unique set of challenges, specifically in the aspect of code reuse.

CI/CD and Shared Environments

Sun, 24 Apr 2022 00:00:00 +0000

Dev/QA/Staging/Production environments are shared resources and, like any shared resource, access to them must be managed; otherwise environment corruption may occur, resulting in loss of data and precious engineering time.

Take for example this scenario where we have a pipeline that deploys to an environment:

Usually, a pipeline execution (PE) includes running database migrations, collating assets, adjusting load balancers, running automated tests against the newly deployed build, and so on. What this means is that this process can take a few seconds to a few minutes. Let’s now consider what will happen when, during the period when PE #1 is still using the environment, another PE comes into play:

If You Want to Work in DevOps

Mon, 28 Mar 2022 00:00:00 +0000

If you’re hoping to land a job that has “DevOps” in the title, understand that your overarching responsibility is to ensure that the organization’s software service delivery process is optimal and that the availability of each service meets user expectations.

To achieve that, you must ensure that the infrastructure supporting each service is robust. This is where your systems design skills come into play.

…and, should a service or its infrastructure experience an unhandled error, that you have instrumentation in place to let you know of it immediately. This is what alerting is for.

Reduce Your CI/CD Config Sprawl

Tue, 01 Mar 2022 00:00:00 +0000

If you’re still experiencing the pain of CircleCI config.yml copypasta even after using orbs, here’s a possible next step in your evolution.

Transcript

In CircleCI, your workflow configuration is stored alongside your code in .circleci/config.yml. This makes it easier to synchronize your CI/CD workflow with your code.

However, this also comes with disadvantages. For one, it can result in duplicate workflow and job definitions. This is especially problematic if you have multiple repositories that have the same workflow. For example, let’s say you have 3 distinct applications all written with the FastAPI framework and deployed to Kubernetes. In this case you’d have to make sure that their config files don’t diverge from one another. Otherwise you’ll have to frequently re-learn how each application is built and deployed.

TDD but for Networking

Fri, 25 Feb 2022 00:00:00 +0000

Tired of spending precious time trying to debug why one AWS resource can’t talk to another? Add VPC Reachability Analyzer to your toolkit, mix it up with some Test-Driven Development (TDD) workflow and take back some of your sanity!

Write the Test Create the Path, Watch it Fail…

Create the path to analyze before your networking config is fully configured then see helpful error messages like the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


{
 "destination": {
 "arn": "arn:aws:ec2:xx-xxxx-x:xxxxxxxx:vpc-peering-connection/pcx-xxxxxxxxx",
 "id": "pcx-xxxxxxx"
 },
 "explanationCode": "NO_ROUTE_TO_DESTINATION",
 "routeTable": {
 "arn": "arn:aws:ec2:xx-xxxx-x:xxxxxxxx:route-table/rtb-xxxxxxxxxxx",
 "id": "rtb-xxxxxxxxxxxxxx"
 },
 "vpc": {
 "arn": "arn:aws:ec2:xx-xxxx-x:xxxxxxxx:vpc/vpc-xxxxxxxxxxxx",
 "id": "vpc-xxxxxxxxxxxxxx"
 }
}

Write the Code Configure the Network, Watch it Pass…

Once you’ve fixed the errors above and re-run the analysis, you’ll get the following report:

Auto-testing code snippets in my blog

Wed, 16 Feb 2022 00:00:00 +0000

I finally got around to auto-testing the code snippets in my blog posts. As of today, the process of writing a blog post with code snippets goes like this:

Step 1: Run the background services

First I’ll run Foreman locally while in my blog’s dir. Foreman starts up three processes in the background: a Jekyll server, my résumé autorenderer, and a snippets autotester (the topic of this post). Each subprocess' stdout and stderr are printed by Foreman, with each line properly prefixed such that it says which process printed out which line:

Dealing with ISO 8601 in Python

Mon, 14 Feb 2022 00:00:00 +0000

ISO 8601 datetime strings are great but they come with surprising special conditions that you may need to consider when the datetime strings come from an external client. There are third-party libraries out there that can ease your burden but, for times when you don’t have the luxury of using anything but stock Python, read on.

NOTE: All code snippets below are available on Github.

Let’s start with the simplest case.

Rediscovering DevOps

Fri, 14 May 2021 00:00:00 +0000

Preamble

This article attempts to cut through the cruft and re-discover what it really means to practice DevOps in your organization. It does this by reviewing the three principles of DevOps: systems throughput, fast feedback, and continuous learning. It further re-inforces these principles with two tried and tested engineering concepts: The Theory of Constraints and Control Theory.

TL;DR

DevOps is just Process Engineering specialized for software delivery. The wealth of knowledge built up by Process and Industrial Engineering will prove useful in delivering software products.

Make Your Old Printer Airprint Capable

Fri, 22 Jan 2021 00:00:00 +0000

Learn how to turn a $10 Raspberry Pi Zero into an Airprint server that allows you to print wirelessly from your PC, Mac, or smartphone.

Links to things mentioned in the video:

Analyzing an Apache Log File

Thu, 24 Jan 2019 00:00:00 +0000

Pop quiz, hot shot! You need to quickly get an idea of the top 5 clients IPs of your website grouped by path and HTTP status code but only between 6PM and 7PM and only for requests with HTTP status code 40x. Quick! What tools do you pick up? Do you use go, python, C? What about the framework? And how do you onboard the rest of your team?

Slow down, take a deep breath. For this scenario, you don’t need anything more than Bash and friends. To get an idea of how to do that, try out this 60-liner bash script to get the data I mentioned above.

Keeping Continuous Integration Continuously Integrating

Thu, 16 Aug 2018 00:00:00 +0000

OUTDATED ARTICLE: This article has been superseded by a newer one. Please refer to that article instead.

Have you ever had to deal with the situation where your CI lags further and further as the day progresses because more builds are getting queued up for your long-running tests? Some of you may have tried to solve that by firing up multiple test environments and then assigning different builds to these different environments thereby achieving some parallelism but this can get very costly. Some of you may have tried converting your long-running tests into nightly jobs but this means you won’t get timely reports on broken builds.

Jenkins on Jenkins - Unit Testing Shared Libraries

Sun, 04 Feb 2018 00:00:00 +0000

If you’ve ever worked with Jenkins shared libraries, you know that they’re a great way to simplify your Jenkins Pipeline DSL scripts by abstracting out common code to a function. For example, this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


stage('Prep') {
 script {
 // Create the release ID for this build
 def version = readFile('VERSION').trim()
 def buildID = env.BUILD_ID
 def shortSha = env.GIT_COMMIT.take(7)
 def prMetadata = ''

 if (env.CHANGE_ID) {
 prMetadata = "pr${env.CHANGE_ID}."
 }

 env.releaseID = "${version}+${prMetadata}.b${buildID}.${shortSha}"
 }
}

Could become this:

Canal and Calico Networking for k8s

Wed, 13 Dec 2017 00:00:00 +0000

Here’s a quick diagram on how Canal and Calico networking is implemented in Kubernetes. If the image below is hard to read, here’s a direct link.

NOTE: While I’ve depicted the iptables rules and routes as boxes in these diagrams, note that they’re not really implemented as Linux devices. Rather, I just wanted to show how the routes are applied to the packets as they make there way to and fro the actual devices.

Visualizing Brooks' Law

Mon, 27 Nov 2017 00:00:00 +0000

“Adding human resources to a late software makes it later”
– Fred Brooks, The Mythical Man-Month

I hear this truism regularly and I can also feel its effects whenever new members join a team I’m in but I’ve never really given it much thought until now. In this blog post, I’ll attempt to dissect and visualize Brooks’ Law to find out what it is exactly about adding new team members that negatively affects short-term team progress.

September SF k8s Lightning Talk

Thu, 02 Nov 2017 00:00:00 +0000

Here’s my lightning talk from last September’s SF Kubernetes Meetup. Thanks to Stack Point Cloud, Bitnami, and Weave Works for making this event happen!

UPDATE: Since that lightning talk, we’ve moved away from the VM-based setup and started investigating other methods that still satisfy the 3 hard requirements mentioned in the talk.

A Framework for Qualitatively Evaluating an Incubator

Wed, 01 Nov 2017 00:00:00 +0000

While drafting a series on CI/CD these past few weeks, I came across an old research paper that I wrote back in my Master of Management program at the Australian National University* in Canberra. My research was focused on creating a wholistic framework for evaluating the efficacy of a business incubator within a community.

It’s still in its nascent form to this day since I haven’t had the chance to continue working on it but I’m sharing it here, warts and all, in case anyone might find this useful. If you do end up basing your work on it or further improve it, please don’t hesitate to contact me. I’d love to see how much further this framework can go!

Introducing the Magic of Computing to Kids

Fri, 20 Oct 2017 00:00:00 +0000

Today, I spent my morning introducing the magic of programming to a class of 2nd graders and it was amazing. It was also one of the more nerve-racking experiences as far as preperations go. I was so nervous I spent the better portion of my morning (starting at 6AM) researching and preparing!

Having the Creative Computing site was a huge thing because it pretty much wrote all the material that I needed and more! For a 45-minute session though, I only had time to get them excited about Scratch and to try out a few very simple exercises. Basically, I boiled down the session to the following activities:

Fully Automated Blog and Resume

Sat, 23 Sep 2017 00:00:00 +0000

I finally got around to fully automating my blog generation including the resume rendering part (both HTML and PDF formats). These are the changes that made that happen.

So now my workflow just involves starting ~~Foreman~~ Goreman in one terminal session while I work inside Vim in another session. tmux, of course, makes all of that easy peasy.

Automation rocks.

Product Radial Evaluation Framework (P-REF)

Sun, 03 Sep 2017 00:00:00 +0000

This post is a load of bullcrap. Don’t use it. I’m just keeping it online for historical purposes

So you’ve come across what you think is the right solution to a market problem and now you’ve gone ahead and built a team to develop it. How can you monitor that you and your team are developing the solution the right way? This framework can potentially help you determine if you’ve covered all your bases and if there are problem spots that you need to address before going any further.

A Team's API Contract

Wed, 30 Aug 2017 00:00:00 +0000

This is an old email that I sent to my teammates. I’m sharing this for posterity and also because I think it applies to any software team.

Hi team!

I hope you’ve had a good week this week and, if not, well it’s a Friday!

I wanted to talk to you about the “API contract” between you and me because it facilitates the team’s morale and productivity. Before I proceed, I want you to fully understand that what I write below IS NOT law. Instead, consider it as a conversation starter: “This is how I see things. Tell me if it’s different from what you are seeing.”

HA Kubernetes Outside the Cloud

Sat, 26 Aug 2017 00:00:00 +0000

Let’s say you want to install a piece of software on your desktop. Let’s make it more fun and say you want to install the latest, coolest, most awesomest (yes, I just made that word a thing!) video game. What’s the first thing you want to check before you buy it? You normally would check the system requirements. How much CPU power does it need? Do you need to buy more RAM? Will it be fine on HDDs (spinning disks) or does it require SSDs? Should you install (and therefore maintain/operate) some liquid-based cooling mechanism on your machine? These are very important questions to ask because it will affect your overall gaming experience. Depending on your preferences, you might get by with the minimum requirements or you might want to soup up your gear before proceeding. It’s really a function of “how much of the total experience do you want to have?” and “how much of it can you afford?”

On Sizing Stories

Sun, 23 Jul 2017 00:00:00 +0000

Let’s take a quick break from coding and talk about planning in a Scrum-based team. Specifically, let’s talk about sizing or assigning points to stories.

Even after more than 8 years* of practicing Scrum, I still encounter teams who find it hard to grasp the idea of story points and, unfortunately, fall back to the old ways of equating a point to a certain number of hours or days. I can’t blame them though because, when I was just starting with Scrum, I’ve read so many conflicting publications on this topic and, combined with all the other “alien” concepts introduced by Agile, it took a while to wrap my head around it. The “aha!” moment happened for me after I read this book about 7 years ago and what I’m about to share below is largely based off of that.

Templating and Testing With Bash

Fri, 12 May 2017 00:00:00 +0000

I needed to write a simple templating system in Bash and wanted to do it with TDD helping me along the way. Here’s what I came up with.

Prior Art

I’m not the first person to do this. I drew my inspiration from the following:

My Requirements

I wanted to be able to write a template script that contained placeholder values to be interpolated by my bash templating system as well as regular variables that should be left untouched. For example:

Load Balancing Sockets

Fri, 03 Feb 2017 00:00:00 +0000

If you need to write a service or agent that can cater to multiple requests in parallel coming in from a single socket, you want to have multiple subprocesses ready to cater to those requests in parallel. But the question now is how do you ensure the requests are load balanced across those subprocesses? Do you build a “Master” process that takes care of load balancing, handing off those requests to the subprocesses as needed?

Fully-tested Ansible Modules from AnsibleFest NY 2016

Mon, 24 Oct 2016 00:00:00 +0000

Writing Ansible Modules Part 5 - Helpful Resources

Sat, 01 Oct 2016 00:00:00 +0000

^{Co-Written by Andreas Hubert}

This is part 5 of a series of articles. For other parts, see the introductory article.

I’m So Proud of You!

You did a rockin’ job finishing this series! Now it’s time for you to continue the journey on your own but not necessarily alone. Here are some resources that will help you become an effective module developer.

The official module checklist - This is a very useful set of items to review especially just before you submit code upstream. Keep it in your bookmarks!

Writing Ansible Modules Part 4 - Submitting Code Upstream

Fri, 30 Sep 2016 00:00:00 +0000

^{Co-Written by Andreas Hubert}

This is part 4 of a series of articles. For other parts, see the introductory article.

Time To Push to Origin!

Run the following from the ansible repo:

$ git add cloud/
$ git add test 
$ git commit -m "First module including tests"
$ git push origin

Head on over to your Github fork and create a pull request out of the commit you just pushed. Once the PR has been created, copy the PR’s URL to your clipboard since you will need to reference it from the next PR we will create next.

Writing Ansible Modules Part 3 - Complete the Module

Thu, 29 Sep 2016 00:00:00 +0000

^{Co-Written by Andreas Hubert}

This is part 3 of a series of articles. For other parts, see the introductory article.

Let’s Implement fetch()

First, let’s write the test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


 @patch('ansible.modules.cloud.somebodyscomputer.firstmod.open_url')
 def test__fetch__happy_path(self, open_url):
 # Setup
 url = "https://www.google.com"

 # mock the return value of open_url
 stream = open_url.return_value
 stream.read.return_value = "<html><head></head><body>Hello</body></html>"
 stream.getcode.return_value = 200
 open_url.return_value = stream

 # Exercise
 data = firstmod.fetch(url)

 # Verify
 self.assertEqual(stream.read.return_value, data)

 self.assertEqual(1, open_url.call_count)

 expected = call(url)
 self.assertEqual(expected, open_url.call_args)

Line 1 - We patch an Ansible function which we expect fetch() to call
Line 4 - The url that we will pass on to fetch(). We’re assigning it to a variable here because we want to verify if it gets passed on to open_url()
Lines 7 to 10 - We mock the IO object that open_url returns to fetch()
Lines 16 to 21 - We verify if fetch() returned the correct data and that it called the underlying open_url() correctly.

Run the test and see it fail. Let’s now write the code that makes it pass. First, add this near the top of your firstmod.py right after the first import line:

Writing Ansible Modules Part 2 - Write Your First Module

Wed, 28 Sep 2016 00:00:00 +0000

^{Co-Written by Andreas Hubert}

This is part 2 of a series of articles. For other parts, see the introductory article.

It’s Just Somebody’s Computer

Let’s write a module for a fictitious cloud provider named Somebody’s Computer. First, let’s create our module’s subdir:

$ mkdir cloud/somebodyscomputer
$ touch cloud/somebodyscomputer/__init__.py

From now on, I’ll refer to this as your module dir.

Let’s Kick The Tires for a Bit

First, let’s create a topic branch from the HEAD of devel and work there. Working on a topic branch has its advantages in that, should new changes be added to upstream/devel, all you have to do is fetch those and then rebase your topic branch on top of it.

Writing Ansible Modules Part 1 - Preparing Your Environment

Tue, 27 Sep 2016 00:00:00 +0000

^{Co-Written by Andreas Hubert}

This is part 1 of a series of articles. For other parts, see the introductory article.

Prerequisites

You’ll need a few things handy before you can continue:

Git branching basics. Learn Git Branching is a great resource!
Python >= 2.7.8 && < 3.0.0. NOTE: Ansible works with Python 3.0+ and you might be able to get the code samples in this article to work on it but I haven’t tried it yet.

Writing Ansible Modules Complete With Tests

Fri, 10 Jun 2016 00:00:00 +0000

^{Co-Written by Andreas Hubert}

While writing my first ansible module, I noticed that there wasn’t any resource that completely described how to get started on my local dev environment. This article documents the steps that I took to get up and running. Hopefully it will be a helpful resource for you too.

I should mention though that the following resources helped me get started quickly in certain aspects. So a big thanks to the these folks! This article builds on the foundations that they’ve started.

My Dev Setup (Part 3)

Sat, 07 Nov 2015 00:00:00 +0000

It’s been over a year and a half since I posted about my development setup. A lot has changed since then so I decided it was time to provide an update. The most drastic change is that I’ve dropped TextMate, Sublime, and Atom and finally took the plunge into the world of Vim and Tmux and, so far, I have been very impressed with the results.

Swap Ctrl and Caps Lock!

This is very important. Seriously. A lot of the commands and actions that I mention below use Ctrl-A and Ctrl-(Vim movement keys). Swapping my Ctrl and Caps Lock keys allows my fingers to stay in the home row most of the time.

My First Attempt at Kubernetes

Thu, 05 Nov 2015 00:00:00 +0000

Notes from CoreOS Fest 2015

Thu, 07 May 2015 00:00:00 +0000

I went to the first CoreOS Fest this week and the following photo, of which I am clearly the main subject, proves it!

Source: @bradfitz on Twitter

Now that that’s out of the way, here are my notes from that event.

Container all the things!

Oh yes, that reminds me. Before I talk about CoreOS Fest, let’s first establish a common ground on this containerization thing. There appears to be some confusion about what it is and whether it makes VMs obsolete (it does not).

Automated Testing, Organizational Learning, and Thermostats

Wed, 28 Jan 2015 00:00:00 +0000

Agile software development is oftentimes touted as being a faster and more efficient way of delivering software, as a way to reduce time to market, and as a means to quickly respond to constantly changing requirements. That’s all well and good until the message is misunderstood to mean “let’s skip documentation and testing and just fix things as we go so that we can deliver within the arbitrarily set deadline.” That’s not to say that Agile is no good. Rather, it’s to point out that the need for software development speed must be tempered by the need to retain organizational learning.

Using Aviator to Seed OpenStack

Thu, 02 Oct 2014 00:00:00 +0000

I added a change to Aviator so that you can use it to seed OpenStack. I had not thought about this scenario when I built Aviator. The assumption from the get go was that it will be used at that point in time when OpenStack has been installed and fully configured. We were very lucky though that the changes needed to support this unexpected requirement were very minimal. In fact, the changes were more syntactic sugar than anything else. I’ll get to that part in a bit. First, let’s see how one might use Aviator to seed OpenStack.

All About the OVSDB Server

Sun, 21 Sep 2014 00:00:00 +0000

In my previous post, I briefly introduced the OVSDB Management Protocol in the process of talking about hardware VTEPs. In this post, I will focus on specific OVSDB Server features. And to do that, I will (mis)use it outside of SDN.

Huh??? But OVSDB == SDN!

Actually, no. OVSDB can exist independent of Open vSwitch. In fact, what’s actually happening under the covers is that Open vSwitch connects to OVSDB and monitors certain tables in the Open_vSwitch database for updates. When OVS receives update notifications from OVSDB, the former then gets to work creating and manipulating bridges, ports, etc.

Understanding Hardware VTEPs

Tue, 09 Sep 2014 00:00:00 +0000

In my previous post, I announced that I open sourced my Open vSwitch lab as well as the scripts and files that I produced as I followed David Mahler’s introduction to VXLAN overlay networks with Open vSwitch. In this post, I’m going to re-purpose my Open vSwitch lab to explore hardware VXLAN tunnel endpoints (HW VTEP).

A Quick Review of Hardware VTEPs

The idea behind hardware VTEPs is to create an overlay network that connects VMs and Physical Servers and make them think that they’re in the same L2 network even if they’re on opposite sides of the globe.

Your Own Open vSwitch Lab

Wed, 03 Sep 2014 00:00:00 +0000

I’ve been spending a lot of time studying Open vSwitch and VXLAN lately and, often, found the need to quickly destroy and recreate my OVS environment. To speed up that process, I created my own Vagrant project which I’ve just made available on my Github account. You may find it here.

With a simple vagrant up command, you will be able to spin up two host machines that talk to each other over a simulated Internet (which is implemented as another VM). Each host machine comes with OVS 2.x and Mininet so that you can spin up a handful of guest machines without consuming too much resources. Each host machine consumes only 256 MB of RAM but can be increased via the Vagrantfile in the project’s root directory.

Aviator now supports Keystone V3 authentication

Sun, 13 Jul 2014 00:00:00 +0000

I just recently merged a significant change to the master branch. This change now allows for authenticating against Keystone’s V3 API. That means that, as of right now, running the create server demo using the following config file works as expected (must use bleeding-edge Aviator from HEAD of master).

Override API version to use

You may have noticed that extra section at the bottom of the sample config file above:

compute_service:
 api_version: v2

That setting allows you to override Aviator’s behavior and make sure that you always use Nova’s v2 API, even if API V3 is available alongside V2. In fact, if more than one API version is available for a service, you will need to specify which one you want to choose in the configuration file.

Hiding Sensitive Data from Logs with Python

Sat, 05 Jul 2014 00:00:00 +0000

^{Co-Written by Hari Dara and uenz}

Last week I’ve had to wrangle with Python’s documentation because I needed one of the apps I’m writing to centrally remove sensitive information from the logs. After several attempts at understanding the documentation, I’ve come to appreciate the built-in logging framework. I also have to admit that it’s more powerful than Ruby’s built in logger.

There are definitely more than one approach to solve this problem, but I will discuss a couple of approaches below:

Hardware VTEP and OVSDB

Fri, 27 Jun 2014 00:00:00 +0000

Here are some of my rough notes on OVSDB and hardware VTEP.

The Rise of Virtual Networks

With the advent of Software Defined Networking (SDN), new networking capabilities are opened up to an organization. Just imagine being able to programmatically request from your provider a two-fold increase in your bandwidth between 10 AM and 12 AM today when you will need to send a bulk of data offsite and then paying only extra for that increase within that time. It’s an exciting time for networking as it finally completes the utility-based computing we’ve been talking about for years now. Granted we can’t do that quick bandwidth burst as of today, but we’re getting there.

Analyzing Team Performance Using Trello Data

Sat, 12 Apr 2014 00:00:00 +0000

UPDATE 2014-04-24: Code related to this post is now available in Github.

We’ve been using Trello exclusively at my new job for managing our tasks. As an outsider, I’d say that the Trello team did a very good job of identifying the application’s core features and relentlessly focusing on those. One of the features that I appreciate is how it made creating and updating cards so easy and fast. As an agile developer for over 5 years now, I’ve found that the lack of this feature is invariably one of the things that stand in the way of a team fully adopting agile practices. That’s also what led me to write Redmine Backlogs back in 2008.

Mocking Objects in Python

Mon, 07 Apr 2014 00:00:00 +0000

Mocking in Python can be initially confusing and the official docs, while informative, don’t make learning it any easier for newcomers. In my case, I came across it when I was still transitioning from Ruby which, I believe, contributed to the confusion. This article takes a gentler approach to learning Python’s mocking library to help you get productive sooner.

Importing Objects: A Review

The unittest.mock library builds on top of how Python implements the import statement so it’s imperative that we have a solid understanding of how it works before we can continue.

My Development Setup (Python Edition)

Sun, 09 Mar 2014 00:00:00 +0000

Now that my work includes coding in Python, the first thing that I paid attention to was all the Python tools that allowed me to replicate my dev setup in Ruby. Here’s what I have so far.

TDD

Testing to me these days is a lot like seatbelts while riding a car. I do not feel safe without them. So mastering a testing framework is always top priority for me when diving into a new language. I tried nose but it appears to have lost favor from one of the open-source communities I’m in.

I Love Elegant Solutions

Sun, 15 Sep 2013 00:00:00 +0000

Last week while I was playing around with ManaMana’s self-testing capability, I came across a problem where I needed the RDSL syntax to allow for preformatted blocks in requirements such as the following:

* A Wiki page displays the following text
 ```
 Dear World,
 Hello!
 ```
 when it is newly generated.

Modifying the RDSL lexer to recognize the preformatted blocks was the easy part. The problem after that was how do I strip whitespace in the requirements while preserving it inside the preformatted blocks? Initially, I thought about the following process:

Testing ManaMana with ManaMana

Fri, 13 Sep 2013 00:00:00 +0000

Last night at the September LA Ruby Meetup, I got the chance to present ManaMana, my BDD framework hobby project, and the tools I used to build it. At one point during the presentation, Michael Hartl (author of Rails Tutorial), asked if ManaMana can test itself. It didn’t at that time.

Obviously, the idea of a self-testing ManaMana gnawed on me for the rest of the night. So earlier today, I decided to give it a go. Now, I have these files in the project and when I run bundle exec guard from the command line and save the requirement or test case files above, ManaMana compiles the test and produces the following output. So yeah, we now have a self-testing test framework with ManaMana.

Mocking Web Services with VCR

Sat, 22 Jun 2013 00:00:00 +0000

When writing applications or libraries on top of 3rd party web services, one often encounters connectivity problems ranging from slow to zero connection. Then there’s the issue of rate limits where your machine can only make X number of requests per day. Any one of these issues is a killer on developer flow. Thankfully, numerous open-source tools abound that let you create a mock of a 3rd-party web service’s API.

My Development Setup

Fri, 07 Jun 2013 00:00:00 +0000

The open source software movement has come a long way and I’m particularly excited by the improvements made in automated code QA and how it empowers a small team (sometimes, a team of one) to develop software that, just a few years ago, required a large group of programmers and testers and long development times mostly comprising of manual code reviews.

How does automated code QA make this happen, you ask? Let me explain by walking you through my local development setup of my latest two open source projects, the ManaMana Framework, and RedStack.

Fun with iperf

Fri, 10 May 2013 00:00:00 +0000

Casual bandwidth testing is easy enough to measure. Just wget a large file from some source on the Internet and you’ll get your results. For more precise measurements, however, this will not do. Some examples of when wget is insufficient are:

You need to measure a machine’s download speed in relation to another machine in the network irrespective of the latter machine’s configuration/hardware (e.g. Apache config, disk speed)
You need to measure a machine’s upload speed in relation to another machine in the network.
You need to measure DL or UL rate for UDP

In these and other cases requiring precise measurement, iperf does the job very well. In my case, I needed to test a physical machine’s ability to throttle the bandwidth of the VMs it hosts. So our set up was to use iperf between a VM in the machine and another host in the network (not a VM in the physical machine, and not the physical machine).

Adventures With Norm

Tue, 26 Feb 2013 00:00:00 +0000

It’s been almost a year since I dived more deeply into automated requirements verification/QA while attempting to apply my new knowledge through a number of Norm prototypes. It’s been slow going, but very interesting thanks to a plethora of useful resources including James Bach’s site and This Freaking Awesome Book.

My biggest takeaway from this so far is that it’s just not feasible to automate the verification of each and every requirement. No doubt it’s possible, but there will be times when the benefits are overshadowed by the costs of automation. There are various reasons why this is so. Sometimes it’s because there are some discrepancies that are hard to describe in writing, but you know it when you see it. In other times, it may be because you just don’t know what the output will look like (e.g. a chart that displays some non-deterministic trend). There are many other reasons but that’s not to say that automation has absolutely no place in those examples. For instance, we could automated the setup and teardown steps of a test case while leaving the body of said test case (or even just the verification step) to the human. And so this is what gave me new ideas that I might add to my next prototype:

Writing a Compiler (Part 1)

Thu, 26 Jul 2012 00:00:00 +0000

This is the first in a series of posts where I will talk about how I wrote compilers for two languages that I designed. These languages, namely “BA Speak” and “QA Speak”, are external domain-specific languages (DSL). They are DSLs because they have a very specific usage and are not general purpose languages. They are external because they are not constrained by the syntax rules of a host language.

About BA Speak and QA Speak

BA Speak and QA Speak are DSLs that I created for a prototype framework I call Norm Framework. You can see the various prototypes of this framework here. Norm was inspired by Cucumber and the Robot Framework which are automated acceptance test frameworks used for verifying a piece of software against a set of specifications.

First impression of Python

Thu, 28 Jun 2012 00:00:00 +0000

I’ve started learning about Python during my free time and one of the things that bothers me is the fact that I have to use the len() function on an object instead of just calling a length method on that object. I’ve read the discussions online about this (e.g. 1, 2, 3) and my attempt to summarize the justifications are as follows:

len() has better clarity of intent
’length’ has a different meaning for different objects
That’s how it’s always been from the beginning

I’m not convinced at all by reason #1. Compare the following:

How We Use Cucumber, The Sequel

Thu, 24 May 2012 00:00:00 +0000

In a previous post, I explained how we used Cucumber. Now, after about a month of using the recommended style, we’ve noticed some ‘usability problems’ and introduced our own styles.

Readability of stepdefs

The following is how our steps used to be defined:

Then /^I can disable notifications$/i do
 Notifier.fill_in 'username', 'jdoe'
 Notifier.fill_in 'password', '123qwe'
 Notifier.click 'login'
 Notifier.click 'disable'

 Notifier.should_be_disabled
end

It’s nice, clean, and readable…for a programmer. However, stepdefs are not exclusive to programmers: testers also need to be able to review them to know if the test is thorough or accurate. We have smart testers who know how to find edge cases or break an application, but they don’t know how to code. Things like dot notation and remembering to enclose a string with quotations are alien to them. They already understand English though and since we found that it’s easy to modify the above code to read like English, we thought it would be better to write stepdefs that the Testers can use within stepdefs. One example of such a stepdef is:

On Cucumber's Opinionatedness

Sat, 21 Apr 2012 00:00:00 +0000

In a previous post, I said that Cucumber’s flexibility is both a boon and a curse to teams who are just getting to know it. I said this because it’s so easy to get started with the Gherkin syntax that users get a little too excited and start to tinker with it in their own way which doesn’t always produce the right results. It’s just like a game of Othello: very easy to learn, but hard to master. One has to understand the principles behind it to win.

How We Use Cucumber

Thu, 19 Apr 2012 00:00:00 +0000

Lately I’ve been very involved in the conversion of our product requirements and test scripts into executable documentation via Cucumber. We initially ran into problems with deciding what goes into the feature files and what goes into the step definitions. More than once we have had to stop and ask ourselves if the feature files are too imperative or too high level. There was a lot of back and forth until we settled on the following style.

We've Moved

Tue, 13 Mar 2012 00:00:00 +0000

This site is now powered by Github Pages and Jekyll.

All about the ECU

Wed, 29 Feb 2012 17:56:00 +0000

What is an ECU?

ECU means “EC2 Compute Unit”

OK, but what does it stand for?

First, let me explain what a VM is. A VM is defined by its:

Processing power
RAM, and
Local storage

What about ECU?

Well, an ECU is just a unit of measurement for a VM’s processing power and has nothing to do with RAM, or storage.

Cool, but what does an ECU really mean??

OK, have a seat.

Using GCC when Xcode 4.2 or 4.3 is installed

Wed, 22 Feb 2012 10:35:00 +0000

I was having problems compiling gems from the terminal in my Mac last week. After further investigation and some Googling, I found out that it was because Xcode 4.3 now uses the LLVM-based gcc and not all native gems support it. I then installed the Command Line Tools for Xcode hoping that would solve the problem. Unfortunately, it didn’t. At least not completely.

After some more digging around, I came to realize that /usr/bin/gcc, which is what gets used by default, is actually symlinked to /usr/bin/llvm-gcc-4.2. I tried setting CC=/usr/bin/gcc-4.2 whenever I installed gems, but somehow that gets lost along the way. Some suggested that I downgrade to Xcode 4.1 or even uninstall Xcode altogether. However, in my case, I use Xcode every now and then so that was not an option.

Trouble accessing Google from the Philippines?

Thu, 16 Feb 2012 12:33:00 +0000

I’m using Globe Tattoo Home DSL and I’ve had to deal with this problem for a number of days and tried different solutions to no avail, until an officemate of mine suggested that it might be a DNS peering problem between Globe and Google. He then told me to set my primary and secondary DNS to 4.2.2.2 and 4.2.2.1 respectively and voila!

As another officemate once said about Internet connectivity issues: it’s always a DNS problem.

Objective C

Thu, 29 Dec 2011 11:03:00 +0000

Objective-C implements weak typing, blocks (aka anonymous functions or closures), key-value coding (aka accessing an object’s methods and properties as if the object was a dictionary), monkeypatching, etc. It’s a dynamic language with the power of C. It’s Ruby and Python’s uglier but more efficient sister. It has its own quirks that might put you off at the start, but the more I learn about it, the more I like it.

Jogging Without Gadgets

Thu, 22 Dec 2011 02:00:00 +0000

When I go out for my regular 10~12k jog, I don’t bring any mp3 player with me. In fact, I don’t carry any gadgets at all: the only items on me are my sleeveless shirt, jogging shorts, and running shoes. It wasn’t always this way. When I started out, I would fill my iPod to the brim with supposedly energizing music. But after a while, the music started to get old, so I’d find more music to replace the old ones. Repeat ad nauseum.

Theory of Constraints: Drum-Buffer-Rope

Mon, 12 Dec 2011 08:19:00 +0000

This is an old report I wrote back in 2005 as a result of simulating different ways to streamline the process in the machine fabrication shop I worked for back then.

Traditionally, the scheduling method used in a small manufacturing shop (let’s call it MMS) has been more of guesswork wherein job orders are issued as they come. The main problem with this scenario is that as more job orders enter the shop floor, the larger the work-in-progress (WIP) inventory gets. As a result, idle times for job orders will severely fluctuate contributing to the variability of delivery times for job orders. The bottom line is that supervisors and sales representatives will not have enough data (and confidence) to come up with an accurate completion time estimate for every job order. Often times, they will resort to padding up the completion dates and/or compromising less critical jobs just to avoid late deliveries.

Elegant Solutions

Tue, 06 Dec 2011 05:58:00 +0000

I like elegant solutions because they are easier to administer. In fact, I believe that a solution is truly elegant if you do not even need to actively administer it. One thing to note about elegance though is that it’s hard to achieve if you’re harried, if you’re out of time, trying to deal with too many things at time, etc. So it’s important to be aware of you current state of mind and be sure that you are focused. Otherwise, you’ll never achieve that elegant solution you have in mind.

Explaining Cloud Computing

Sat, 03 Dec 2011 03:46:00 +0000

Cloud computing is a hot topic these days and while nearly every IT professional knows what it is, the rest of the world is still unsure about it. In my current job, part of my responsibilities is to explain this concept in the simplest terms especially for non-technical people and I do this by saying that cloud computing simply changes how we acquire computers or software. The following two slides are usually part of my arsenal:

Rails, Sinatra, or Express (for Node)? Decisions, decisions.

Thu, 01 Dec 2011 12:32:00 +0000

I’ve been evaluating these three frameworks yesterday and while I have to say that the speed of Express is really going to be critical in the long run, I think it’s premature to use it at this point in time. What I need to do right now is get a prototype up and running quickly so as to keep myself motivated. Sinatra is really the same as Express (albeit a little slower and less of the asynchronous goodness) so I’ll have to avoid using it at this time.

Things That Are Smaller Than Apple, Inc.

Thu, 24 Nov 2011 09:28:00 +0000

This may not be a slideshow, but it certainly is a good example of how to present numbers. Notice how this infographic does away with fancy graphics and yet gets the point across. I’d recommend using this for any comparison data. Chart purists might have a beef with the fact that it uses areas instead of the more technically accurate bar charts, but I think that it’s really effective in getting the point across.

Drupal, WordPress, Joomla!

Mon, 21 Nov 2011 21:25:00 +0000

Today I will pick a slide that I like and suggest improvements to it.

I like the whimsical chart even though it’s not technically accurate because it sends the message across very clearly. Just one thing I’d change there is to move the CMS names from the legend at the bottom to immediately to the right of each plot. There’s also more than enough whitespace in this slide that could’ve been used to make the chart easier to read.

Redmine Dashboard

Sun, 20 Nov 2011 10:22:00 +0000

A little something I wrote while I was on a plane from Manila to LA a few months back. This is actually the earliest prototype of the mini-project I was talking about earlier. I used backbone.js and I really feel that it helped me speed up development. It took a little time to get my head wrapped around the collection concept but once I understood that, everything else made a lot of sense. Here’s a screenshot of how it looks like once you get it working:

Acromag

Fri, 18 Nov 2011 21:23:00 +0000

If you have to show a diagram or a workflow, always make sure the audience knows where to start looking and where to look next. This slide was very confusing and even had a number of elements that didn’t help explain what was going on.

Original Slide

My Rendition

Some notes: Apart from the fact that I have no idea where the process starts and ends from looking at the slide above, I also have no idea about monitoring diesel engines. So I didn’t bother fully replicating the information in my rendition. Key takeaways here though are:

A Buyer's Guide to Storage Virtualization

Thu, 17 Nov 2011 21:21:00 +0000

Bulletpoints can be useful sometimes, but great care must be taken when using them. You don’t want your slides to replace you, the speaker. Even worse is if you use your bullets as speaker notes. Use an index card for crying out loud!

Original Slide

My Rendition

Don’t hide behind bulletpoints. Make the audience focus on you and what you have to say. Practice your speech and clean it up so that you only have to say the most important things. Afterwards, share with them some takeaway notes that they can read on their own time.

FAQ generator

Thu, 17 Nov 2011 09:30:00 +0000

Here’s a quick and dirty ruby script I made to extract issues from a project in Redmine and output it to an HTML file. I originally wrote this to compile issues in a project named ‘FAQs’ where we internally discuss FAQs and their answers before publishing to the wild. Check it out at Github.

After looking at that code, you might ask why I used pandoc instead of plain old erb or some other Ruby-based templating gem. The reason is that I originally started out by outputting a PDF file but then requirements changed and I had to come up with something very quickly. Needless to say, my fingers are itching to clean up this code. Soon…

Radisys

Tue, 15 Nov 2011 21:17:00 +0000

This slide was presenting too much information at once so I broke it down into 3 different slides to highlight 3 very important things for the audience.

Original Slide

My Rendition (Broken down to 3 slides)

Original slides are here.

HDS Cloud Strategy

Wed, 09 Nov 2011 21:09:00 +0000

3D charts, really, are unecessary: it distorts the visual information making side by side comparison difficult. To make matters worse, the “3D-fication” of the bar chart changes one of the data points below such that its color doesn’t match what’s in the legend.

Original Slide

My Rendition

Original slides are here.