Introduction

Welcome! This is the start of a series of posts where I am going to document (and probably update) my personal AWS setup. I spun up my first Amazon EC2 instance back in 2010, and, since then, grown my setup to use 8 different AWS accounts, mostly automated with Terraform. This series will cover all choices & tradeoffs I made, and also document the Terraform code I use to set it all up. So grab a beverage of your choice while I turn coffee into code.

I’ve been neglecting my blog, and decided it is time to start writing again. So like any good software developer, I don’t actually write a post, I start by tinkering with my blog setup. I initially ran it as a Jekyll generated static site on GitHub using GitHub Pages, and then as part of a live-stream, moved it to use AWS Amplify to host it in on of my many AWS accounts. Coming back to it a few years later, I still remember the headaches of dealing with different Ruby versions, getting rvm set up, and then only getting it going. The paging is currently not working, so I’ll be trying to move over to Hugo for this.

The Problem

As I was pasting something into Chrome this morning, I, once again, expected the Home key to take me to the start of the URL. It. Did. Not. So I decided that I won’t be able to change this ingrained behaviour and need to fix it. First Google hit led me to a post by Damian Guard that I will be pasting below to keep as notes for myself. Everything from here on is from his post.

The Problem

I woke up this morning wanting to start on a long post about Terraform and how I ended up with the structure shown in my repo. I recently reinstalled my Macbook using Netatalk for a timemachine , and as I fired up my vagrant image, it started to pull down the Ubuntu box first and then started configuring it. This is very useful, but it is slow. And it broke. I forgot to add -y to the apt-get when I initially created the install script:

I am currently unable to upgrade to the latest version of Slack due to this bug. That combined with my user profile confusion (originally I set up using a US account, then changed to an South African one) that causes requests to update to versions of iPhoto which aren’t available in my region made me decide that it is time for a reformat.

The Problem

To create a full backup to my raid’ed server, I need to set up netatalk. I started reading this post but decided that I didn’t want to pollute my server with all of those installs. Seeing that I already have Mesos and Marathon running, I went the docker route. I found this repo and proceeded to create the following Marathon config:

I’ve been working with multiple AWS accounts for the last few months between various organisations. Logging into each one when I need to make a change quickly became tedious and slow. Each environment (dev, test, staging, production) has their own AWS account. The need to log in stems from taming the infrastructure with Terraform for systems that have been set up by hand and dealing with the discrepancies between them, so I tend to jump between dev and staging very often. Being in this is a very normal state of affairs as most people starting out with AWS haven’t worked with any infrastructure creation automation. I will create another post detailing how to start this taming process.

I’ve been working on a pair of Centos servers using GluserFS for a volume that is shared by various other servers. Each time the server reboots, I had to log in and manually start the service. Turns out this is due to the networking no yet being started when the glusterd service starts. I found this post with the solution:

• Execute systemctl enable NetworkManager-wait-online
• Add the following to /lib/systemd/system/crond.service under [Unit]:

Requires=network.target
After=syslog.target auditd.service systemd-user-sessions.service time-sync.target network.target mysqld.service

This will allow glusterd to be started after the network has come up.

I just upgraded my chef-clients via the omnibus_updater cookbook when things started breaking:

192.168.5.5 [2016-02-18T21:41:13+02:00] WARN: Current  apt_package[apt-transport-https]: /var/chef/cache/cookbooks/datadog/recipes/repository.rb:24:in `from_file'
192.168.5.5
192.168.5.5   ================================================================================
192.168.5.5   Recipe Compile Error in /var/chef/cache/cookbooks/chef-wrapper-omnibus-updater/recipes/default.rb
192.168.5.5   ================================================================================
192.168.5.5
192.168.5.5   NameError
192.168.5.5   ---------
192.168.5.5   uninitialized constant Chef::REST
192.168.5.5
192.168.5.5   Cookbook Trace:
192.168.5.5   ---------------
192.168.5.5     /var/chef/cache/cookbooks/omnibus_updater/libraries/omnitrucker.rb:84:in `url'
192.168.5.5     /var/chef/cache/cookbooks/omnibus_updater/recipes/downloader.rb:27:in `from_file'
192.168.5.5     /var/chef/cache/cookbooks/omnibus_updater/recipes/default.rb:25:in `from_file'
192.168.5.5     /var/chef/cache/cookbooks/chef-wrapper-omnibus-updater/recipes/default.rb:27:in `from_file'
192.168.5.5
192.168.5.5   Relevant File Content:
192.168.5.5   ----------------------
192.168.5.5   /var/chef/cache/cookbooks/omnibus_updater/libraries/omnitrucker.rb:
192.168.5.5
192.168.5.5    77:        if(url_or_node.is_a?(Chef::Node))
192.168.5.5    78:          url = build_url(url_or_node)
192.168.5.5    79:          node = url_or_node
192.168.5.5    80:        else
192.168.5.5    81:          url = url_or_node
192.168.5.5    82:          raise "Node instance is required for Omnitruck.url!" unless node
192.168.5.5    83:        end
192.168.5.5    84>>       request = Chef::REST::RESTRequest.new(:head, URI.parse(url), nil)
192.168.5.5    85:        result = request.call
192.168.5.5    86:        if(result.kind_of?(Net::HTTPRedirection))
192.168.5.5    87:          result['location']
192.168.5.5    88:        end
192.168.5.5    89:      end
192.168.5.5    90:
192.168.5.5    91:    end
192.168.5.5    92:  end
192.168.5.5    93:

Googling for uninitialized constant Chef::REST only yielded some results from March 2013, so I decided to upgrade the omnibus_updater cookbook from 1.0.4. to 1.0.6 by editing metadata.rb. I bumped the dependency version as well as my wrapper cookbook version, but when I ran berks install, I got the following:

Most people would have experienced the issue of setting values in web.config for a project on different environments, i.e. the connection string for the database. My first attempt at resolving this was to simply create multiple configurations and build the appropriate one per environment. This has multiple issues: you are including sensitive information in your build artifact, creating different builds for the same version (to allow different values) and tightly coupling your build process to your environment values. This will require a rebuild if your connection string changes, which could be problematic if you are unable to build a specific version easily. But it won’t be the exact same version as you would need to commit the new config value, re-tag and finally rebuild the solution. Sounds like too much effort for a simple change.

Ran into an issue where a Linux instance running on AWS in a private subnet was not updating the system time via NTP. First check was for the config file, but it had a list of servers, both inside and 1 outside AWS:

server 0.amazon.pool.ntp.org
server 0.us.pool.ntp.org
server 1.amazon.pool.ntp.org
server 2.amazon.pool.ntp.org

From this post (I would like to link to it, but it has been 8.5 years since I wrote this, and only found the missing link today on 2024/05/29) I tried both ntpdate and ntpdate-debian with the following results: