Posts on

Automating my personal AWS setup with Terraform

Tue, 11 Jun 2024 00:00:00 +0000

Introduction

Welcome! This is the start of a series of posts where I am going to document (and probably update) my personal AWS setup. I spun up my first Amazon EC2 instance back in 2010, and, since then, grown my setup to use 8 different AWS accounts, mostly automated with Terraform. This series will cover all choices & tradeoffs I made, and also document the Terraform code I use to set it all up. So grab a beverage of your choice while I turn coffee into code.

Migrating my Jekyll blog to Hugo with Gen-AI

Tue, 28 May 2024 00:00:00 +0000

I’ve been neglecting my blog, and decided it is time to start writing again. So like any good software developer, I don’t actually write a post, I start by tinkering with my blog setup. I initially ran it as a Jekyll generated static site on GitHub using GitHub Pages, and then as part of a live-stream, moved it to use AWS Amplify to host it in on of my many AWS accounts. Coming back to it a few years later, I still remember the headaches of dealing with different Ruby versions, getting rvm set up, and then only getting it going. The paging is currently not working, so I’ll be trying to move over to Hugo for this.

Remapping the Home and End keys on OSX

Thu, 09 Feb 2017 00:00:00 +0000

The Problem

As I was pasting something into Chrome this morning, I, once again, expected the Home key to take me to the start of the URL. It. Did. Not. So I decided that I won’t be able to change this ingrained behaviour and need to fix it. First Google hit led me to a post by Damian Guard that I will be pasting below to keep as notes for myself. Everything from here on is from his post.

Yak Shaving - Makefiles

Sun, 05 Feb 2017 00:00:00 +0000

The Problem

I woke up this morning wanting to start on a long post about Terraform and how I ended up with the structure shown in my repo. I recently reinstalled my Macbook using Netatalk for a timemachine , and as I fired up my vagrant image, it started to pull down the Ubuntu box first and then started configuring it. This is very useful, but it is slow. And it broke. I forgot to add -y to the apt-get when I initially created the install script:

Setting up a Timemachine server using Mesos/Marathon

Sun, 15 Jan 2017 00:00:00 +0000

I am currently unable to upgrade to the latest version of Slack due to this bug. That combined with my user profile confusion (originally I set up using a US account, then changed to an South African one) that causes requests to update to versions of iPhoto which aren’t available in my region made me decide that it is time for a reformat.

The Problem

To create a full backup to my raid’ed server, I need to set up netatalk. I started reading this post but decided that I didn’t want to pollute my server with all of those installs. Seeing that I already have Mesos and Marathon running, I went the docker route. I found this repo and proceeded to create the following Marathon config:

Securing AWS environments using role switching

Sat, 03 Sep 2016 00:00:00 +0000

I’ve been working with multiple AWS accounts for the last few months between various organisations. Logging into each one when I need to make a change quickly became tedious and slow. Each environment (dev, test, staging, production) has their own AWS account. The need to log in stems from taming the infrastructure with Terraform for systems that have been set up by hand and dealing with the discrepancies between them, so I tend to jump between dev and staging very often. Being in this is a very normal state of affairs as most people starting out with AWS haven’t worked with any infrastructure creation automation. I will create another post detailing how to start this taming process.

Getting Glusterd to start at boot on Centos

Sun, 12 Jun 2016 00:00:00 +0000

I’ve been working on a pair of Centos servers using GluserFS for a volume that is shared by various other servers. Each time the server reboots, I had to log in and manually start the service. Turns out this is due to the networking no yet being started when the glusterd service starts. I found this post with the solution:

Execute systemctl enable NetworkManager-wait-online
Add the following to /lib/systemd/system/crond.service under [Unit]:

Requires=network.target
After=syslog.target auditd.service systemd-user-sessions.service time-sync.target network.target mysqld.service

This will allow glusterd to be started after the network has come up.

Upgrading a Chef cookbook with Berkshelf

Thu, 18 Feb 2016 00:00:00 +0000

I just upgraded my chef-clients via the omnibus_updater cookbook when things started breaking:

192.168.5.5 [2016-02-18T21:41:13+02:00] WARN: Current apt_package[apt-transport-https]: /var/chef/cache/cookbooks/datadog/recipes/repository.rb:24:in `from_file'
192.168.5.5
192.168.5.5 ================================================================================
192.168.5.5 Recipe Compile Error in /var/chef/cache/cookbooks/chef-wrapper-omnibus-updater/recipes/default.rb
192.168.5.5 ================================================================================
192.168.5.5
192.168.5.5 NameError
192.168.5.5 ---------
192.168.5.5 uninitialized constant Chef::REST
192.168.5.5
192.168.5.5 Cookbook Trace:
192.168.5.5 ---------------
192.168.5.5 /var/chef/cache/cookbooks/omnibus_updater/libraries/omnitrucker.rb:84:in `url'
192.168.5.5 /var/chef/cache/cookbooks/omnibus_updater/recipes/downloader.rb:27:in `from_file'
192.168.5.5 /var/chef/cache/cookbooks/omnibus_updater/recipes/default.rb:25:in `from_file'
192.168.5.5 /var/chef/cache/cookbooks/chef-wrapper-omnibus-updater/recipes/default.rb:27:in `from_file'
192.168.5.5
192.168.5.5 Relevant File Content:
192.168.5.5 ----------------------
192.168.5.5 /var/chef/cache/cookbooks/omnibus_updater/libraries/omnitrucker.rb:
192.168.5.5
192.168.5.5 77: if(url_or_node.is_a?(Chef::Node))
192.168.5.5 78: url = build_url(url_or_node)
192.168.5.5 79: node = url_or_node
192.168.5.5 80: else
192.168.5.5 81: url = url_or_node
192.168.5.5 82: raise "Node instance is required for Omnitruck.url!" unless node
192.168.5.5 83: end
192.168.5.5 84>> request = Chef::REST::RESTRequest.new(:head, URI.parse(url), nil)
192.168.5.5 85: result = request.call
192.168.5.5 86: if(result.kind_of?(Net::HTTPRedirection))
192.168.5.5 87: result['location']
192.168.5.5 88: end
192.168.5.5 89: end
192.168.5.5 90:
192.168.5.5 91: end
192.168.5.5 92: end
192.168.5.5 93:

Googling for uninitialized constant Chef::REST only yielded some results from March 2013, so I decided to upgrade the omnibus_updater cookbook from 1.0.4. to 1.0.6 by editing metadata.rb. I bumped the dependency version as well as my wrapper cookbook version, but when I ran berks install, I got the following:

Parameterizing Web.config

Sat, 06 Feb 2016 00:00:00 +0000

Most people would have experienced the issue of setting values in web.config for a project on different environments, i.e. the connection string for the database. My first attempt at resolving this was to simply create multiple configurations and build the appropriate one per environment. This has multiple issues: you are including sensitive information in your build artifact, creating different builds for the same version (to allow different values) and tightly coupling your build process to your environment values. This will require a rebuild if your connection string changes, which could be problematic if you are unable to build a specific version easily. But it won’t be the exact same version as you would need to commit the new config value, re-tag and finally rebuild the solution. Sounds like too much effort for a simple change.

NTP on AWS

Wed, 16 Dec 2015 00:00:00 +0000

Ran into an issue where a Linux instance running on AWS in a private subnet was not updating the system time via NTP. First check was for the config file, but it had a list of servers, both inside and 1 outside AWS:

server 0.amazon.pool.ntp.org
server 0.us.pool.ntp.org
server 1.amazon.pool.ntp.org
server 2.amazon.pool.ntp.org

From this post (I would like to link to it, but it has been 8.5 years since I wrote this, and only found the missing link today on 2024/05/29) I tried both ntpdate and ntpdate-debian with the following results:

Using the Git tag as a version number via PSake

Sat, 04 Apr 2015 00:00:00 +0000

I have been spending most of my time splitting up a monolithic system into separately deployable NuGet packages. The system is monolithic in the sense that there are multiple domains that are deployed as independent WCF services, but all reference the same set of base projects. This means that all the source sits in the same repository. The first step has been to move these common libraries into NuGet packages served from a local network folder and referencing them in each of these solutions as a package rather than project.

External MSBuild for WebApi in VS 2013 fails

Mon, 30 Mar 2015 00:00:00 +0000

Earlier today, I ran into the following error when trying to build a new WebApi solution using the command-line and MSBuild:

This is odd as v11 is Visual Studio 2012 and don’t even have 2012 installed. The issue manifests when you try to build the solution with: msbuild /t:Clean;Rebuild /v:q /p:Configuration=$projectConfig $source_dir\$projectName.sln /p:Platform="Any CPU"

The cause and solution is discussed on Stack Overflow - TL;DR version is this is due to adding the feature for older versions of Visual Studio, but then not setting the correct version number. To fix the issues, remove the following lines from your csproj file:

Git checkout error for public repo

Fri, 27 Feb 2015 00:00:00 +0000

One of those djissis duh moments - trying to check out a public git repo, but failing due to authentication …

git clone git@github.com:cobusbernard/Scripts.git
Cloning into 'Scripts'...
Warning: Permanently added the RSA host key for IP address '192.30.252.131' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

So I checked my ssh setup / keys, but nothing in there looks dodgy:

Replacing mdadm drive

Thu, 19 Feb 2015 00:00:00 +0000

Update 2016-01-09: In the previous version, I never added the command for creating the array and since then I needed it for a new server. When the array starts rebuilding, it will look like there is a failed drive: this is normal. When it builds from scratch, it is actually doing the rebuild process and will indicate a failed drive until the array is rebuilt.

# mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1

# cat /proc/mdstat
md0 : active raid5 sdd1[3] sdc1[1] sdb1[0]
 4294702080 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
 [====>................] recovery = 20.1% (433131624/2147351040) finish=420.1min speed=68006K/sec

As you can see from the above, this isn’t very fast. I found this article with some tips on increasing the speed. The one that worked for me was increasing the stripe-cache_size (tip 3):

Moving DNS to Amazon Route 53, Dynamic Updates

Tue, 17 Feb 2015 00:00:00 +0000

In my previous post, I set up OpenVPN on my home network and everything was awesome. Until this morning: I could not connect to my VPN. I had forgotten to set up some kind of dynamic DNS updater for it. That should be easy enough, I had previously done this using DynDNS. Only problem was that the service is no longer free. This shouldn’t be too much of a problem as I have a couple of my own domains - yes, I will one day still get round to finishing my pet project ‘Tinkle Tones’ ;)

Home VPN with OpenVPN

Sat, 14 Feb 2015 00:00:00 +0000

Having just set up an external VPN to pretend to be in the US, I wanted one to get into my home network as well when on the road. I ran into the issue while in the USA recently where I was unable to purchase Steam games as their servers picked up that I was in the States, but my payment method was South African. Never thought I would need to spoof my own IP Address to pretend to be in South Africa.

Personal VPN - OpenVPN in Docker on DigitalOcean

Fri, 13 Feb 2015 00:00:00 +0000

I was playing around on a random site today, when I received the message This content is not available in your country yet. As a South African, we run into this a lot as our online presence it not that large and most international companies do not feel it is financially viable to run their services here. So I decided it is time to get a VPN going to avoid this. I decided to use a Digital Ocean Droplet for the task - costing $5 a month is cheap enough. The smallest one has 512MB Ram, 20GB SSD, 1 core and 1TB of transfer - this will be plenty to get the VPN up and possibly some other services. Signup was very quick and I was happy to see that they support 2 factor auth via Google Authenticator.

Converting Putty keys to OpenSSH

Sun, 08 Feb 2015 00:00:00 +0000

I was playing with Docker yesterday and needed to convert a key that I had been using in Windows with Putty for a while. First attempt with puttygen didn’t work at all, then I found this Stack Overflow post, copying the details here so I know where to find it next time :)

To install the required tools, use

Ubuntu: sudo apt-get install putty-tools
OSX: brew install putty

After the required tools have been installed, use the following 2 commands to extract the private and public portion of the key:

Basic Jekyll GitHub Blog

Mon, 26 Jan 2015 00:00:00 +0000

So I finally got round to sorting out the tech blog. I chose to use Jekyll rather than WordPress as I didn’t want to deal with constant updates, security vulnerabilities and backing up of the site. Jekyll generates a static site based on the posts created, but still has the advantages of a CMS like tags and categories. Posts are written using Markdown in a plain text file, making it very easy to do the formatting. You are able to generate the site locally before publishing / pushing it elsewhere. This is where one of the other benefits come in: GitHub Pages. You are able to create a repository with the format <GitHub Username>.github.io - in my case, that is cobusbernard.github.io. GitHub itself uses Jekyll, and by creating this repository, it automatically creates the Jekyll site for you. No backup is required as your site is already under version control in GitHub!

Linux SSH rate limiting

Mon, 28 Jan 2013 00:00:00 +0000

Everyone who has an SSH port open to the world knows the amount of brute force attempts you will get. It doesn’t matter if you only accept keys, the script kiddies will still try. Easiest fix for this is rate limiting: you can only attempt to login 3 times per 10 minutes. This does not include successful logins, only failed ones. To do this, use the following IPTables commands:

sudo iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH --rsource
sudo iptables -A INPUT -m recent --update --seconds 600 --hitcount 4 --rttl --name SSH --rsource -j DROP
sudo iptables-save > /etc/iptables_rules

Add the following to /etc/rc.local before the ‘exit':

Using SSH Config

Thu, 12 Apr 2012 00:00:00 +0000

If you use SSH on Linux a lot, you will know that having to specify the key each time you want to access a server is a pain. You really don’t want to have a single key for all servers as you would need to revoke it globally in the event that it was compromised. The ssh config file allows you to specify multiple values to use for specific hosts:

To edit the ssh config, open it in vim (or your favourite editor):

Chrome as default browser in Xubuntu

Tue, 10 Apr 2012 00:00:00 +0000

If you need to set Chrome as the default browser in Xubuntu, run the following from your terminal:

gconftool-2 --type string -s /desktop/gnome/url-handlers/http/command "google-chrome %s"

Install Java JDK using terminal

Tue, 27 Mar 2012 00:00:00 +0000

To install the Java JDK using only the terminal, follow the following steps:

Download the JDK from:

http://www.oracle.com/technetwork/java/javase/downloads/java-se-jdk-7-download-432154.html

Then run the following commands (change the filename as required):

tar -xvf jdk-7-linux-i586.tar.gz
sudo mv ./jdk1.7.0 /usr/lib/jvm/jdk1.7.0
sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.7.0/jre/bin/java 1
sudo update-alternatives --config java
sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk1.7.0/bin/javac 1
sudo update-alternatives --config javac

To test that everything is working:

java -version

Should output:

java version "1.7.0"
Java(TM) SE Runtime Environment (build 1.7.0-b147)
Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)

javac -version

Should output:

Migrating repo from SVN to Git RPC error

Sat, 04 Feb 2012 00:00:00 +0000

While I was working on Dementium II HD, I wanted to migrate the repository from SVN to Git. It was being hosted on a machine in the one studio and served the external developers over a 10mbit ADSL connection. The upstream rate is ~400kbit, so imagine trying to checkout a 26GB repository. I ran into the following error when I tried to import the SVN repo into a Git one:

$ git push -v origin svn-merge-trunk:svn-trunk
Pushing to https://git-repo/git
Counting objects: 2840, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2788/2788), done.
error: RPC failed; result=22, HTTP code = 411
fatal: The remote end hung up unexpectedly
Writing objects: 100% (2840/2840), 3.07 MiB | 474 KiB/s, done.
Total 2840 (delta 2202), reused 0 (delta 0)
fatal: The remote end hung up unexpectedly

After a bit of Googling, I found a solution for this:

Ubuntu on Hyper-V

Mon, 25 Jul 2011 00:00:00 +0000

Getting Ubuntu to run on Hyper-V was a bit of an issue for Windows Server 2008, here are some things to look for. In the boot menu, add vga16fb.modeset=0 to the boot options by hitting F6 to disable framebuffer mode that is really slow under Hyper-V.

Do normal install, don’t worry about the red “can’t find network issue”.

Enable the Hyper-V modules:

echo "hv_vmbus" >> /etc/initramfs-tools/modules
echo "hv_storvsc" >> /etc/initramfs-tools/modules
echo "hv_blkvsc" >> /etc/initramfs-tools/modules
echo "hv_netvsc" >> /etc/initramfs-tools/modules
update-initramfs –u

Disable Framebuffer (otherwise the text screen scrolls by like a 9600 modem)