Pete Wilcock

Walking the talk of IR35 – by quitting a contract

Pete — 2021-02-22

Should you offer notice on a contract when it’s not compliant? I did, and I’ll tell you why.

Reading Time: 6 minutes

Winning the Work

A while back, I agreed to commence on a contract for a large Financial Services institution. After completing the ‘interview’ process (a format which, in my previous article on IR35, shouldn’t exist for B2B), I was sent a contract by the agency. I had this promptly reviewed for IR35 compliance and, as it always does, it came back with a couple of recommended changes. One of the offending clauses was a requirement for my business to notify the agency ‘by 9am’ if for any reason I wasn’t able to work that day. The language was deemed to be unduly controlling – in the event my business’ staff are out of action, I will of course notify my client as a professional courtesy. But to have this written as a strict requirement smacks too much of making sure you’re reporting in like an employee. It’s an example of confused writing that afflicts the average contract and unwittingly structures and refers to clauses as if the engagement is employer/employee and not, as it should be, business to business.

There was a significant delay going back and forth, as the agency’s legal advisers were pushing back on the minor changes, either claiming that they were redundant or outright accusing my own advisers of not understanding the wording properly. Eventually the agency decided to use a different contract template from their parent company which was compliant with being clearly outside-IR35 , and we went ahead. Because of all the toing and froing the desired commencement date was pushed slightly but I eventually turned up on the client site ready to get started….

Hitting the ground running?

The client was perhaps not best prepared for my arrival.

Many contractors will have had a similar experience where they arrive on site only for it to occur to the client for the first time that there’s a variety of system on-boarding, familiarisation, discussion on project objectives et al. that are needed to get started. Consequently the first few days are spent sat in limbo whilst you await turgid processes to be enacted and then grind their way through the system. In a proper B2B relationship this is something that should happen asynchronously ‘offline’ before you even commence. Since this is an expensive industry to engage a consultancy in, it’s terribly poor value for money to have your consultant turn up only to then allow them to spin their wheels to no productive end. This is exacerbated by the illogical insistence of the client to have you on-site continuously, even when it’s clear there’s nothing to require you to be physically present.

This doesn’t feel right

It was at this point the alarm bells really started to ring. I was expecting to have a significant discussion to clearly define what would be deliverable in the engagement’s duration (6 months), but this didn’t transpire. What actually happened is that I was given a brief overview of their system’s chequered history – hard to maintain, poor processes, extreme difficulty in deploying and upgrading – and I was then asked to attend to a couple of minor work items within their existing constraints.

This didn’t sound right to me, but I supposed this might be a tentative water-testing exercise to ensure I was capable of what my business had represented during ‘interview’. After knocking these off pretty easily, I was then in the awkward position of having to go back and ask for more. What do I do now? What do you want next? These are questions a contractor should never have to ask. A clearly defined project of work with objectives and milestones should allow you to work fairly autonomously whilst only providing the client with updates or asking for some clarifications. That also means you shouldn’t need to be on-site unless there’s a clear reason to be. Some senior purse-string types have an unfortunate tenancy to want to see ‘bums on seats’ to justify the costs, but I maintain that unless there is a clear requirement, the consulting business has the discretion as to whether they need to be there.

What didn’t help was that their system was entirely undocumented. No wiki pages or comments in code describing how any of their system worked – it all existed in the heads of two senior engineering leads. Whilst a lot of it can be worked out by an experienced engineer, albeit very slowly and with much frustration, there are often various areas of ‘magic’ that simply need explaining. This constant dependence again just felt too employee-esque – there wasn’t enough information available to do any independent working.

So I sat there, bored out of my mind, attending to a couple of rudimentary tweaks to their existing system whilst documenting other parts for my own future reference.

Breaking Point

After exactly 3 weeks, I felt I had no choice but to offer notice on the contract. There wasn’t a project here to do. I was sitting there, wasting time, attending to dribs and drabs within an existing and pretty poor system. A bad system isn’t a problem – quite the contrary, it represents a bursting seam of potential to be mined – with a formalised appraisal, project plan to remediate, and then execution and delivery. That’s what you engage a consultant contractor to do – not have them sit there like an employee doing whatever occurs to you on an entirely ad-hoc basis.

Since my contract was actually with the agency – I contacted them to offer notice. Amusingly they then asked if I’d told the client and I reminded them that, technically, they were the ones I had to offer notice to, and it was up to them to notify their own client. Which they then dutifully do, and exactly 10 minutes later the client asks for a meeting.

Hope for the future?

First of all – ouch, it felt awkward. Having to tell a perfectly nice bunch of people that you can’t work for them any more – but also that it’s nothing personal. There are very many ways this can turn into a very nasty and unpleasant conversation if not handled well.

Much to my relief, it was actually a really positive experience. After carefully explaining my concerns around the nature of the work, lack of specification, and the need for IR35 compliance, they completely conceded to all of it. They admitted that they often treat contractors like backfilled employees, and whilst they were somewhat aware of IR35 regulations they agreed they needed to be better, and that engagements should be properly scoped pieces of work. It was simply that they were under too much day-to-day pressure, and their need was too urgent to stop to try to work it out properly. They’d also not had any top-down guidance from management, but also seemingly felt like they didn’t have enough internal political punching power to make the structural changes required. In a word, they were between a rock and a hard place.

I felt sincerely sorry for them. They were genuinely nice guys, working hard, doing their best under pressure. I also felt like I was leaving them in the lurch when they clearly needed help – and yet everything I’d said was true, and to remain was just too much of a risk for my business. They asked if I’d consider re-engaging if they could get their act together and come up with a project specification – and I said I would. I also said that such an effort wouldn’t be wasted regardless, as the next contractor would have a much better time of it.

Why not just shut up and keep your head down?

A lot of contractors might take this attitude, and fair enough. There are a three factors that allowed me to take such a position, namely:

My business has cash reserves. Contractors usually refer to this as a ‘warchest’. It means that if I don’t have a contract right now I won’t die of hunger any time soon.
My industry is a busy one – at least for the moment. Endless demand for particular skills means there’s always someone knocking at the door wanting to know if I’m available. Quite the fortunate luxury not available to many people.
I’m annoyingly principled whilst being sensibly cautious. Sometimes you deliberately shoot yourself in the foot because it’s the right or safest thing to do.

Three simple things, and yet if I didn’t have the first two I seriously doubt I’d have the third. If I needed the work badly and contracts were few and far between, could I really afford to argue the toss with an agency or client that their terms of engagement weren’t strictly by the regulations? My fellow contractors have often told me that it’s not worth the martyrdom to quit a contract on the principle. Consistently late payments, poor treatment, horrendous working environment – sure, switch to a new contract as soon as possible. But because you’re trying to future-proof against possible non-compliance with IR35? Madness.

Not all contractors are equal

It does make me wonder – how many ‘businesses’ are falling foul of the rules not because they’re willfully non-compliant, but because they have no choice? The responses to my last article raised an important point that I hadn’t addressed – some individuals are being compelled to register and work through a limited company. Clients and agencies are often demanding it as part of the contract, and individuals want the boxes ticked and so they comply. But unlike me, they don’t want to be a business, and probably don’t think of themselves as such. They’re in prime position to fall foul of the rules, probably have very little awareness of that fact, and will not be well-armed if and when the taxman comes knocking.

https://www.contractoruk.com/forums/ir35-reform/129193-forced-into-contracting-read.html

Not every industry has a specialist-skills shortage, and very many people can’t be precious or demanding about their terms. Yet at the current time Her Majesty’s finest will come down on these people with a sledgehammer if given the merest hint that there’s avoided tax to be clawed back, legitimately or not.

It’s a sorry state of affairs where, in classic style, the Treasury’s response to a relatively minor issue is to nuke it from orbit, because it’s the only way to be sure. The collateral damage to legitimate industry has already been felt in the public sector, but it’s at least hoped that the private sector will be much less-inclined to bend the knee, whilst also proving itself more adept to ensuring technical and genuine compliance with the rules, no matter how pedantic those rules are.

—-

Related: IR35 is easily avoided, but it’s time to get with the programme

Recommended Daily Allowance (RDA) for Electrolytes while fasting

Pete — 2019-12-27

If you’re on a keto diet, intermittant-fasting, or water-fasting, you might be supplementing with electrolytes to alleviate Keto-flu, heart palpitations, headaches, muscle-shakes, and a variety of other ailments that can happen when you’re restricting your food intake.

But what is the reference intake, or recommended daily allowance, for these electrolytes?

The list below breaks down the essentials.

Sodium Chloride (Salt)

No more than 6g of salt (2.4g of sodium) a day

Potassium

3500mg a day for adults

Magnesium

300mg a day for men (19 to 64 years)
270mg a day for women (19 to 64 years)

*Figures based on UK NHS recommendations.

What should I take for electrolyte supplementation whilst fasting?

This is not official medical advice (consult your Doctor, as they say). but my personal recommendation is MyProtein Electrolyte tablets. They’re easy to swallow, and are very simple and effective. Rather than trying to work in the electrolytes in some other way or just chugging salt directly (yuk)

If you work out and sweat a lot, 2 of these tablets a day will help maintain your electrolytes under a normal diet. Whilst fasting or on keto, I’d cautiously suggest 3-5 tablets spaced throughout the day. DO be very careful with how many you take and start small – excessive consumption of electrolytes can have many negative side-effects. See this LiveStrong article for details.

AWS Lambda: “ModuleNotFoundError No Module named _foo or foo” Solution

Pete — 2019-09-23

There are two reasons you might encounter this issue on AWS Lambda.

You’re attempting to import a library that isn’t available by default in the Lambda Runtime environment.

In which case you need to include the dependencies in the /python directory of your function package or included Lambda layer.

You’ve already done that but it’s still not working? Does it seem like the function is just denying reality?! Read on…

You’ve included libraries for the wrong version of Python

Depending on the module, python might insist that you provide the exact version that matches the Lambda runtime that you’re using in AWS. This means if you’re using Python 3.7 in AWS, but did a pip install in your build environment on Python 3.6, it might not work.

Easiest solution would be to ensure you have a build environment that can use pip3.7 install -target /targetdir -r requirements.txt to ensure you’ve got matching versions.

I’ve also observed what can only be described as OS-specific weirdness when trying to do this in MacOS with ostensibly the correct Python version. For safety I ended up performing my packaging builds using the Official Python 3.7 docker image, which worked perfectly.

Using Poppler/pdftotext and other custom binaries on AWS Lambda

Pete — 2019-09-20

A recurring question with AWS Lambda functions is how to properly use third-party binary executables in the Lambda execution environment. A lot of people like to use precompiled binaries but often they’re not available and the only option is to compile the package you want to use from source.

Before I get into detail I’ll break it down into the 3 required steps:

Compile the desired package from source in the Lambda Runtime kernel
Package up the executables and dependent shared libraries into a zip
Include the package within your Lambda Function/Layer and modify functions settings to find it

Each of these I found to be pretty poorly documented and spent a mind-numbing afternoon working it out from snippets I’d found elsewhere. In this example I’m talking about poppler-utils, a required package for many PDF and OCR conversion tools, but the concept is broadly similar when trying to bundle any third-party binary into a Lambda function. You can’t just do a yum or apt-get in Lambda, after all.

Compile the desired package from source in the Lamba Runtime Kernel

Quick lesson: Lambda natively supports a variety of different language runtimes: Python 2.7, 3.6, 3.7, Node.js 8.10, 10, Ruby 2.5, Java 8, Go 1.x, .NET Core 1.0 and 2.1.

Each one is based on a specific Amazon Linux AMI and Kernel version, and if you want to compile your own binaries from source, you have to do it in the matching AMI to ensure it works correctly. Other guides I’ve seen recommend that you spin up an EC2 instance and do it that way, but it’s current year and Docker can do the same thing but locally and for free!

So we head over to the Official Dockerhub for Amazon Linux and pull and appropriate image. At the current time Python is based on amzn-ami-hvm-2018.03.0.20181129-x86_64-gp2 which annoyingly enough did not have a precise docker image available, so I opted for the general tag of amazonlinux:2018.03

From your preferred local command line terminal running docker, do the following:

docker pull amazonlinux:2018.03
docker run --name amzn -d -t amazonlinux:2018.03 cat
docker exec -it amzn bash

A standard trick to hold a container open is to run it in daemon mode with cat at the entry command. The container will now run until we decide to kill it, and we get started by hopping inside with a docker exec command for bash.

Poppler has a dependency on OpenJPEG, which must be installed first, and all of the compilation tools we need are missing and must also be certain versions, so we’ve got to bootstrap a bunch of packages within our docker image:

yum -y install openjpeg-devel libjpeg-devel fontconfig-devel libtiff-devel libpng-devel xz gcc gcc-c++ epel-release zip cmake3

cd /root
curl -L https://github.com/uclouvain/openjpeg/archive/v2.3.1/openjpeg-2.3.1.tar.gz | tar xvz
cd openjpeg-2.3.1
mkdir -v build &&
cd       build &&
cmake3 -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_INSTALL_PREFIX=/usr \
      -DBUILD_STATIC_LIBS=OFF .. &&
make && make install && cd ../../

That’s OpenJPEG installed. Next for Poppler itself:

curl https://poppler.freedesktop.org/poppler-0.59.0.tar.xz | tar xJv
cd poppler-0.59.0/ && ./configure --enable-static --enable-build-type=release && make && make install

And voila, we’ve manually compiled Poppler in an Amazon Linux docker image. Now, what the heck do we do with it? This is the step that I’ve found so very badly explained elsewhere.

Package up the executables and dependent shared libraries into a zip

This is the trickiest part in a way because whilst you can probably follow steps to compile what you want in a local filesystem, what then? How do we extract what we need to get into Lambda? What do we even need?! Sadly this will vary from package to package so I’ve got no fixed answer (but read on for what to do with Poppler), but as a guide the make and make install will both create the binaries that you’re after (perhaps in /usr/bin or /usr/local/bin as well as a some shared libraries in/usr/lib64 /lib64 or /lib that these binaries will rely on.

In the case of Poppler the compiled utility binaries exist (in this example) in in /root/poppler-0.59.0/utils and include things like pdfimages, pdffonts, pdftohtml, pdftotext, and pdftoppm. Chances are you’re reading this because you need one of these.

The shared libraries you should probably include are noted below, because now we’re going to package everything we need up.

Still within our docker image, do the following:

cd /root
mkdir -p package/lib
mkdir -p package/bin
cp /usr/lib64/{libopenjpeg.so.2,libtiff.so.5,libjpeg.so.62,libpng12.so.0,libfreetype.so.6,libfontconfig.so.1,libjbig.so.2.0} /root/package/lib
cp /lib64/{libz.so.1,libexpat.so.1} /root/package/lib
cp poppler-0.59.0/poppler/.libs/libpoppler.so.70 /root/package/lib
cp poppler-0.59.0/utils/.libs/{pdftotext,pdfinfo,pdfseparate} /root/package/bin

Modify the last line there to include any of the poppler-utils that your Lambda function needs. Now to zip it all up:

cd package
zip -r9 ../package.zip *

Now sitting in /root/package.zip is everything we need! But wait, we’re still working within our fragile and volatile Docker container here – as we haven’t mounted any shared volumes – everything we’ve done inside the container will be lost the moment we kill it. That would be bad! We can safely run exit to exit the container and go back to our local shell, because the container is still running from our original cat command that started it.

Now we’re back into our local shell, we can copy the file from within the container to our local filesystem.

docker cp amzn:/root/package.zip package.zip

Now we’ve got our local file, package.zip, that has everything we need. Now we need to put this in Lambda!

Include the package within your Lambda Function/Layer and modify functions settings to find it.

Firstly, I’d strongly recommend using a Lambda Layer for this. Layers are what they sound like – they’re easily included packages of dependencies that you can reuse with your Lambda functions without having to include everything with each one individually.

So, all we need to do is Create a Lambda layer and upload our newly created zip as its source.

The mild headache in Lambda is knowing what the default environment variables are for paths and what those paths look like in the execution environment. If you upload a package into a Lambda layer, what paths do they convert to?

The root of your function itself runs in /var/task

Layers appear to be overlaid into /opt

So if your Lambda Layer zip contains the folders /bin and /lib they’ll be translated to /opt/bin and /opt/lib in the lambda execution environment.

If you were to include these same folders in your function itself, rather than a layer, they’d be places in /var/task/bin and /var/task/lib

This is important to know as the two important environment variables have the following defaults:

PATH:/usr/local/bin:/usr/bin/:/bin:/opt/bin
LD_LIBRARY_PATH: /lib64:/usr/lib64:$LAMBDA_RUNTIME_DIR:$LAMBDA_RUNTIME_DIR/lib:$LAMBDA_TASK_ROOT:$LAMBDA_TASK_ROOT/lib:/opt/lib

So without doing anything else, Lambda automatically picks up the paths from our Layer and will find them in the execution of your function. NOTE if you were to include a bin directory with your main function, this would not be in the PATH variable by default and wouldn’t get picked up!

Finally, it’s important to note that if you want to place libraries or binaries in other paths, you can do so as long as you overwrite the environment variables in the Lambda function itself.

I’ve seen some crazy workarounds implemented for this on the internet. People copying binaries into Lambda’s /tmp directory and then fudging their functions to manually look for the executable there – but in general this isn’t necessary if the function is properly configured and you’re not doing something strange like implementing a custom runtime!

My experience with the new “remote” AWS Certified Cloud Practitioner Exam

Pete — 2019-08-30

AWS announced on the 28th August 2019 that the entry-level certification – AWS Certified Cloud Practitioner – was now available to be taken remotely from the comfort of your home or office.

This certification is comparatively recent, having been released in December 2017, and would be redundant to anyone who had already sat an Associate-level certification or above. For this reason I hadn’t bothered to trek to an exam centre to sit it, but this new release gave me the chance to lazily knock it off the list in the ongoing quest to “catch ’em all”. Here’s my experience:

The first thing to note is that the announcement doesn’t immediately make it clear how to begin the process, and on logging into my Certification Account wasn’t much help. After some poking around you discover that is is only the Pearson Vue provider who offer this remote exam and NOT PSI. To register, select the “Schedule New Exam” button, and select “Schedule with Pearson VUE” next to the AWS Certified Cloud Practitioner exam. On the Select Exam Delivery Option page, choose the option “At my home or office.” Online proctoring is not available in China, Japan, and South Korea.

Whilst the exam purports to be available 24×7, this is still constrained by the availability of online proctors, who invigilate (monitor you) during the exam. In my case, I logged on at 10am in the morning and the first spot available to me was 8.45pm that same evening – so I booked it!

Set-up

Now the fun begins! Pearson Vue deliver the exam through a piece of software called OnVUE. Before booking the exam they encourage you to download and pre-test the software to ensure it will work on your system, and that you’re able to comply with the various requirements of sitting the exam remotely. Aside for some technical specifications of running the software, here’s what you’ll need:

A laptop or desktop computer
Webcam
Microphone
Mobile phone
Reliable internet connection

The pre-test of the software is identical to the steps you must undertake in the 15 minutes before your allocated start time for the exam. The steps are as follows:

Enter the access code provided by PearsonVUE when downloading the software.
Enter your mobile telephone number to receive a text message with a link to ‘ProctorCam’ – a service which will ask you to take pictures of:
- Your face
- The front and back of your government-issued ID
- Your desk when standing from the front, from the back, to the left and to the right. Essentially a 360 degree view of the room you’ll sit the exam in.

After this is complete, the test will check your system to ensure there isn’t any forbidden software running (in my case Malwarebytes had to be turned off) as well as closing any browser windows to help ensure you can’t cheat. In the pre-test you’re then asked a single test question and the test is complete.

The Exam

In the real exam, it instead gets serious – your webcam is turned on with a notice to inform you that it’s recording and your provided details are being checked by a proctor. In my case, a proctor connected after a couple of minutes and spoke to me. They were very polite but very firm in checking the area around my desk to ensure I had no paper, no phone, no stationary, no clutter – basically nothing at all. I had to pan my camera around (after complying by throwing all of my usual desk clutter onto the floor) to satisfy the proctor. I also had to unplug my scanner, and be seen to physically disconnect the cable of my second monitor. I had to remove my wristwatch, and show my wrists and hands to the webcam to demonstrate I had nothing written on them. Additionally I had to take my headphones off during the exam but I was allowed to put them back on should I need to speak to a proctor.

Finally, I was allowed to sit back down and attempt to perfectly align my face in the centre of my camera in order for the exam to be released to me. This took a couple of minutes as my face wasn’t quite perfectly set! Eventually the exam got going, and from this perspective it was like any other exam sat in a test centre. Multiple choice answers to standard questions whilst selecting ‘Next’ to move on to the next one. 100 minutes allowed for the exam and 65 questions total.

However after around 5 questions in, a different proctor suddenly connected to my session and asked me to once again confirm that I had no paper or notes nearby, and also to ask if I was able to physically move the second monitor which I’d earlier disconnected. Once again I got up and panned my camera around, and was able to move my second monitor out of the way to their satisfaction, and then I was allowed to get back on with the exam – but only after being reminded that I wasn’t allowed to move or even read the questions out to myself aloud.

So, feeling enormously self-conscious the entire time, I sat as rigidly as possible without moving my head from dead-centre of my camera – fearful of even panning my eyes down to the ‘Next’ button at the bottom of the screen, just in case it was thought I was looking down at some secretly-hidden notes, and completed the exam as quickly as possible.

Without being too boastful, I completed the 65 questions in less than 25 minutes, submitted the exam without reviewing any answers (which is an option), answered a few standard survey questions about my experience, and was informed that I’d passed – the score and results apparently being uploaded to my certification account in a few days. This is a curiosity of AWS exams recently that the score and result aren’t delivered immediately (which it used to be) as I can’t think of any technical reason this would be the case.

Conclusion

But there you have it – an exam completed remotely and I didn’t have to leave the house. But was this more convenient than having to go to an official proctored test centre? Well… not really if I’m being honest. The requirement of having to take so many pictures, clearing out the whole area of where you’re sitting, and feeling under the spotlight much more so than usual in an exam rather negated the convenience of being able to do it at home.

Of course, for certifications to have any merit, AWS must do all it can to try to eliminate cheating and stopping impersonators sitting the exam for you. So all of the measures described above are perfectly reasonable and were as effective as they could be. But could I have still cheated somehow? Sure… even being on webcam wouldn’t stop a determined cheat from putting some kind of notes somewhere – or by cleverly manipulating the testing area to have someone offer you advice. It’s for this reason I suspect we’ll only see this remote option offered for the lowest level of certification. Cheating here would be a lot of effort for a relatively modest qualification – so I believe this offering is a nice balance between convenience and credible risk of impropriety by the candidate.

More importantly, I think this is designed to encourage both technicians and managers to get an easier initial footing on the AWS certification ladder – after which graduating up to the Associate, Professional, and Speciality exams in official test centres will be a lot easier.

Speaking for myself – if I was sitting one of the 3 hour Professional exams I’d find it a lot less stressful to calmly sit in a real exam centre than worry about whether the online proctor thinks my home desk’s cup-coaster is a secretly-encoded cheating aide!

If you’re studying for the Cloud Practitioner Certification, I will, as always, highly recommend A Cloud Guru for their excellent standard of teaching and preparation for the exam. Check out their AWS Certified Cloud Practitioner 2019 course here.

Fixing “com.amazon.coral.service.InternalFailure” when using ACM

Pete — 2019-08-13

If you’re using the AWS ACM console to create a certificate, and at the final stage you get this annoying and useless com.amazon.coral.service.InternalFailure, I’ve got the solution for you.

You’re probably working in an AWS Organization with a Service Control Policy (SCP) or a restricted IAM user where you’ve been given the acm:* permissions thinking this is enough. Sadly not, the solution is you additionally need to add:

kms:CreateGrant

to your IAM or SCP policy in order to successfully create the certificate request. Given certificates need to be accessed by Key Management Service (kms) it makes sense that a new certificate needs the permission to create a grant for it.

That’s it, hope this saved you from fruitless googling!

IR35 is easily avoided, but it’s time to get with the programme

Pete — 2019-08-01

I’ve read a lot of articles lately talking in doomworthy terms about the Off-Payroll working rules from April 2020 that will apply to the private-sector for contractors. Similar rules have been in force for the public-sector since 2017, with many widely predicting that private-sector IT firms will be either curtailing the use of contractors or forcing them to become permanent employees.

If you somehow haven’t already read an article explaining what all this means, in short the burden of IR35 status determination shifts from the contractor’s business to the end client or ‘fee-payer’ from April 2020. This is designed to catch ‘shadow-employees’ – those who earn money without Tax and National Insurance Contributions deducted – by threatening the fee-payers directly with fines and a large retrospective tax bill if they’re discovered to be using contractors that fall within IR35 – or in other words if they’re considered basically employees where their use of a limited company is just a means to pay less tax on employment earnings.

The taxman really hates this, and there are certainly situations where this is exactly what’s happening. The simple logic goes that if you were to remove all the indirection that a limited company ‘intermediary’ (or Personal Service Company (PSC) as HMRC like to call it) provides, and still have the same working relationship with the client, would you essentially just be working like an employee? If the answer is yes, then logically you should be taxed like one.

Just an attention-grabbing article headline?

The reason I say IR35 is easily avoided is because…well, it is. A significant majority of contractor engagements legitimately fall outside of IR35, but risk being considered inside because of a fundamental lack of understanding of the rules by clients, agencies, and contractors alike. Agencies often recruit for permanent employees and contractors both, and have somewhat sleep-walked into a tendency to use the same processes for contractor engagement as they do for ‘permies’. They use the same recruitment terminology, and the engagement follows the same routine of sending a CV and having an ‘interview’ with the client. This is not how a B2B relationship works.

Meanwhile clients are usually just looking for talent to fulfil a need, and might first look for a permanent employee but resort to backfilling with a contractor due to specialist demand and contention in the IT market. Their unfortunate mentality is that a contractor is essentially an employee but just paid differently.

And sadly, contractors themselves who aren’t terribly savvy let themselves be carried along in the processes set forth by clients and agencies alike who have got the wrong end of the stick as to how working arrangements and engagement terms should work. They send you a bunch of forms to fill out, and give you a contract, and because you want the work you fill it all out and assume this is how it’s supposed to be.

The combination of these poor-diligence factors from all parties is a situation where if HMRC come knocking and want to know about a particular engagement, you’re going to have a heck of a time trying to pretend this is a legitimate business-to-business relationship and not just an employee by any other name.

The Good News

All of this is easy to overcome. Inevitably, it just requires a bit of education and a realignment of the way you engage a business to service the requirement of another business. These are my top tips whether you’re a client looking for talent directly, or an agency seeking a contractor:

Don’t ask me for my CV. A CV is for an individual, and I’m a business. Ask for my business’ experience profile, and I’ll send you all of the skills my company is able to provide along with an overview of its previous clients and achievements. It’ll contain all of the information you’re looking for.
Don’t send me a job description. A JD is for an employee. What your client should have is a project-based scoped requirement for the delivery of work. If this is compressed into a job title, e.g. ‘IT Engineer’, you’re doing it wrong. What does the client actually want achieved within the timeframe they have in mind? Sorry, you’ll need to have a think and fully specify it.
Don’t tell me you want to put ‘me’ forward for an ‘interview’. Employees have interviews – I’m just the person representing the business. I’ll have an exploratory discussion with the client to determine whether we think there’s a feasible working relationship our businesses can have. If we both agree, great!
Remember that because you’re engaging my business, it won’t necessarily be the same person doing all of the work all of the time. At any time the person my business provides might be substituted out for someone else of equal skill who is able to do the work instead. We’ve written that into the contract, so don’t be surprised if it happens.
I don’t want to know about how cool or fun your office workplace is, or how the people who work there go for pizza and beers on a Friday afternoon. That’s great for your employees, but I’m running my own business here and I don’t need to know about that. Similarly I don’t want a company-branded nametag, or t-shirt, or any of the loot that you give your employees. I don’t want to partake of any of your employee benefits. Thanks for the offer though.
Don’t tell me who my line manager will be. I don’t have a line manager. I work for my own company. I might, on the other hand, have a senior point of contact in the client’s business. That’s fine with me, I need to know who to speak to after all, but let’s not get confused that I’m under someone’s direction and control. I’m not.
Don’t tell me there are opportunities to work from home. Work from home? I’m a business, I’ll be working from my own office (which might happen to be in my home) unless there are reasons to be present on the client site. There might be security constraints that require physical presence, meetings, scheduled updates, etc. But generally speaking you don’t get to control where my employee works. Don’t give ‘me’ my own desk – it’s not my desk, it’s your desk. My business is interested in a productive working relationship, so we’ll discuss the best solution as part of our initial discussion.
I don’t really want to use your equipment. I’ve got my own equipment that my business has bought and paid for. So unless there is an impossible constraint that requires me to use your laptop (usually security, restrictions on encryption, migration of data off-site, etc.), I’ll be using my company’s gear. Thanks though!
Don’t send my business a contract that aligns with IR35 insofar as the wording of the contract goes if you’re not absolutely certain that it properly reflects what the working-relationship with the client will be. An actual IR35 investigation will be far more interested in those critical working practices as described above than whether the contract sounds correct.
After this project (that the client has carefully specified) concludes, we both understand that the client not obligated to provide any new work to my business, nor is my business obligated to undertake it. If the client has got a new project they’d like my business to work on, we’ll put together a separate contract with those specifications embedded as before.

If all of this sounds really unreasonable, or precious on the part of the contracting business, I can only apologise but also reiterate that this is precisely how it’s supposed to be. Anything less than the above and you risk falling foul of IR35 Off-Payroll rules that would rightfully identify you as effectively an employee, or a ‘permitractor’ as we like to say in the industry. You’re part of the problem. Fee-paying clients have full discretion on how they arrange their internal project-based work requirements, and it should be pretty straightforward, and not the enormous burden that everyone is making it out to be, to properly package work into engagements that legitimate contractor businesses can consume. The taxman is relying on businesses ultimately being too fearful or just plain feeble to address these issues confidently.

Some of this will require some tweaking in your expectations, your processes, your use of terminology and more importantly your mentality. IR35 is designed to target people using limited company intermediaries as a thin shell for tax avoidance. Ultimately it’s pretty trivial to properly align your terms of engagement to guarantee that your search for a contractor is correctly and legitimately outside of IR35, and way beyond April 2020 private-sector business can continue to legitimately engage contractors as much as they wish, provided they are mindful and take note of the points above.

It might be possible (but you’d really need to convince me) that your business just cannot comply with the above for one reason or another. At which point you concede and advertise the ‘role’ as inside IR35 and search for contractors on that basis. You pay their Tax and NI and increase your costs. I’d expect them to be harder to find mind you, as their business is probably off working for a client that doesn’t have your problem – although I’ll agree that this is somewhat dependent on industry.

Contractors are businesses, treat them like one. It’s time to get with the programme.

Sense-checking AWS Cost Explorer Reserved Instance Purchase Recommendations

Pete — 2018-03-31

In November AWS announced Reserved Instance Purchase Recommendations in the Cost Explorer.

Great! I thought, finally some free native support for what a lot of cost-analysis companies offer at a price.

And yet when I look at the recommendations, they seem completely wrong to me. Each recommendation provides a link to see the usage that it was based on – but the main issue there is that Cost Explorer only knows your total EC2 usage hours per day.

Why is this important? Because RI applied utilisation only span usage within an hour.

Here’s an example:

I have an autoscaling web app that runs c5.large instances with a normalisation factor of 4. It idles at 2 instances outside of working hours, where it autoscales up to 10 for 8 hours a day.

Over the day my normalised units are:

2 instances x 4 units x 24 hours = 192
8 instances x 4 units x 8 hours = 256
Total units: 448.

That’s consistent every day. AWS sees 448 units per 24 hours = 13.3 normalised units per hour. At 4 per instance, that’s a recommendation of 3 c5.large Reserved Instances.

So I buy 3 instances. Since I have at least 2 all day, that’s 100% utilisation for those. But the remaining one is only being used for 33% of the day, in the 8 hour period when I scale above 2.

To recap:

I’ve bought 3 Reserved c5.large Instances. No upfront, N.Virginia, at a price of $39.42 each per month which I’m paying whether I use them or not.
If I buy the third instance on demand, for 8 hours a day, it costs me only $20.68 per month. But I’ve bought a Reserved Instance for it as recommended, and it’s now costing me $39.42 instead.

So that’s the problem – the recommendations are completely useless for anything but baseline steady-state load. If you scale during the day, particularly if you have very short windows of high peak usage – it’ll equate to a high number of normalised unit/hours that the recommendation is based on, which completely fail to map to the application of the reserved instance billing as this spans only across an hour.

This is a significant loss with this small example, and factored out over a large estate the recommendations and indicated savings are irresponsibly misleading. To work properly the tool needs to see a moving hour-window of concurrent instance usage across the day to work out if a lack of full utilisation would result in an overall saving. Currently, for an average auto-scaled application, I can’t see that it would – but in it’s current state it’s impossible to calculate with any accuracy.

References

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/apply_ri.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts-reserved-instances-application.html
https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/ri-recommendations.html

TL;DR: Recommendations are calculated over an aggregated day, Reserved Instance billing is only applied over an hour, so they’re worse than useless.

Docker-in-Docker Private Repository “No Basic Auth Credentials”

Pete — 2018-03-18

Recently I was frustrated in a Jenkins build when I was running Docker-in-Docker to build and push a container to AWS Elastic Container Registry (ECR).

The error on push was a familiar `no basic auth credentials` which means some issue with the credentials stored in ~/.docker/config.cfg (or perhaps ~/.dockercfg in earlier versions).

In this case I initially couldn’t understand the error, as the Jenkins declarative pipeline was using a docker.withRegistry function for the registry login, and this was being successfully written to, so what was going on?

Eventually it occurred to me, although it’s not obvious at first – as we’re running docker-in-docker, you might assume that the credentials are looked for relative to where the Docker daemon is running (i.e. on the host), but actually it’s being looked for relative to where the client is calling the daemon from. In this case – within the container. The docker.withRegistry that I was doing with Jenkins was creating credentials on the host – not within the container where the client itself was running.

There were two possible solutions here – one is to ensure you run the docker login command within the client context of the docker-in-docker container, or to mount the .docker directory on the host into the container using something like `-v /root/.docker:/root/.docker` depending on what user you’re running your containers as.

How not to hire a DevOps Engineer

Pete — 2018-02-15

This is a true story, although names and details have been removed or changed to protect the awful. Quotes however are verbatim.

As often happens when you have ‘DevOps’ in your job title, I received some recruitment agency spam from someone I’ll call Jane, c/o LinkedIn, alerting me to a ‘fantastic opportunity’ that I should call to discuss. I replied and asked Jane to send over the job spec. Her reply was ‘Send me your CV first, and I’ll send you the spec’.

Huh. Given that my LinkedIn profile is complete and public – there should be no need to hand over my personal data before I even get to see the job description, bearing in mind that I was cold-contacted first, but in any case I fired it over along with an indication of my salary expectations as was requested. I received the spec along with a warning: ‘Keep the company name to yourself!’. We’ll call the company Sprockets Incorporated (not their real name, but what a great name!)

Jane then called to more formally introduce herself, and asked to meet with me, stressing that she liked to meet everyone she represents, and we agreed a date for a few days time. At 4pm the day before the meeting I received an email from Jane which said:

Hope you are well. We have a meeting set up for tomorrow at 11am, unfortunately I’m going be out of the office therefore need to cancel our meeting.

I’ve also had feedback from Sprockets and unfortunately they won’t be progressing to interview.

I’d like to thanks you for your interest and wish you luck in your job search going forward.

Well, what a coincidence! The company didn’t want to proceed to an interview and simultaneously, just by chance, her schedule had suddenly filled up and she could no longer meet me. Almost like the second I wasn’t immediately valuable to her, I wasn’t worth meeting and summarily dropped.

I almost replied to remark as much, but instead politely asked if there was any feedback. It transpires that my salary expectations were slightly higher than Sprockets were looking for, although not by much. I said to Jane that money was not the single most important factor in any job, so I would be happy to negotiate on that point and have a conversation with them on that basis.

Suddenly, it’s all back on and Jane is my best friend again. She also wants to know if I know any Test Leads that I can put her in touch with because she has other roles to recruit for. Ahem. If you’d like a professional referral you can talk to me about the signing bonuses you’re offering, otherwise I’m not doing your job for you!

I have a telephone call with Sprockets, which converts into a face to face interview arranged for a couple of days later. Jane sends me an interview confirmation for the ‘Test Lead’ role, and I point out I’m interviewing for the DevOps position. To which she says ‘Oops’, and incredibly asks again if I know of any Test Leads that might be looking.

As timing would have it, I’m having another interview the day after Sprockets with a company called Spacemax (still not a real name). The interview with Sprockets goes great – everyone seems really nice and they seem like they’d be a good place to work. During the course of the interview they ask if I’m looking anywhere else, so I admit to having another interview the next day (it can’t hurt). To my surprise, a job offer comes through later that same day. I thank them and ask for some time to consider, at least to allow me to have the other interview, but I assure them that afterwards I’ll get back to them promptly.

Suddenly, I can’t get rid of Jane. She calls me several times, pressuring me to take the offer – telling me Sprockets is great. Not only that, but that her agency has the exclusive contract for Sprockets, and if I let this pass up I won’t see the job advertised anywhere else. I already know that’s a lie, of course, because in the meantime I’ve already been contacted on LinkedIn by another agency representing the same role. I repeat that I’d like to have the Spacemax interview and I’ll let her know after that.

Unfortunately for me, the interview with Spacemax goes great too, and they make me an offer on the spot. Now I have a difficult decision to make – I like both companies and there’s not a lot to separate them in terms of salary or benefits. Luckily my phone was on silent mode, because it turns out Jane was calling me several times during the interview to see if I’d made up my mind about Sprockets yet. As I’m walking out of Spacemax, my phone rings again and it’s Jane. I tell her I’ve been made an offer and will need time to think.

This doesn’t go down well. She assures me Sprockets are a great company, with a bright future, and I’d do really well there and she’s not just saying that, and then goes into a little three minute sales pitch about why this place is great and how I should definitely take the job. Incredibly when she’s finished, she says to me: ‘So what do you think now?‘ Wait, what? Did she think a couple of minutes of off-the-cuff pressure selling was going to sway me into accepting?

Because I like to keep everyone in the loop, I let both companies know I have competing offers and I’ll be deciding shortly. I make it clear this is not a tactic to play one offer off against the other – I’d just like a little time to consider. It’s damn tricky, I really do like them both.

Jane keeps calling me, Sprockets are keen to convince me and various senior members of their team call me individually to talk through their take on the business and why it’d be great if I joined them. After each call Jane is back onto me – I am now dreading having to pick up the phone to her – to see if any of it has done the trick in swaying me. She tells me Sprockets is much better than Spacemax, and has a better future ahead. She also tells me that she was so confident that I was the right person for the job that I was the only candidate she put forward. I wish I could say this is a bit of embellishment for the sake of the story, but it’s not. If I’m the only candidate she’s put forward, it’s because she couldn’t find anyone else – or perhaps they’d been snapped up by the other, much more competent agency. I wish I had.

After deliberating, I decide I’m going with Spacemax. I thank Sprockets profusely for their consideration and tell them genuinely that it was an incredibly marginal and difficult choice, and wished them the very best for the future.

I receive the following email from Jane:

Thanks for your email. Really disappointing but you’ve made your decision.

I wish you every success and if anything changes please get in touch.

Do you know of anyone else that may be interested?

Sheesh! Crass and absurd.

But wait, the story isn’t quite over yet. I start with Spacemax a week later (the turnaround on interview and starting was crazy!), and precisely 5 days into my new job I get the following email from Jane:

Hi Pete

How are you? How’s the new role going? Hope it’s all working out.

I’d love the opportunity to work with SpaceMax and wondered if you could help me/introduce me to right person to discuss recruitment?

Speak soon

Sweet Jesus. I ignore this, and a couple of weeks later I see that I have a missed call from Jane. I email her, asking what the call was about. She replies to repeat her request for an introduction to be able to recruit for us. Having now finally had enough, I tell Jane that I felt her approach was completely inappropriate, particularly in light of the fact she was so recently pushing me very hard NOT to work for Spacemax.

I believe that even had I been interacting with a credible, competent agency, I’d have made the same decision, but wow Jane did not make it easy and I can only imagine that Sprockets would have been furious if they’d known they were being represented by someone so unprofessional.

If you ever want to hire anyone, for anything, don’t do this.

On the flip side, here’s what the other recruiter representing Spacemax said to me about my competing offers:

Obviously if you go for Spacemax it is in my interests, but I appreciate it’s a decision that’s entirely up to you and Sprockets are another good company.

Much, much better.

Ansible Slack Failure Handler

Pete — 2018-02-07

So you want a simple Slack failure handler for Ansible to ping your alerts channel whenever a deployment fails. Despite a thorough search I couldn’t find any examples that did this adequately, and even the solution here is a little constrained. The basic requirements are:

A Slack notification on any task failure.
The name of the task.
The name of the host.
The error debug message.

Sadly there isn’t one global failure handler configuration in Ansible. I investigated the native Slack Callback plugin available in Ansible 2.x (and not to be confused with the existing Slack module) but this seemed to be more of a generic passthrough for outputting all plays into Slack, failed or not, which isn’t what I wanted. After discarding the idea of a custom callback plugin for my purposes (which could work, but felt overly complex), I settled on a per-playbook failure role.

This is not an actual ‘handler’ as per Ansible parlance, but I was coming from experience of Chef where I could have a global failure handler baked into the Chef-client config.

To make this work we need to use Playbook Blocks (as of 2.0) and essentially enclose the entire playbook in a block/rescue. The main hassle here is that block does not support the top-level pre-task role block (and wouldn’t catch any failures therein), and so I had to convert all of my role calls to tasks that used include_role instead.

A simple playbook example looks as follows:

playbooks/playbook.yml

- hosts: "{{ target_host | default('127.0.0.1') }}"
  gather_facts: true

  tasks:
  - block:
    - include_role:
        name: install_app
    - name: Greet the world
      shell: echo "hello world!"
    - fail:
       msg: "I've gone and failed the play!"
    rescue:
      - include_role:
          name: slack_handler
          tasks_from: failure

And in my slack_handler role (for reusability):

roles/slack_handler/tasks/failure.yml

- name: Notify Slack of Playbook Failure
  slack:
    username: 'Ansible'
    color: danger
    token: "{{ slack_webhook.split('https://hooks.slack.com/services/')[1] }}"
    channel: "#deployment-alerts"
    msg: "Ansible failed on *{{ ansible_hostname }} ({{ inventory_hostname }})* \n
    *Task*: {{ ansible_failed_task.name }} \n
    *Action*: {{ ansible_failed_task.action }} \n
    *Error Message*: \n ```{{ ansible_failed_result | to_nice_json }}``` "
  delegate_to: localhost

ansible_failed_task and ansible_failed_result are two currently painfully undocumented (shout-out to Brian Coca for pointing me in the right direction) but delightfully detailed variables that are populated on playbook failure. ansible_failed_task is a map that contains a lot of data, so you may want to add additional debug for your purposes. The raw error message is single-line JSON so we prettify it before sending it using the Slack module using our token. The string split is a bit of an ugly hack to extract the webhook token part from the full webhook URL which is used elsewhere in my plays and passed at deploy time. For some reason the Slack module requires only the token rather than the full URL, in contrast to a lot of other integrations that want the whole thing.

The remaining hassle is the need to implement this per playbook, but aside from the possibility of a custom callback plugin this seems like the simplest way to implement this in Ansible currently.

ASICS Marathon Training Plans

Pete — 2017-08-21

The original ASICS Marathon training plans remain one of the most popular, much loved guides for those preparing for their first marathon (or marathon improvers). Having used these myself, I was slightly frantic recently when I realised that Runners World, who had originally hosted the plans, removed them from their site sometime in August 2017.

To avoid these being lost in the internet ether, I quickly located copies and rehost them here for people like me that want a solid reference guide for their marathon training plans 🙂

ASICS_TRAININGPLANS_Sub 3.00
ASICS_TRAININGPLANS_Sub 3.30
ASICS_TRAININGPLANS_Sub 4.00
ASICS_TRAININGPLANS_Sub 4.30
ASICS_TRAININGPLANS_Sub 5.00

Amazon Web Services has Zombie Apocalypse ToS Clause

Pete — 2016-02-10

Courtesy of the AWS subreddit, I was alerted to the fact the recently-updated AWS Service Terms (to covertheir new 3D Lumberyard Game Engine) have included this specific clause:

57.10 Acceptable Use; Safety-Critical Systems. Your use of the Lumberyard Materials must comply with the AWS Acceptable Use Policy. The Lumberyard Materials are not intended for use with life-critical or safety-critical systems, such as use in operation of medical equipment, automated transportation systems, autonomous vehicles, aircraft or air traffic control, nuclear facilities, manned spacecraft, or military use in connection with live combat. However, this restriction will not apply in the event of the occurrence (certified by the United States Centers for Disease Control or successor body) of a widespread viral infection transmitted via bites or contact with bodily fluids that causes human corpses to reanimate and seek to consume living human flesh, blood, brain or nerve tissue and is likely to result in the fall of organized civilization.

It’s good to know that AWS are accommodating all possible contingencies for social and economic failover.

Amazon Web Services (AWS) Release new g2.8xlarge GPU instance

Pete — 2015-04-17

Today AWS announce a long-awaited upgrade to their G2 family of instances – the g2.8xlarge. The big brother of the 2x, which was hitherto the only GPU-backed instance available on the AWS platform.

Here’s how the two compare in specifications:

Instance	vCPU	ECU	Memory (GiB)	Instance Storage (GB)	EC2 Spot Price ($/hr)
g2.2xlarge	8	26	15	60 SSD	$0.08
g2.8xlarge	32	104	60	2 x 120 SSD	$0.32

The four GPUs are the same NVIDIA GRID K520 seen in the 2x instance, and as you can see by the numbers the 8x is exactly four times larger in every respect. The indicative spot price at the time of writing was also very close at roughly 4x the cost.

In my previous post where I ran a benchmark of the g2.2xlarge instances using the Distributed.net RC5-72 project, I re-ran the same test using an 8x. You will not be surprised to learn that the results shows a linear increase in the crunching keyrate to roughly 1.7GKeys/sec (previously 432 Mkeys/sec on the 2x).

Is bigger better?

AWS’ fleet and pricing structure is generally linear. For an instance twice the size, you pay twice the cost, both in spot and on-demand. The major difference that is not very clearly advertised is that the network performance is greater for larger instances. AWS are vague as to what ‘Low’, ‘Moderate’, and ‘High’ mean in terms of raw speed (many others have tried to benchmark this), but in the largest instances this is explicitly stated at 10 Gigabit. If you assume a larger box is pumping out more data, it will need a network connection to match. But you also assume that an instance only generating 1/4 as much data will be equally well served by a ‘Moderate’ connection.

A real world use case

In my day job I set up a variety of analyses on genetic data that is supplemented by EC2 computation clusters (The recent AWS Whitepaper on Genomic Architecting in the Cloud is a really useful resource I can throw at scientists when they have questions). I investigated the viability of G2 instances and for a specific analysis that was GPU-capable, it did indeed run roughly 3-4 times faster than the same job running on a single CPU core. The problem was memory – each job used roughly 3-5GiB of memory meaning I couldn’t run more than 3 or 4 jobs on a single g2.2x GPU at once.

However on a r3.8xlarge – a CPU instance with 32 cores and 244GiB memory, I could run 32 concurrent jobs with memory to spare. Sure, the jobs took 30 minutes each instead of 10, but I could run 32 of them.

Then I drilled down on cost/benefit. The G2.2x was $0.08 on spot, and the r3.8x was $0.32. Four times as much per hour to run, but with 10 times as many jobs running. It ended up being a no-brainer that a CPU instance was the way to go.

Perhaps this is a poor example, because the capabilities of genetic analysis is badly limited by the tools available for the specific job, and it’s reasonably rare to find anything that is built for multi-threading, let alone something designed specifically to run on GPUs. The implementation of these analysis tools are black box and we’re not software developers. Our tests were probably very bad exemplars for the power of a GPU instance, but it did show that a mere 15GiB RAM on the 2X just wasn’t anywhere near enough. 60GB on the 8x is a little better but in my use case it still wouldn’t offer me any additional benefit because I wouldn’t be able to leverage all of the GPUs I’m paying for (our software just isn’t good enough). FastROCS, the example cited in Jeff Barr’s AWS Blog annoucement about the g2.8x also mentions the 15GiB of the 2x being the limiting factor, so presumably they’re running jobs that can leverage more GPU in a single job without a proportional increase in memory usage.

The main benefit of one vertically-scaled box four times the size is speed. If your application can utilise four GPUs simultaneously within the memory limits then you could, for example, transcode a single video extremely quickly. If speed is the main factor in your service delivery then this is the instance for you. If however you’re running smaller jobs that are less time-critical that the 2x can handle just as well, there’s little benefit here, unless you consider the logical management of four GPU instances to be more hasslesome than running one four times the size. But then all of your GPU eggs are in one basket, and if the single instance goes down so does all of your capacity.

As with all AWS usage your chosen method of implementation will depend on what’s right for you. This is going to be great news for a lot of people – but unfortunately not me!

CPanel Webserver on EC2 (or How I learned to stop worrying and love the cloud )

Pete — 2015-02-25

It’s been a long time coming but my company is now fully based on Amazon Web Services EC2 for our web hosting. It’s been a long journey to get here.

For more than 15 years we’ve cycled between a variety of providers who offered different things in different ways. First our sites were hosted by Gradwell, on a single shared server where we paid a per-domain cost for additional hosting. We left them in 2004 after 6 years and following a very brief flirtation with Heart Internet we moved to a now-defunct IT company called Amard. This gave us our first taste of CPanel as a way of centralising and easily managing our hosted sites. It was great – so great that it made moving to ServerShed (also defunct) in 2005 very easy.

This was our first dedicated server with CPanel and we enjoyed that greater level of control while still having the convenience of a largely self-managing service. In 2007 we hopped ship to PoundHost where we remained until last week. Three or four times in the last 8 years we’ve re-negotiated for a new dedicated server and battled with the various frailties of ageing hardware and changing software. Originally we had no RAID and downloaded backups manually, then software RAID0, then hardware RAID0, then most recently with S3 backups on top.

Historically ‘the cloud’ was a bit of an impenetrable conundrum. You knew it existed in some form, but didn’t really understand what it meant or how its infrastructure was set up. Access to the AWS Cloud in the early days, as far as I understand, was largely command-line (CLI) based and required a lot of knowledge to get going on the platform. It required a lot of mental visualisation and I’m sure the complexity would have been beyond me back then. Some services didn’t exist, others were in their infancy. Everything was harder.

In that respect I almost don’t mind being a bit late to this party. It’s only in the last two or three years that it seems the platform has been opened up to the less-specialised user. Most CLI functions have been abstracted into a pretty, functional web console. Services interact with each other more fluidly. Access controls that govern permissions to every resource in a really granular way have been introduced. Monitoring, billing, automated resource scaling, load balancing and a whole host of other features are now a reality and can boast really solid stability.

Even then migration has not been a one-click process. CPanel do offer a VPS-optimized version of the Web Host Manager (WHM) and that’s the one we’re running. Its main boast is that it uses less memory than the regular version, apparently as a concession to the fact that a Virtual Machine is more likely to exist as a small slice of an existing server and won’t have as many resources allocated to it. It looked to be the best fit.

Then we needed to find a compatible Amazon Machine Image (AMI) to install it. It seemed to be largely a toss-up between CentOS and RedHat Enterprise Linux. We’d used CentOS 6 in the past with good results (and unlike RHEL it requires no paid-up subscription), so we fired it up and slowly started banging the tin into shape. Compared to other set-ups there is very little AWS-specific documentation on setting up a CPanel environment, so we were mostly sustained by a couple of good articles by Rob Scott and Antonie Potgieter. CPanel themselves wrote an article on how to set up the environment but this didn’t quite cover enough and already the screenshots and references there are out of date. I will write a comprehensive overview of ‘How to Set up CPanel on EC2’ in another article, but to conclude here I will talk about what made this transition so tentative for us that we waited almost a year after setting up AWS before we went live with our main server on the platform:

Fear. Non-specific, but mostly of the unknown. It’s irrational, because when you’re buying managed hardware from resellers you never actually get to see the physical box you’re renting. It’s no more tangible to you than an ephemeral VM is, and yet there’s something oddly reassuring in knowing that your server is a physical brick of a thing loaded into a rack somewhere. If something goes wrong, an engineer that you can speak to on the phone can go up to it and plug in a monitor to see what’s happening. It’s not suddenly going to disappear without a trace.

Not so with the cloud. It’s all a logical abstraction. You don’t know, you’re not allowed to know, precisely where the data centres are. You don’t know how the internal configuration works. Infrastructure is suddenly just software configuration and we all know how easy it is to make a mistake. Click the wrong box during the set-up and you might have your root device deleted accidentally, or be able to terminate the server with all of your precious data because you didn’t enable Termination Protection. If you’re a little careless with your access keys and they become public, you’ll find your account has been compromised to run expensive clusters farming bitcoins at your expense.

Terrifying if you don’t know what you’re doing. So you investigate, learn slowly, play on the Free Tier. Make things, break things. Migrate one small site to make sure it doesn’t explode. Back it up, kill it, restore it, and be absolutely confident that this is really going to work.

And it does. For the first time I feel like we’re our own hosting provider. I spec the server, I buy it by the hour, I configure and deploy it, and Amazon is merely the ‘manufacturer’. Except that everything happens in minutes and seconds instead of weeks and days. Hardware failure is handled as simply as just stopping and restarting the instance where it will redeploy on different hardware. If you’ve architected for failure as you should, backups can be spun up in less than ten minutes. if I get nervous about spiralling running costs I can just flip the off switch and the charges stop, or the Trusted Advisor can offer suggestions on how to run my configuration more efficiently.

It’s telling that every company that I’ve ever bought a physical server from are now pushing cloud-based offerings to their customers too. The time of procuring your own hardware is passing and being replaced by more dynamic, more durable, more robust solutions.

You’d be forgiven for thinking this whole post was merely a sly advert for AWS, but I really am just a humble end user. Admittedly one that has been completely converted, and I haven’t even enumerated the full list services that I now use, to say nothing of the others that I haven’t even had time to investigate. This is a great time to make the transition, because while adoption has been growing at a huge rate, I think it’s going to skyrocket to even greater heights in the next few years.

If you don’t have a pilot for the journey, it’s time to train one.

Server-wide Referrer Spam Blocking for Cpanel WHM

Pete — 2015-02-22

If like me you maintain a WordPress website, you might have recently noticed a number of referrals to your site coming from one of the following:

buttons-for-website.com,
buttons-for-your-website.com,
best-seo-offer.com,

or some other unknown, dodgy-sounding website that clearly isn’t linking to you legitimately.

This is referral spam that seeks to achieve high Google PageRank by gaming the system to create the appearance of popularity. One major way to do that is to get a high number of external backlinks and referring domains. These little spambots will visit your site and attempt to spam your WordPress installation with comments if it can, as well as dropping the appearance of referring traffic into your analytics and logs. It’s very annoying because it’s not legitimate traffic and you’ll need to know to exclude it.

It is possible to block these referrals on a per-site basis by adding the following to your site’s .htaccess file:

# Block Referer Spam
RewriteEngine On
RewriteCond %{HTTP_REFERER} buttons\-for\-website\.com
RewriteRule ^.* - [F,L]

This is great, but what if you have a whole server of WordPress domains? Adding this directive manually to each one would be very tedious. Luckily this becomes easy in CPanel with a couple of configuration changes.

1) Follow this guide on creating a custom VirtualHost template within your CPanel installation. Once you have created both the vhost.default and ssl_vhost.default files, open them up and add the following lines to the VirtualHost directives:

RewriteEngine On
RewriteOptions Inherit

2) Go to your WHM in ‘Apache Configuration->Include Editor’ and add the blocking directive (# Block Referrer Spam referenced above) to the ‘Pre-Virtual Host Include’ area for All versions.

So what we’re doing here is telling the Apache config to inherit any Rewrite rules from the main server context for each virtual host by making a custom VirtualHost template to say that. Then we’re specifying that rule in the main server context which will inherit into each hosted site.

In this way the entire server will have their sites tested for the referral block and apply it globally. If you’re maintaining a client server this is the ideal way of taking care of this problem automatically. If you spot any more insidious referrals to your sites in future, just update the Pre-include config to check for the appropriate rewrite condition.

If you were curious, I checked and the ‘Buttons for website’ site has some 800,000 external backlinks from over 10,000 domains – that’s a lot of effort invested in trying to bump up its rank. As of today, it has a Google PageRank of 0. Spam shouldn’t and seemingly doesn’t pay!

Amazon Web Services (AWS) now provides billing in local currency

Pete — 2015-02-17

Developers and businesses around the world will be breathing a huge sigh of relief today as Amazon Web Services finally announced the ability to have invoices generated and charged in one of 11 new local currencies.

Since its launch in 2006, AWS has exclusively billed in US dollars worldwide, causing a currency-conversion and bank fee headache for all but U.S. customers. As a multi-billion dollar, multi-national operation, the introduction of this feature is long overdue and brings AWS into line with other providers such as Microsoft Azure in offering bills in the currency you’re most familiar with.

The new list of supported currencies, when using an elligible Visa or MasterCard to pay your bill are:

Australian Dollars (AUD)
Swiss Francs (CHF)
Danish Kroner (DKK)
Euros (EUR)
British Pounds (GBP)
Hong Kong Dollars (HKD)
Japanese Yen (JPY)
Norwegian Kroner (NOK)
New Zealand Dollars (NZD)
Swedish Kronor (SEK)
South African Rand (ZAR)

This new preference can be set in your AWS ‘Billing and Cost Management’ account under ‘Account settings’. The changes take effect immediately and your estimated bill for the month is then presented to you in your local currency.

The one important factor to note is that AWS pricing remains firmly in dollars, and all prices presented to you on purchasing remain that way. Amazon appear to be offering an exchange rate that changes daily so all estimates are based on that, and will naturally be susceptible to currency fluctuations – so CFO headaches aren’t gone entirely! Presumably the final bill for the month will be based on the exchange rate for the given currency on that day, but when compared to the existing method – getting hammered by your bank for currency conversion fees at terrible transfer rates – this is substantially preferential and will result in lower overheads for all non-US AWS customers.

Raspberry Pi 2 Model B with Distributed.net dnetc RC5-72 client

Pete — 2015-02-10

A little while ago I wrote up a summary of the distributed.net RC5-72 project. One of my habits over the years has been to run the good old cow client on every new computer I’ve built just to see how the speed compares.

So when I picked up a new Raspberry Pi 2 Model B this week, this habit held true. For this I installed the ARM/embi client v2.9110.519 (sadly dated 2012 – there have been very few client updates in the last few years), and it ran without any issues. By default it doesn’t detect the Pi 2’s quad core architecture, so requires manually setting the Performance options to use 4 cores.

It’s not exactly speedy, but then this is a computer that I can fit in my back pocket.

Results

Four simultaneous crunchers took 1 hour 3 minutes to complete 4 stat units, at a combined keyrate of 4.5Mkeys/sec. At that rate the Pi could crunch around 91 stat units per day. That means that running by itself, the Pi could complete the remaining work on the project in a mere 32 million years. I don’t think the warranty lasts that long.

The CPU temperature held fairly steady at 54 Celsius, and I will also note that I didn’t overclock the Pi so this was running at the default 900mhz.

While the Pi does boast a Broadcom GPU with 1GB RAM (shared with CPU), there are no compatible crunching clients for a GPU test, and I suspect the project is in such a lacklustre state now that I don’t foresee any budding development on one soon. This was still a fun little test, and isn’t that much slower than a decent home PC would have been 15 years ago.

How to modify ‘Deletion on Termination’ flag for EBS volume on running EC2 instance.

Pete — 2015-02-05

When launching an EC2 instance on Amazon Web Services, the EBS volume is set to ‘Delete on Termination’ by default. Most of the time this is fine as you’d often rather make a snapshot of your drive to enable you to boot up multiple copies of the same instance, or if you’re a developer that’s creating and terminating instances regularly it would be a nightmare to have all of these orphaned EBS volumes cluttering up your account.

But what do you do when you have a running instance where the EBS was set to Delete and you’ve changed your mind and want to keep it? The AWS documentation is surprisingly vague about this, mostly talking about setting the flag on launch. So how do you do it for a running instance? And for that matter, how can you tell if you set an existing instance to Delete on Termination? It might have been a while and it’s hard to remember if you ticked that checkbox.

How to check the EBS ‘Delete on Termination’ flag

It’s a little buried. Go to your EC2 management console and click on ‘Instances’. Click on the instance you’re curious about, and then under the ‘Description’ tab, scroll down to ‘Block devices’, and click on the appropriate EBS volume. This will pop up an attribute box which will state the Delete on Termination flag. This seems to be the only place in the whole AWS console to check this information!

How to modify the EBS ‘Delete on Termination’ flag

The only way to do this is by using the AWS CLI, at the current time there’s no way to do this using the web console.

You can do this in one of two ways, either by specifying a JSON attribute file with the modifications you want to apply to the instance, or by doing it in-line with escape characters.

aws ec2 modify-instance-attribute --instance-id i-a3ef245 --block-device-mappings /path/to/file.json

with a .json file in the a format such as:

[
    {
    "DeviceName": "/dev/sda1",
    "Ebs": {
      "DeleteOnTermination": false
      }
    }
]

Or without using a .json file inline like this:

aws ec2 modify-instance-attribute --instance-id i-a3ef245 --block-device-mappings "[{\"DeviceName\": \"/dev/sda\",\"Ebs\":{\"DeleteOnTermination\":false}}]"

Note: You will need to configure the AWS cli (command from terminal: ‘aws configure’) either with user credentials with a policy that grants permissions to the appropriate EC2 instance, or run the CLI on an instance which has EC2 Role permissions.

Now hopefully you’ll be able to save that EBS volume after changing your mind about the termination flag!

Distributed.net RC5-72 on Amazon Web Services (AWS) EC2 Cluster – A modern solution to an old problem

Pete — 2014-11-16

History

Anyone kicking around the internet since the early days will have heard of distributed.net‘s RC5-72 distributed computing project. Arising from RSA labs Secret-Key Challenge, the project sought to utilise distributed computing power to perform a brute force attack in an attempt to decrypt a series of secret messages encrypted using the RC5 block cipher. Sponsored by RSA Labs, a $10,000 reward was offered to the participant whose machine was responsible for finding the correct key for each of the challenges, starting at 56-bit encryption, and scaling up to 64, 72, 80, 88, 96, 104, 112, 120, and 128.

I started contributing to the 64-bit instance of the project (termed RC5-64) back in 2001, and with the combined computing power of around 300,000 participants, the key was cracked after almost 5 years of work in July 2002. This project required the testing of 68 billion key blocks (2^64 keys) and found the correct key after searching 82.7% of the total keyspace.

A new project to tackle the next message, encrypted with a 72-bit key, was started on December 2nd 2002. This project required the testing of 1.1 trillion key blocks (2 ^ 72 keys) – 256 times larger than the original project that had taken 5 years to complete. After a few years it became apparent that this was going to take ‘a very long time’, and RSA Labs withdrew the challenges in May 2007, along with the $10,000 prizes. Shortly after this news, distributed.net announced that they would continue to run the project and would fund a $4000 prize alternative.

Today

As of today the project has made it through 3.378% of the keyspace after 12 years of work. At the current rate the project anticipates hitting 100% in a mere 219 years.

You can imagine that so many years of effort has dulled the enthusiasm of its participants. The distributed.net statistics report that there have been some 94,000 unique participants to RC5-72 (significantly down on the 300,000 of the previous project), but that only 1200 of them remain active. With the rise of other distributed computing projects in the early 2000s such as SETI@home, Folding@home and many others, this humble project has been rather forgotten by the internet at large. The advent of Bitcoin and other e-currencies have led to people turning their spare processing power to more profitable ends.

And yet, the RC5-72 project’s overall keyrate remains higher today than it ever has been in the past. The reason for this was the development and widespread use of powerful GPUs in home computers. I’m the first to admit I don’t really understand the finer points of how computer hardware works, but GPUs turned out to be roughly 1000 times faster than even the fastest commercial CPU at crunching through keys. Traditional CPU crunching makes up less than 10% of the total daily production, with the vast majority coming from the GPU ATI Stream technology (70%), with the rest coming from NVIDIA CUDA (5%) and the more recent OpenCL – supported on both ATI and NVIDIA hardware (14%).

As much as I salute distributed.net for continuing to maintain the project and run the the supporting key servers to distribute the work and run the stats, I have to say I’ve never seen any active effort to promote it. The website is rather dated and to my memory has the exact same design that it had in 2001 when I first started. There are no social media buttons, no methods of incentivisation, and even some of the more basic things like keyrate graphs have been broken for months (or years) and nobody has worried about fixing them.

A possible solution

I know that it must be difficult to commit any real effort to something that has been a back-burner operation for almost 10 years, but the completionist in me is desperate to somehow mobilise the modern massive internet to attack the project with gusto and get it to 100% in a month. I know that’s a laughably ridiculous suggestion, because the scale of the work required is massive. If you had a reasonably powerful ATI graphics card that could churn through one billion keys per second, you might be able to crunch around 19,000 work units per day if the GPU was dedicated to the task 24/7. At that rate you could expect to complete the project in around 1.6 million years.

So 1200 people aren’t going to get this done, even though the top 100 of those can pump out almost 10 million work units a day.

The advent of ‘the cloud’ has made potentially unlimited computing power available to anyone in the world – at a cost. Amazon Web Services (AWS) have the largest compute infrastructure of any provider and I’ve spent quite a while familiarising myself with the platform. It occurred to me that a key-crunching test of RC5-72 was in order.

The Test

For the basic test, I provisioned a compute-heavy c3.8xlarge instance running a traditional Linux x86 CPU client, and a GPU g2.2xlarge instance running the CUDA 3.1 distributed.net client, and I latterly also tested the OpenCL client.

The keyrate and cost results were as follows:

Instance Type	dnetc Client	Keyrate (Mkeys/Sec)	EC2 Spot Price ($/hr)
c3.8xlarge	v2.9109.518	180	$0.32
g2.2xlarge	v2.9109.518 (CUDA 3.1)	423	$0.08
g2.2xlarge	v2.9109.520 (OpenCL)	432	$0.08

The OpenCL client was the winner, offering 432 million keys/sec for 8 cents an hour. It should be noted that this falls far short of the best possible recorded speed from a GPU, where an ATI Radeon HD 7970 can do a stunning 3.6 billion keys/sec – although this benchmark list is at least a year old so it’s probable there exists cards out there that are even more powerful. Compared to that a mere 0.43 billion/sec is only 11% as powerful.

The potential advantage of the AWS cloud is seemingly not in its raw speed, but its scale. I can’t run 10 graphics cards at home but I can run 10 instances of the dnetc client. So that’s what I did for around 36 hours. The cost of 10 instances for 1 day equated to $0.80 x 24 = $19.20. Not bank-breaking but quite a lot to achieve a total speed of 4.32 billion keys/sec. That singular 24 hour effort put me at #26 in the top 100 rankings for the day, and the 87,000 work units completed bolstered my grand total to 1.4 million units over the 12 years I’ve been working on the project. A fairly hefty chunk relative to my total effort, but still a tiny drop in an enormous ocean.

Getting to 100%?

After a few calculations, I estimated that I would require 346,000 nodes like this, running 24/7 for one year, to complete the project. Assuming I could maintain a spot price of 8 cents an hour, it would cost ‘only’ $232 million to provision the cloud to complete this project. I think it’s pretty unlikely I could crowd source this from the internet in order to complete an old cryptography project, but there are other, more simple solutions.

1) Better GPUs. AWS aren’t the only cloud provider and one of the many others may provide GPU instances with better computational power – but none of them offer AWS’ novel ‘spot pricing’ model to get instances at rock-bottom prices, so this is unlikely to be cheaper.

2) Don’t use the cloud at all. Somehow compel millions of people to start running the client on their high-end graphics cards at home and work too. Might happen… but not without a concerted social media effort, refresh of the website, and some kind of modern fun ‘game’ incentivisation thing. Doing it for the fun of cryptographic curiosity is unlikely to motivate too many casual users.

3) More bespoke supercomputers like this one. The Center for Scientific Computing and Visualization Research at the University of Massachusetts have built a bit of a Frankenstein super-computer from old Playstation and AMD Radeon graphics cards. It’s seemingly used for other computational purposes, but the excess capacity is put into the RC5-72 project and recently has been churning out an eye-popping 1.2 million work units a day (more than 10% of the total work of the top 100) – cool huh?

4) Maybe Amazon would like to take on the challenge directly to demonstrate the power of their cloud. They’ve done things like this in the past (like creating the #72 supercomputer in the world with 26,496 cores rated at 593 Tflops/sec) and would presumably do it for free as a bit of a boast, but then there would also be the fear that the other tenacious users on the project would feel a bit cheated by having a giant multinational come along and solve the problem without them. But it’s also possible that the joy of having it complete would be worth the disappointment of having not achieved it personally.

This entire experiment and post was a bit of nostalgic indulgence for me. Rc5-72 has been a tempting Everest of distributed computing and back in the day I wanted to be part of the pioneering team that conquered it. Now there are only a few of us left, and we haven’t even reached basecamp yet. At this point I’d gladly hop on a helicopter to the summit except I don’t know where to get one.