Amazon Web Services, DevOps

If you’re using the AWS ACM console to create a certificate, and at the final stage you get this annoying and useless, I’ve got the solution for you.

You’re probably working in an AWS Organization with a Service Control Policy (SCP) or a restricted IAM user where you’ve been given the acm:* permissions thinking this is enough. Sadly not, the solution is you additionally need to add:


to your IAM or SCP policy in order to successfully create the certificate request. Given certificates need to be accessed by Key Management Service (kms) it makes sense that a new certificate needs the permission to create a grant for it.

That’s it, hope this saved you from fruitless googling!

DevOps, Recruitment

This is a true story, although names and details have been removed or changed to protect the awful. Quotes however are verbatim.

As often happens when you have ‘DevOps’ in your job title, I received some recruitment agency spam from someone I’ll call Jane, c/o LinkedIn, alerting me to a ‘fantastic opportunity’ that I should call to discuss. I replied and asked Jane to send over the job spec. Her reply was ‘Send me your CV first, and I’ll send you the spec’.

Huh. Given that my LinkedIn profile is complete and public – there should be no need to hand over my personal data before I even get to see the job description, bearing in mind that I was cold-contacted first, but in any case I fired it over along with an indication of my salary expectations as was requested. I received the spec along with a warning: ‘Keep the company name to yourself!’.  We’ll call the company Sprockets Incorporated (not their real name, but what a great name!)

Jane then called to more formally introduce herself, and asked to meet with me, stressing that she liked to meet everyone she represents, and we agreed a date for a few days time. At 4pm the day before the meeting I received an email from Jane which said:

Hope you are well. We have a meeting set up for tomorrow at 11am, unfortunately I’m going be out of the office therefore need to cancel our meeting.

I’ve also had feedback from Sprockets and unfortunately they won’t be progressing to interview.

I’d like to thanks you for your interest and wish you luck in your job search going forward.

Well, what a coincidence! The company didn’t want to proceed to an interview and simultaneously, just by chance, her schedule had suddenly filled up and she could no longer meet me. Almost like the second I wasn’t immediately valuable to her, I wasn’t worth meeting and summarily dropped.

I almost replied to remark as much, but instead politely asked if there was any feedback. It transpires that my salary expectations were slightly higher than Sprockets were looking for, although not by much. I said to Jane that money was not the single most important factor in any job, so I would be happy to negotiate on that point and  have a conversation with them on that basis.

Suddenly, it’s all back on and Jane is my best friend again. She also wants to know if I know any Test Leads that I can put her in touch with because she has other roles to recruit for. Ahem. If you’d like a professional referral you can talk to me about the signing bonuses you’re offering, otherwise I’m not doing your job for you!

I have a telephone call with Sprockets, which converts into a face to face interview arranged for a couple of days later. Jane sends me an interview confirmation for the ‘Test Lead’ role, and I point out I’m interviewing for the DevOps position. To which she says ‘Oops’, and incredibly asks again if I know of any Test Leads that might be looking.

As timing would have it, I’m having another interview the day after Sprockets with a company called Spacemax (still not a real name). The interview with Sprockets goes great – everyone seems really nice and they seem like they’d be a good place to work. During the course of the interview they ask if I’m looking anywhere else, so I admit to having another interview the next day (it can’t hurt). To my surprise, a job offer comes through later that same day. I thank them and ask for some time to consider, at least to allow me to have the other interview, but I assure them that afterwards I’ll get back to them promptly.

Suddenly, I can’t get rid of Jane. She calls me several times, pressuring me to take the offer – telling me Sprockets is great. Not only that, but that her agency has the exclusive contract for Sprockets, and if I let this pass up I won’t see the job advertised anywhere else. I already know that’s a lie, of course, because in the meantime I’ve already been contacted on LinkedIn by another agency representing the same role. I repeat that I’d like to have the Spacemax interview and I’ll let her know after that.

Unfortunately for me, the interview with Spacemax goes great too, and they make me an offer on the spot. Now I have a difficult decision to make – I like both companies and there’s not a lot to separate them in terms of salary or benefits. Luckily my phone was on silent mode, because it turns out Jane was calling me several times during the interview to see if I’d made up my mind about Sprockets yet. As I’m walking out of Spacemax, my phone rings again and it’s Jane. I tell her I’ve been made an offer and will need time to think.

This doesn’t go down well. She assures me Sprockets are a great company, with a bright future, and I’d do really well there and she’s not just saying that, and then goes into a little three minute sales pitch about why this place is great and how I should definitely take the job. Incredibly when she’s finished, she says to me: ‘So what do you think now?‘ Wait, what? Did she think a couple of minutes of off-the-cuff pressure selling was going to sway me into accepting?

Because I like to keep everyone in the loop, I let both companies know I have competing offers and I’ll be deciding shortly. I make it clear this is not a tactic to play one offer off against the other – I’d just like a little time to consider. It’s damn tricky, I really do like them both.

Jane keeps calling me, Sprockets are keen to convince me and various senior members of their team call me individually to talk through their take on the business and why it’d be great if I joined them. After each call Jane is back onto me –  I am now dreading having to pick up the phone to her – to see if any of it has done the trick in swaying me. She tells me Sprockets is much better than Spacemax, and has a better future ahead. She also tells me that she was so confident that I was the right person for the job that I was the only candidate she put forward.  I wish I could say this is a bit of embellishment for the sake of the story, but it’s not. If I’m the only candidate she’s put forward, it’s because she couldn’t find anyone else – or perhaps they’d been snapped up by the other, much more competent agency. I wish I had.

After deliberating, I decide I’m going with Spacemax. I thank Sprockets profusely for their consideration and tell them genuinely that it was an incredibly marginal and difficult choice, and wished them the very best for the future.

I receive the following email from Jane:

Thanks for your email. Really disappointing but you’ve made your decision.

I wish you every success and if anything changes please get in touch.

Do you know of anyone else that may be interested?

Sheesh! Crass and absurd.

But wait, the story isn’t quite over yet. I start with Spacemax a week later (the turnaround on interview and starting was crazy!), and precisely 5 days into my new job I get the following email from Jane:

Hi Pete

How are you? How’s the new role going? Hope it’s all working out.

I’d love the opportunity to work with SpaceMax and wondered if you could help me/introduce me to right person to discuss recruitment?

Speak soon

Sweet Jesus. I ignore this, and a couple of weeks later I see that I have a missed call from Jane. I email her, asking what the call was about. She replies to repeat her request for an introduction to be able to recruit for us. Having now finally had enough, I tell Jane that I felt her approach was completely inappropriate, particularly in light of the fact she was so recently pushing me very hard NOT to work for Spacemax.

I believe that even had I been interacting with a credible, competent agency, I’d have made the same decision, but wow Jane did not make it easy and I can only imagine that Sprockets would have been furious if they’d known they were being represented by someone so unprofessional.

If you ever want to hire anyone, for anything, don’t do this.

On the flip side, here’s what the other recruiter representing Spacemax said to me about my competing offers:

Obviously if you go for Spacemax it is in my interests, but I appreciate it’s a decision that’s entirely up to you and Sprockets are another good company.

Much, much better.

Ansible, DevOps

So you want a simple Slack failure handler for Ansible to ping your alerts channel whenever a deployment fails. Despite a thorough search I couldn’t find any examples that did this adequately, and even the solution here is a little constrained. The basic requirements are:

  1. A Slack notification on any task failure.
  2. The name of the task.
  3. The name of the host.
  4. The error debug message.

Sadly there isn’t one global failure handler configuration in Ansible. I investigated the native Slack Callback plugin available in Ansible 2.x (and not to be confused with the existing Slack module) but this seemed to be more of a generic passthrough for outputting all plays into Slack, failed or not, which isn’t what I wanted. After discarding the idea of a custom callback plugin for my purposes (which could work, but felt overly complex), I settled on a per-playbook failure role.

This is not an actual ‘handler’ as per Ansible parlance, but I was coming from experience of Chef where I could have a global failure handler baked into the Chef-client config.

To make this work we need to use Playbook Blocks (as of 2.0) and essentially enclose the entire playbook in a block/rescue. The main hassle here is that block does not support the top-level pre-task role block (and wouldn’t catch any failures therein), and so I had to convert all of my role calls to tasks that used include_role instead.

A simple playbook example looks as follows:


- hosts: "{{ target_host | default('') }}"
  gather_facts: true

  - block:
    - include_role:
        name: install_app
    - name: Greet the world
      shell: echo "hello world!"
    - fail:
       msg: "I've gone and failed the play!"
      - include_role:
          name: slack_handler
          tasks_from: failure

And in my slack_handler role (for reusability):


- name: Notify Slack of Playbook Failure
    username: 'Ansible'
    color: danger
    token: "{{ slack_webhook.split('')[1] }}"
    channel: "#deployment-alerts"
    msg: "Ansible failed on *{{ ansible_hostname }} ({{ inventory_hostname }})* \n
    *Task*: {{ }} \n
    *Action*: {{ ansible_failed_task.action }} \n
    *Error Message*: \n ```{{ ansible_failed_result | to_nice_json }}``` "
  delegate_to: localhost

ansible_failed_task and ansible_failed_result are two currently painfully undocumented (shout-out to Brian Coca for pointing me in the right direction) but delightfully detailed variables that are populated on playbook failure. ansible_failed_task is a map that contains a lot of data, so you may want to add additional debug for your purposes. The raw error message is single-line JSON so we prettify it before sending it using the Slack module using our token. The string split is a bit of an ugly hack to extract the webhook token part from the full webhook URL which is used elsewhere in my plays and passed at deploy time. For some reason the Slack module requires only the token rather than the full URL, in contrast to a lot of other integrations that want the whole thing.

The remaining hassle is the need to implement this per playbook, but aside from the possibility of a custom callback plugin this seems like the simplest way to implement this in Ansible currently.