Slack & Rundeck Integration Using AWS Services

Overview

Your team uses Rundeck to automate ad-hoc and routine procedures in a data center or cloud environment. Your team also uses Slack for collaboration and communication. 

Typically Rundeck jobs are stuck behind a web or command-line interface and are difficult to access. 

This tutorial will give you the knowledge to create a Slack app to directly invoke a Rundeck job. 

High-Level Design

This integration uses the following components:
  • Slash commands created via a custom Slack application point to a URL hosted on the AWS API Gateway
  • AWS API Gateway resource calling an AWS Lambda function.
  • Lambda function invoking a Rundeck job via Rundeck API.
  • KMS & VPC NAT Gateway are required for security purposes.


Steps

Rundeck API Setup

Rundeck API is an extensive alternative to the web interface. These APIs are implemented using HTTPS POST and secured by an API token. 

  • Login to Rundeck and generate an API token (permissions permitting) from the Profile option.

    • Here is an example to test the job execution API.
      • Retrieve a job UUID for the Rundeck job that you want to execute. You can obtain this from the Job Definition.
      • Construct an HTTPS POST command as in the example below.

This should execute the job on the token owner's behalf. You need to get this working before proceeding to the next steps.

Typical failure points are:
  • Rundeck is not accessible via the URL (firewall, private URL)
  • The token is not valid
  • Token permissions are not sufficient to perform the API request.

Creating AWS Lambda Function

Let us now create a simple Lambda function to make an HTTPS request equivalent to the POST command tested above. 



This example will use JavaScript, but you can implement this functionality in any language that is supported by AWS Lambda.

You can let AWS create the Execution Role. If all goes well, you should see the default AWS Lambda screen.


Before proceeding do a "Test". Lambda will ask you to configure the "Test Event". It will show a default template. Simply give it a name and move on. You should see a successful execution at the top of the screen. 



Now that you have the basic lambda function working, we will replace the default code with the code that will invoke the Rundeck API. Replace the contents of the index.js file with the code given below.

Remember to replace your-token, your-rundeck-url, your-rundeck-port, and your-job-id with the correct values for your implementation.

var https = require('https');

/*
 * We will secure this later. 
 * It is not good practice to have api token in the clear
 */

let rundeck_auth_token = 'your-token';

function exec_rundeck_job(jobid) {
	return new Promise(function(resolve, reject) {
		if (jobid === "")
		{
			console.log("error : no jobid");
			reject("No rundeck job defined for request. Contact support!");
			return;
		}
		
		console.log("executing jobid " + jobid);
		var post_options = {
				host: 'your-rundeck-url',
				port: 'your-rundeck-port',
				path: '/api/1/job/' + jobid + '/executions',
				method: 'POST',
				headers: {
					'X-Rundeck-Auth-Token': rundeck_auth_token
				}
		};

		// Set up the request
		var post_req = https.request(post_options, function(res) {
			res.setEncoding('utf8');
			var body = [];

			res.on('data', function(chunk) {
				console.log('Response data: ' + chunk);
				body.push(chunk);
			});
			
			res.on('end', function() {
				console.log('Response end : ' + body);
				resolve("Response received");
			});
		});

		post_req.on('error', error => {
			reject(error);
		});

		// post the data
		post_req.write(JSON.stringify({}));
		post_req.end();

	});

}

exports.handler = async(event) => {    
	var user_message = "";
	var response = "";
	var error = false;
	var msg = "";
	
	await exec_rundeck_job("your-job-id").then(function() {
		user_message = "Tada!";
		error = false;
	})

	.catch(function(body) {
		error = true;
		user_message = body;
	});
	msg = {
			text: user_message
	};
	response = {
			statusCode: (error ? 500 : 200),
			body: JSON.stringify(msg)
	};

	return response;

};

Use Test to make sure that the lambda function is working as expected. We are going to be building on this code, so you need to make sure that it is correct.

Securing API Token via AWS KMS

It is bad practice to store API tokens in the clear. AWS Lambda provides functionality to store encrypted data in the Lambda Environment and retrieve it when the Lambda function is executed. This is a CPU expensive process and such data should be stored at the global level so that it is available after the first execution.

Here are the basic steps. This can be a little tricky if you have not done them before.

Create KMS Key -  KMS Blog for details.

Create & Read Encrypted Environment Variable - Lambda Encryption Blog for details 

Now test the function after all of these steps are completed. 

Securing Lambda Function via AWS NAT Gateway

Rundeck servers are mostly used in the development/operations environment. Most likely they are not publicly accessible and are protected by IP address allowed-listing. It is possible to achieve this in AWS by using VPC and NATs. This is a fairly advanced topic and I will provide details in another post. But here are the high-level steps.

  • Create a new VPC for Lambda functions
  • Create a NAT within this VPC
  • Create public/private subnets
  • Assign an Elastic IP to the public subnet
  • Deploy Lambda function in the VPC
  • Whitelist the Elastic IP to access Rundeck.

Creating API Gateway Resource

Slack Slash commands essential invoke an HTTP POST on a server. AWS allows us to use the API Gateway to call the Lambda function that we created. This is the simplest method without setting up a server to receive the message and then invoke the lambda function. 

  • Log in to AWS Account
  • Create API
  • Create a Resource
  • Create Method (POST)
    • Configure the Integration Request to LAMBDA_PROXY and provide details as requested. (Lambda function region/name)
    • Use the Test option to validate the method.
      • If you have created the Lambda function as indicated in the earlier section of this tutorial then you should see that your Lambda function is called, and the rundeck job executed.
  • Deploy API
    • Use the Deploy API option under the Actions banner.
    • You will need to specify a Deployment Stage. 
    • API Gateway will publish a URL for the API deployed. Make a note of this - it will be needed later when configuring the Slack Slash command.

Creating Slash Commands

Create a Slack App - "Create New App" option at https://api.slack.com/apps.

 
Specify the name of your app, and connect it to the workspace. After you create the app you will the following screen.


 
Create Slash Commands - Select the "Slash Commands" option.


Select the "Create New Command" option and enter the information requested. The Request URL will be the URL that you created in the API Gateway step.

 
 
 
The final step in this journey is to "Install App" into your workspace. Follow the instructions.

 
 
You can now go to Slack and type the /command that you created. If everything is OK then you will see a short help that you provided.


Finish typing your command, and test the complete path.  

Debugging

There are many failure points in this path. If you have been testing along the way, then the final integration should work without any issues. The only expected problem can be a "time-out" in Slack. Slack waits a maximum of 5 seconds for the entire operation to complete.

Improvements

This example implements a very simple Lambda function. All the functionality is hard-coded. You will need to make the following enhancements for this to become production worthy.
  • Check token sent from Slack to validate that it is your Slack command that is calling the Lambda function
  • Extract commands and parameters to execute the correct rundeck job.

The API Gateway also did not use an API key. If this is not being hidden behind a VPC then you should add additional security by deploying an API key and adding that functionality to your Slack command.

Additional Help

Useful instructions/directions/background is found at https://api.slack.com/tutorials/tags/slash-commands.

 


Comments

Post a Comment

Popular posts from this blog

AWS Lambda - Encrypted Environment Variable

Image
Environment Variable Creation After pasting your token you need to encrypt it. This is done by checking the  Enable Helpers  checkbox. Prior to hitting the  Encrypt  option you need to decide which Key to use. Let us use the KMS key that we created above.    When you hit the  Encrypt  option, you will need to select the key and then hit the  Encrypt  button. You will now see the encrypted token in the environment variable. Hit  Save  to complete the step. Adding Code to Read Encrypted Environment Variable The decryption is done via aws-sdk. Here are some lines that need to be added to the code that we had previously used. These lines need to be added at the top of index.js var aws = require('aws-sdk'); // Set the region aws.config.update({ region: '<your-aws-region>' }); // token is not assigned let rundeck_auth_token; // this the encrypted variable from the environment const encrypted = process.env['rundeck_auth_token']; These lines need to be added at t
Read more

AWS Key Management Service

Image
KMS Key Creation A key needs to be created in the  KMS environment  before it can be used in setting up the environment variable. Select the key administrators from the list. Finally, pick the user and/or roles that can access this policy. Each lambda has a "role" that was created or selected during the creation process. Make sure that you select the "role" correctly. If you have forgotten the "role" that is being used in the lambda function, you can retrieve it from the lambda configuration screen. Once all the parameters have been defined, the key can be saved.
Read more