AWS Lambda Cloud

This Jenkins plugin uses AWS Lambda to host jobs execution.

Jenkins delegates to AWS Lambdas the execution of the builds on Lambda based agents (runtimes). Each Jenkins build is executed on a Lambda execution that is released at the end of the build.

• use GitHub Issues to report issues / feature requests

Given the limitations of the AWS Lambda engine :

• jobs can't exceed a duration of 15 minutes
• jobs can't use more than 512 MB of storage

• Jenkins with at least version 1.635 (beware of using Lambda agents with the remoting library compatible with your jenkins)
• an AWS Account

IMPORTANT : Jenkins will by default mark as unavailable any agent with low space on disk or low space in temp. Given the limited space of 512MB available in the writable temp filesystem of a Lambda, you must adapt or disable this behavior under the "Nodes Configuration" section of your Jenkins (url: /computer/configure).

Use the Jenkins plugin manager to install the AWS Lambda Cloud

See this repository on how to build and deploy Lambda Agents for Jenkins.

Assuming you already deployed at least one Lambda agent

Jenkins will need to be allowed to invoke Lambda Functions. So you must :

• configure AWS Credentials in Jenkins using the CloudBees AWS Credentials plugin
• unless your Jenkins is running as an AWS Resource (ECS task, EKS pod, EC2 instance)

Those credentials will match an IAM User or a IAM Role which will need to have the following permissions :

Version: '2012-10-17'
Statement:
  - Effect: Allow
    Action:
      - lambda:ListFunctions
    Resource:
      - "*"
  - Effect: Allow
    Action:
      - lambda:InvokeFunction
    Resource:
      - "*"

See below in the "Advanced Configuration" part for finer-grained IAM permissions.

1. Install this plugin
2. Go the Cloud configuration
   1. on older Jenkins versions, go at the bottom of /configure page
   2. on more recent Jenkins versions, go to the /configureClouds page
3. Add a Cloud of type AWS Lambda Cloud
4. Configure the cloud
   1. use AWS Credentials or let is empty if your Jenkins is running in an AWS Context
   2. configure the region
   3. optional: configure the Jenkins URL (will default to Jenkins URL) in cases where your need internal communications. The Lambda functions must be able to reach thsi URL.
   4. Agent connection timeout : The time in seconds to wait before giving up on an agent connection (typically idle agent).
5. Add a Lambda Function
   1. Set a label (to reuse in your jobs/pipelines)
   2. Select the Function Name (the list is dynamically retrieved from AWS)
6. Add as much functions as you need (type, tools, size, ...)

If you need to declare functions in other contexts (credentials, regions, ...), you will have to create another Cloud.

using basic groovy script

Basic configuration using default credentials and default region if Jenkins is running as an ECS task or an EC2 instance with 2 samples lambdas :

• jnlp-agent-git-bash with label lambda-git
• jnlp-agent-git-bash-node with label lambda-node

import io.jenkins.plugins.aws.lambda.cloud.LambdaCloud;
import io.jenkins.plugins.aws.lambda.cloud.LambdaFunction;

import jenkins.model.Jenkins

jenkins = jenkins.model.Jenkins.get()

c = new LambdaCloud("aws-lambdas", null, '')
f = new LambdaFunction('jnlp-agent-git-bash', "lambda-git");
f2 = new LambdaFunction('jnlp-agent-git-bash-node', "lambda-node");
c.setFunctions([f, f2]);
jenkins.clouds.add(c);
jenkins.save()

It is a good practice to restrict the permissions on your Jenkins and allow it to execute only the functions that you have planned it to use. A nice way to do it would be tag your Lambda functions and apply IAM restrictions by conditions on those tags. Unfortunately, AWS Lambda does not support Authorization based on tags https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html. So to apply fine grain permissions, we will need to use the names of the functions.

In the deployment sample (serverless), all lambdas are deployed with jnlp-agent prefix.

So to limit the invocation permissions of Jenkins to only those Lambda functions, we can use this policy :

Version: '2012-10-17'
Statement:
  - Effect: Allow
    Action:
      - lambda:ListFunctions
    Resource:
      - "*"
  - Effect: Allow
    Action:
      - lambda:InvokeFunction
    Resource:
      - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:jnlp-agent-*"

this is the Cloudformation compatible format of an IAM Policy. Replace !Sub, ${AWS::Region} and ${AWS::AccountId} if needed.

By default Jenkins do estimate load to avoid over-provisioning of cloud nodes. This plugin will use its own provisioning strategy by default, with this strategy, a new node is created on Lambda as soon as NodeProvisioner detects need for more agents. In worse scenarios, this will results in some extra nodes provisioned on Lambda, which will be shortly terminated.

If you want to turn off this Strategy you can set SystemProperty io.jenkins.plugins.aws.lambda.cloud.lambdaCloudProvisionerStrategy.disable=true

1. check that you have defined the label xxxxx in one of your Lambda Function. You can provide a list of labels for the same function separated by whitespaces (not commas).
2. Check the Jenkins logs or Lambda Agent Logs (see below)

Look above at the important notice in "Jenkins Required Configuration".

Depending on your system, you can consult logs :

• on the /log/all url
• or configure a recorder on the io.jenkins.plugins.aws.lambda.cloud package with at least INFO level

If you see errors like is not authorized to perform: lambda:InvokeFunction on resource:, check the permissions of the credentials you set on the cloud (explicitly or implicitly).

Look at the Cloudwatch console.

TO COME : a build wrapper to give you a quick access to the lambda log group/stream of execution.