Lockable Resources

Jenkins Plugin GitHub release Jenkins Plugin Installs Build Status GitHub license Maintenance Crowdin Join the chat at https://gitter.im/jenkinsci/lockable-resources

This plugin allows defining lockable resources (such as printers, phones, computers, etc.) that can be used by builds. If a build requires a resource which is already locked, it will wait for the resource to be free.


Support

“Open source” does not mean “includes free support”

You can support the contributor and buy him a coffee. coffee Every second invested in an open-source project is a second you can't invest in your own family / friends / hobby. That`s the reason, why supporting the contributors is so important.

Thx very much for supporting us.


Usage

Adding lockable resources

  1. In Manage Jenkins > Lockable Resources (under the Configuration category)
  2. Click Add Resource in the top bar

Each lockable resource has the following properties:

  • Name - A mandatory name (not containing spaces!) for this particular resource, i.e. DK_Printer_ColorA3_2342
  • Description - Optional verbose description of this particular resource, i.e. Printers in the Danish Office
  • Labels - Optional space-delimited list of Labels (A label can not containing spaces) used to identify a pool of resources. i.e. DK_Printers_Office Country:DK device:printer, DK_Printer_Production, DK_Printer_Engineering
  • Reserved by - Optional reserved / locked cause. If non-empty, the resource will be unavailable for jobs. i.e. All printers are currently not available due to maintenance. This option is still possible, but we recommend to use the Manage Jenkins > Lockable Resources page

A resource is always the one thing that is locked (or free or reserved). It exists once and has an unique name (if we take the hardware example, this may be office_printer_14). Every resource can have multiple labels (the printer could be labeled dot-matrix-printer, in-office-printer, a4-printer, etc.). All resources with the same label form a "pool", so if you try to lock an a4-printer, one of the resources with the label a4-printer will be locked when it is available. If all resources with the label a4-printer are in use, your job waits until one is available. This is similar to nodes and node labels.

Using a resource in a freestyle job

When configuring the job, select This build requires lockable resources. Please see the help item for each field for details.

Using a resource in a pipeline job

When the lock step is used in a Pipeline, if the resource to be locked isn't already defined in the Jenkins global configuration, an ephemeral resource is used: These resources only exist as long as any running build is referencing them.

Examples:

Acquire lock

Example for scripted pipeline:

echo 'Starting'
lock('my-resource-name') {
  echo 'Do something here that requires unique access to the resource'
  // any other build will wait until the one locking the resource leaves this block
}
echo 'Finish'

Lock with a reason

You can specify a reason why the resource is being locked. This is displayed in the lockable resources UI while the resource is locked:

lock(resource: 'staging-server', reason: 'Running integration tests') {
    echo 'Deploying to staging'
}

The reason helps other users understand why a resource is unavailable.

Example for declarative pipeline:

pipeline {
  agent any

  stages {
    stage("Build") {
      steps {
        lock(label: 'printer', quantity: 1, resource : null) {
          echo 'printer locked'
        }
      }
    }
  }
}

Setting quantity to null, 0 or a smaller number, all available resources of that label are locked at once.

Take first position in queue

lock(resource: 'staging-server', inversePrecedence: true) {
    node {
        servers.deploy 'staging'
    }
    input message: "Does ${jettyUrl}staging/ look good?"
}

It is not allowed to mixed inversePrecedence and priority.

start time job resource inversePrecedence
00:01 j1 resource1 false
00:02 j2 resource1 false
00:03 j3 resource1 true
00:04 j4 resource1 false
00:05 j5 resource1 true
00:06 j6 resource1 false

Resulting lock order: j1 -> j5 -> j3 -> j2 -> j4 -> j6

lock (queue) priority

lock(resource: 'staging-server', priority: 10) {
    node {
        servers.deploy 'staging'
    }
    input message: "Does ${jettyUrl}staging/ look good?"
}
start time job resource priority
00:01 j1 resource1 0
00:02 j2 resource1 <default == 0>
00:03 j3 resource1 -1
00:04 j4 resource1 10
00:05 j5 resource1 -2
00:06 j6 resource1 100

Resulting lock order: j1 -> j6 -> j4 -> j2 -> j3 -> j5

Resolve a variable configured with the resource name and properties

lock(label: 'some_resource', variable: 'LOCKED_RESOURCE') {
  echo env.LOCKED_RESOURCE
  // properties of the first acquired resource are available both without an index ...
  echo env.LOCKED_RESOURCE_PROP_ABC
  // ... and with the numeric index (0 == first acquired resource)
  echo env.LOCKED_RESOURCE0_PROP_ABC
}

The properties of the first acquired resource are exposed both with the 0 index (LOCKED_RESOURCE0_PROP_ABC) and without it (LOCKED_RESOURCE_PROP_ABC). The un-indexed form is a convenience alias for the first acquired resource and is always present, even when multiple resources are locked.

When multiple locks are acquired, each will be assigned to a numbered variable:

lock(label: 'some_resource', variable: 'LOCKED_RESOURCE', quantity: 2) {
  // comma separated names of all acquired locks
  echo env.LOCKED_RESOURCE

  // first lock
  echo env.LOCKED_RESOURCE0
  echo env.LOCKED_RESOURCE0_PROP_ABC
  // the un-indexed alias always refers to the first lock's properties
  echo env.LOCKED_RESOURCE_PROP_ABC

  // second lock
  echo env.LOCKED_RESOURCE1
  echo env.LOCKED_RESOURCE1_PROP_ABC
}

Note: The combined variable joins the names with commas, so splitting it is only safe while no resource name contains a comma. Resource names may contain one. Prefer the numbered variables (LOCKED_RESOURCE0, LOCKED_RESOURCE1, …) whenever a lock can resolve to more than one resource.

Skip executing the block if there is a queue

lock(resource: 'some_resource', skipIfLocked: true) {
  echo 'Do something now or never!'
}

Update resource properties

The updateLock step allows pipelines to dynamically manage lockable resources without using the Jenkins UI.

Create a new resource:

updateLock(resource: 'my-resource', createResource: true, setLabels: 'env-test team-a')

Modify labels on an existing resource:

// Replace all labels
updateLock(resource: 'my-resource', setLabels: 'new-label1 new-label2')

// Add labels (keeps existing)
updateLock(resource: 'my-resource', addLabels: 'additional-label')

// Remove specific labels
updateLock(resource: 'my-resource', removeLabels: 'old-label')

// Add and remove in one step
updateLock(resource: 'my-resource', addLabels: 'new', removeLabels: 'old')

Set a note on a resource:

updateLock(resource: 'my-resource', setNote: 'Updated by build #${BUILD_NUMBER}')

Delete a resource:

updateLock(resource: 'my-resource', deleteResource: true)

Note: Resources cannot be deleted while locked, queued, or reserved.

Detailed documentation can be found as part of the Pipeline Steps documentation.

Jenkins label parser allows sophisticated filtering

The plugin uses the Jenkins-internal label parser for filtering lockable resources. A full list of supported operators and syntax examples can be found in the official documentation.

lock(label: 'labelA && labelB', variable : 'someVar') {
    echo 'labelA && labelB acquired by: ' + env.someVar;
}

lock(label: 'labelA || labelB', variable : 'someVar') {
    echo 'labelA || labelB acquired by: ' + env.someVar;
}

lock(label: 'labelA || labelB || labelC', variable : 'someVar', quantity : 100) {
    echo 'labelA || labelB || labelC acquired by: ' + env.someVar;
}

Multiple resource lock

lock(label: 'label1', extra: [[resource: 'resource1']]) {
	echo 'Do something now or never!'
}
echo 'Finish'
lock(
  variable: 'var',
  extra: [
    [resource: 'resource4'],
    [resource: 'resource2'],
    [label: 'label1', quantity: 2]
  ]
) {
  def lockedResources = env.var.split(',').sort()
  echo "Resources locked: ${lockedResources}"
}
echo 'Finish'

More examples are here.


Dynamic resource behavior

When new resources are added to the system, waiting jobs can automatically pick them up:

  • Pipeline jobs waiting in the lock step queue are re-evaluated immediately
  • Freestyle jobs waiting in the Jenkins build queue are re-evaluated via queue maintenance

This allows you to dynamically add resources to your resource pool without requiring waiting jobs to be restarted.

Example

If a job is waiting for a resource with label printer and all existing printer resources are locked, adding a new resource with the label printer will allow the waiting job to acquire it immediately.

Limitations

  • Modifying labels on existing resources does NOT trigger re-evaluation. Only adding new resources triggers waiting jobs to re-evaluate. If you change labels on an existing resource, you can manually call LockableResourcesManager.get().refreshQueue() via Script Console to notify waiting jobs.
  • Removing resources does not affect waiting jobs (they continue waiting for other resources with matching criteria).

Node mirroring

Lockable resources plugin allow to mirror nodes (agents) into lockable resources. This eliminate effort by re-creating resources on every node change.

That means when you create new node, it will be also created new lockable-resource with the same name. When the node has been deleted, lockable-resource will be deleted too.

Following properties are mirrored:

  • name.
  • labels. Please note, that labels still contains node-name self.
  • description.

To allow this behavior start jenkins with option -Dorg.jenkins.plugins.lockableresources.ENABLE_NODE_MIRROR=true or run this groovy code.

System.setProperty("org.jenkins.plugins.lockableresources.ENABLE_NODE_MIRROR", "true");

Note: When the node has been deleted, during the lockable-resource is locked / reserved / queued, then the lockable-resource will be NOT deleted.


Remote lockable resources

A resource can be locked from a different Jenkins controller. This is for hardware that only one controller owns but several need to use - a test rig, a licence dongle, a PLC - without duplicating the resource definition on every controller and hoping the duplicates never disagree.

The controller that owns the resource is the server. Any controller that locks it is a client. A controller can be both. From the pipeline it is one extra argument:

lock(resource: 'plc-01', serverId: 'server-a') {
  echo 'This build holds plc-01, which lives on server-a'
}

Everything else about lock() behaves as it does locally: label, quantity, variable, extra, priority, skipIfLocked, inversePrecedence and the allocate timeout all work the same way, and the queue on the server is shared between local and remote waiters. A remote lock fails closed: if the server cannot be reached, the step fails rather than running the block unlocked.

On the server (the controller that owns the resource)

  1. Manage JenkinsConfigure SystemLockable Resources (Server)Enable remote API.
  2. Set Expose label(s) - one or more labels, separated by spaces. Only resources carrying one of them can be seen or locked remotely. It is empty by default, which exposes nothing: an enabled API on its own hands out no resources.
  3. Create a Jenkins user for the clients, give it the Lockable Resources / RemoteUse permission, and generate an API token for it (UserSecurityAPI Token).

A client holding a lock heartbeats while it works. If the heartbeats stop for 60 seconds the server marks the lease STALE and says so on the lockable resources page, but it does not release it: a client that has gone quiet may still be driving the hardware, and handing the resource to someone else on that guess is the one outcome a lock exists to prevent. Releasing a stale lease is an administrator's decision, taken from that page.

On the client (the controller that locks it)

Add the server under Lockable Resources (Client)Remote servers: a Server ID (the name used in serverId:), the Remote Jenkins URL, and a Credentials ID - a username/password credential holding the server user's name and API token. Set Client ID to something that identifies this controller; it is what the server shows next to the lock.

Delegated mode

Setting Forced server ID on a client sends every lock() on that controller to the named server, including locks that name no serverId at all. It also overrides a serverId a pipeline did name, which is the point of it - the administrator decides where locks go - but it does mean a pipeline cannot opt out. The override is logged in the build log.

Maintenance

Turning off Accept new acquire requests on a server refuses new acquires while leaving locks already held alone: heartbeats and releases keep working, and waiting clients retry rather than fail. Use it to drain a server before taking it down.

A remote can also be switched off from the client side (Enabled on the entry) without deleting its configuration.

Locking from something that is not Jenkins

The same locks are reachable over REST, so scripts and non-Jenkins tools can take them too. See remote-api-curl.md for the endpoints and a worked example.


Improve performance

To be safe thread over all jobs and resources, need to be all operations synchronized. This might lead to slow down your jobs. The jenkins self, has low CPU usage, but your jobs are very slow. Why?

The most time are spend to write the lockable-resources states into local file system. This is important to get the correct state after Jenkins reboots.

To eliminate this saving time has been added a property DISABLE_SAVE.

  • The best way is to use it with JCaC plugin. So you are sure, you have still correct resources on Jenkins start.
  • When you set pipeline durability level to PERFORMANCE_OPTIMIZED, it makes also sense to set this property to true.

Note: Keep in mind, that you will lost all your manual changes!

Note: This option is experimental. It has been tested in many scenarios, but no body know.

To allow this behavior start jenkins with option -Dorg.jenkins.plugins.lockableresources.DISABLE_SAVE=true or run this groovy code.

System.setProperty("org.jenkins.plugins.lockableresources.DISABLE_SAVE", "true");

Detailed lock cause

Tle plugin step lock() will inform you in the build log detailed block cause. The size of cause depends on count of ordered resources and size of current queue. To eliminate big unreadable logs we limited the size. To see all cause change the properties as follow:

System.setProperty("org.jenkins.plugins.lockableresources.PRINT_BLOCKED_RESOURCE", "-1");
System.setProperty("org.jenkins.plugins.lockableresources.PRINT_QUEUE_INFO", "-1");

PRINT_BLOCKED_RESOURCE means how many of ordered resources are printed. Per default 2. PRINT_QUEUE_INFO how many queue items are printed. Per default 2.

0 means disabled -1 means all / unlimited.

Configuration as Code

This plugin can be configured via Configuration-as-Code.

Example configuration

unclassified:
  lockableResourcesManager:
    declaredResources:
      - name: "S7_1200_1"
        description: "S7 PLC model 1200"
        labels: "plc:S7 model:1200"
      - name: "S7_1200_2"
        labels: "plc:S7 model:1200"
      - name: "Resource-with-properties"
        properties:
          - name: "Property-A"
            value: "Value"

Properties description, labels and properties are optional.

Fields like reservedBy, reservedTimestamp or note are not supported, they will be ignored.

Remote locking

Remote locking is configured under the same key. A controller may act as a server, as a client, or as both, so the two groups are independent:

unclassified:
  lockableResourcesManager:
    # Server side: this controller lets others lock its resources
    remoteApiEnabled: true
    exposeLabel: "remote-enabled"
    acceptNewAcquires: true      # maintenance switch; default is true

    # Client side: this controller locks resources on others
    clientId: "jenkins-client-1"
    forcedServerId: ""           # non-empty enables delegated mode
    remotes:
      - serverId: "server-a"
        url: "http://jenkins-server-a:8080/jenkins"
        credentialsId: "server-a-token"
      - serverId: "server-b"
        url: "https://jenkins-server-b.example.com/"
        credentialsId: "server-b-token"
        enabled: false           # configured but not used; default is true

    declaredResources:
      - name: "plc-01"
        labels: "plc remote-enabled"

Only resources whose labels include one of the exposeLabel labels are reachable remotely, so plc-01 above is, and a resource labelled only plc is not.


lockable-resources overview

The page Manage Jenkins > Lockable Resources (also accessible at <jenkinsRootUrl>/lockable-resources/) provides a management dashboard for all lockable resources. When accessed from Manage Jenkins, it renders with the sidebar layout; when accessed directly via the URL, it renders as a standalone page.

The page has four tabs: Overview, Resources, Labels, and Queue.

Overview

Provides at-a-glance stats including a resource donut chart (locked/reserved/free), utilization percentages, top labels, and queue metrics (longest wait, most contended resource). Cards are clickable to jump to the relevant tab.

Resources

Provides a status overview of all resources and actions to change resource status.

Resources can be created, edited, and deleted directly from this tab using the Add Resource button and per-row edit/delete actions (requires CONFIGURE permission). Bulk actions (reserve, unreserve, unlock, steal) are available via multi-select checkboxes.

Name Permission Description
Add Resource CONFIGURE Creates a new lockable resource.
Edit (Gear Icon) CONFIGURE Edits an existing resource (name, description, labels, properties).
Delete CONFIGURE Deletes a resource (only if not locked, reserved, or queued).
Reserve RESERVE Reserves an available resource for currently logged user indefinitely (until that person, or some explicit scripted action, decides to release the resource).
Unreserve RESERVE Un-reserves a resource that may be reserved by some person already. The user can unreserve only own resource. Administrator can unreserve any resource.
Unlock UNLOCK Unlocks a resource that may be or not be locked by some job (or reserved by some user) already.
Steal lock STEAL Reserves a resource that may be or not be locked by some job (or reserved by some user) already. Giving it away to currently logged user indefinitely (until that person, or some explicit scripted action, later decides to release the resource).
Reassign STEAL Reserves a resource that may be or not be reserved by some person already. Giving it away to currently logged user indefinitely (until that person, or some explicit scripted action, decides to release the resource).
Reset UNLOCK Reset a resource that may be reserved, locked or queued.
Note RESERVE Add or edit resource note.

Labels

Provides an overview over all lockable-resources labels.

Note: Please keep in mind, that lockable-resource-label is not the same as node-label!

Queue

Provides an overview of currently queued requests with server-side pagination for performance with large queues. A request is queued by the pipeline step lock(). When the requested resource(s) is currently in use (not free), then any new request for this resource will be added into the queue.

A resource may be requested by:

  • name, such as in lock('ABC') { ... }
  • label, such as in lock(label : 'COFFEE_MACHINE')

Note: Please keep in mind that groovy expressions are currently supported only in free-style jobs. Free-style jobs do not update this queue and therefore can not be shown in this view.

Note: An empty value in the column 'Requested at' means that this build has been started in an older plugin version - 1117.v157231b_03882 and early. In this case we cannot recognize the timestamp.

Permissions

The plugin defines its own permission group under Manage Jenkins > Security > Authorization:

Permission Implied by Description
View Jenkins.READ See the lockable resources page and resource status.
Configure Jenkins.ADMINISTER Create, edit, and delete resources.
Unlock Jenkins.ADMINISTER Manually unlock a resource.
Reserve Jenkins.ADMINISTER Reserve or unreserve a resource.
Steal Jenkins.ADMINISTER Steal a lock from another build.
Queue Jenkins.ADMINISTER Reorder the lock wait queue.
RemoteUse Jenkins.ADMINISTER Acquire and release locks through the remote lock REST API. Grant it to the machine users whose API tokens remote client controllers use.

All mutating permissions default to ADMINISTER, preserving existing behavior. Admins can delegate individual permissions to specific roles without granting full admin access.


Upgrading from 1102.vde5663d777cf

Due an issue is not possible anymore to read resource-labels from the config file org.jenkins.plugins.lockableresources.LockableResourcesManager.xml, which is generated in the release 1102.vde5663d777cf

This issue does not effect instances configured by Configuration-as-Code plugin.

A possible solution is to remove the <string> tags from your org.jenkins.plugins.lockableresources.LockableResourcesManager.xml config file manually, before you upgrade to new version (Keep in mind that a backup is still good idea).

Example:

change this one

<labels>
  <string>tests-integration-installation</string>
</labels>

to

<labels>
  tests-integration-installation
</labels>

Changelog


Report an Issue

Please report issues and enhancements through the Jenkins issue tracker in GitHub


Contributing

Contributions are welcome, please refer to the separate CONTRIBUTING document for details on how to proceed! Join Gitter channel to discuss your ideas with the community.


License

All source code is licensed under the MIT license. See LICENSE