×
Find plugins

CA-APM
1.0Minimum Jenkins requirement: 1.642ID: ca-apm

Allows users to collect CA APM performance data and publish it in Jenkins and allows the users to publish build number, status etc into CA-APM APM Team Center(ATC)

For ca-apm 1.x please refer to CA APM Plugin 1.x


CA APM Plugin 2.x and later

Work in progress..


Jenkins is a tool to automate continuous delivery pipelines. A most common use of Jenkins is to run tests against the new builds of an application. The CA APM Jenkins plug-in extension automates comparing the performance of builds. The plug-in provides abstraction to run the plug-in after any LoadGenerator, define multiple comparison strategies and define multiple output handlers.

  • LoadGenerator :You can use Jmeter and CA Balzemeter as the load generators
  • Comparison Strategies: CA Jenkins plugin supports the following comparison strategies:
    • Mean Latency: Compares the average value of a metric of a transaction with its corresponding value of benchmark build.
    • Static Threshold: Compares the value of the metric with the corresponding threshold value defined in the configuration.
  • Output Handlers: Email HTML Report, JSON, and Chart Output, Histogram Chart Output

Features

  • Ability to fetch multiple metrics using regex as KPI for tracking
  • Ability to configure multiple fail conditions
    • Metric A less than or greater than a constant value OR
    • Metric A in comparison to Metric B
  • Ability to send notification via Email (if smtp server is configured)
  • Ability to view KPI's per build or cross build dashboard in Jenkins&nbsp
  • Ability to decorate APM Team Center 10.1 nodes using custom attributes like build status, number and date
  • Ability to just notify via email: If "justEmail" is checked the build will NOT be marked as fail even if the conditions are met but only a notification will be sent
  • Ability Jump to Webview in context (for both metric and time range) from build or x-build dashboard

Requirements

Jenkins

Jenkins version 2.60.1 or newer

CA APM Jenkins plug-in Packaging

The CA APM Jenkins plug-in package contains the following folder and files:

  • Properties
    • performance-comparator.properties
  • Repo
    • ca-apm-jenkins.hpi (HPI file)
    • ca-apm-api.jar
    • ca-apm-core.jar
    • ca-apm-default-strategy.jar
  • javadoc directory - Contains java documentation about comparison-strategy, output-handler, and available helper utilities

Setup

Install the Jenkins plug-in

This section describes the steps to install the plug-in. You can install the Jenkins plug-in from the Jenkin's Manage Plug-ins page.

Follow these steps:

  1. Select JenkinsManage JenkinsManage plugins.

    The Manage Plug-in page opens.
  2. Select the Advanced tab.

    The HTTP Proxy Configuration page opens.
  3. Browse and upload the performance-comparator-jenkins.hpi file available in the Repo directory.

    Wait for the installation to complete.

    Note: Ensure that no jobs are running

  4. Restart the Jenkins when installation is complete.

Using the Jenkins Pipeline Syntax to Run the CA APM Jenkins plug-in

  Jenkins job can be configured with any load generator to generate the load on the application, as below, before running the ca-apm plugin. 

  Example : 

  1. Blazemeter

    To use BlazeMeter in pipeline syntax you need to have BlazeMeter Plugin

    pipeline {
     agent any
     stages {
        stage('Blazemeter'){
            steps{
               script {
                loadGeneratorName = env.STAGE_NAME;
                loadGeneratorStartTime = System.currentTimeMillis();
                blazeMeterTest credentialsId:'<credentials_id>',
                serverUrl:'https://blazemeter.ca.com',
                testId:'<test_id>',
                notes:'',
                sessionProperties:'',
                jtlPath:'',
                junitPath:'',
                getJtl:false,
                getJunit:false
                loadGeneratorEndTime = System.currentTimeMillis();
              }
            }
        }
        stage('CAAPMPerformanceComparator') {
            steps {  
                 caapmplugin performanceComparatorProperties: "${env.WORKSPACE}/properties/performance-comparator.properties",
                 loadGeneratorStartTime: "$loadGeneratorStartTime",
                 loadGeneratorEndTime: "$loadGeneratorEndTime",
                 loadGeneratorName: "$loadGeneratorName"
            }
        }   
     }
    }
  2. Jmeter

    Sample pipeline scripts with jmeter.  Please refer the TBD

    pipeline {   
       agent any     
       stages {
    	stage('JMeter'){ 
    		steps{   
                script {
                     loadGeneratorName = env.STAGE_NAME;
                     loadGeneratorStartTime = System.currentTimeMillis();
                     sh "${env.WORKSPACE}\\<jmeter_home>\\bin\\jmeter.sh -n -t <jmx_file_path>"
                     loadGeneratorEndTime = System.currentTimeMillis();
                }              
            }
        }            
       stage('CAAPMPerformanceComparator') {
          steps {  
               caapmplugin performanceComparatorProperties:"${env.WORKSPACE}/properties/performance-comparator.properties",
               loadGeneratorStartTime: "$loadGeneratorStartTime",
               loadGeneratorEndTime: "$loadGeneratorEndTime",
               loadGeneratorName: "$loadGeneratorName"
         }
       }   
      }
    }



    Note: Each Jenkins job will be binded with the parameters loadGeneratorName, loadGeneratorStartTime, loadGeneratorEndTime of the Load Generator and the same parameters will be passed to the plugin.

Follow these steps:

  1. Add the predefined stage in your existing pipeline job.
  2. Run the Jenkins job.

Important: The Jenkins plug-in and the properties files must be in the same machine.

Plugin Configuration

Configure properties in performance-comparator.properties file.

Download the performance-comparator.properties file from  http://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/ca-apm/ca-apm-dist/<version>/ca-apm-dist-<version>.zip

Please refer the Configure the CA APM Jenkins Plugin Properties

Default Comparison Strategies and Output Handler Available with the plug-in

The plug-in has the following built-in Comparison Strategies and Output Handlers.

List of Available Comparison-Strategies

By default the following strategies are executed:

  • MeanLatencyStrategy 
  • staticThresholdStrategy

If you do not want to execute the default strategy, remove the corresponding strategy name from comparisonstrategies.list property value list.

For example, for meanLatencyStrategy, it can be any RegEx like *Frontend.*Health: Average Response Time \\(ms\\). Do not remove Health: Average Response Time \\(ms\\), else it looses its logical meaning.

By default, the following comparison-strategies are available with the plug-in:

Mean Latency Comparison Strategy

The strategy compares the average value of the metric between current build and benchmark build with threshold of five percentage. You can set a custom value for the <metricName>.threshold property. The following code shows the sample configuration:

#MeanLatency Comparison-Strategy
#threshold value, e.g.,2. the build will fail,if the build.fail property value is set true and if the metric's avg value difference
#between current and benchmark build cross this value
cpu.threshold=5
#agent name of the application,  e.g., .* means any agent
cpu.agentspecifier=.*
#metric path , it will be specific to agent, application. e.g, .*CPU.*Processor 0:Utilization % \\(aggregate\\)
cpu.metricspecifier=.*CPU.*Processor 0:Utilization % \\(aggregate\\)
#comparator class name excluding the "ComparisonStrategy", e.g., MeanLatency for MeanLatencyComparisonStrategy
cpu.comparator=MeanLatency
#list of output handlers for this strategy. available output handlers : plaintextemail,jsonfilestore,chartoutputhtml,histogramoutputhtml
cpu.outputhandlers=plaintextemail,jsonfilestore,chartoutputhtml,histogramoutputhtml


Static Threshold Strategy

The strategy compares your Concurrent Invocation metric of transactions of the current build with the given threshold value (2).

#StaticThreshold Comparison-Strategy
#threshold value,e.g., 2.  it will be compared with the average value of metric for current  build
concurrentinvocations.threshold=2
#agent name of the application, e.g., .*
concurrentinvocations.agentspecifier=.*
#metric path , it will be specific to agent, application. e.g., .*Business Segment.*Health:Concurrent Invocations
concurrentinvocations.metricspecifier=.*Business Segment.*Health:Concurrent Invocations
#comparator class name excluding the "ComparisonStrategy", e.g., StaticThreshold for StaticThresholdComparisonStrategy
concurrentinvocations.comparator=StaticThreshold
#list of output handlers for this strategy. available output handlers : plaintextemail,jsonfilestore,chartoutputhtml,histogramoutputhtml
concurrentinvocations.outputhandlers=plaintextemail,jsonfilestore,chartoutputhtml,histogramoutputhtml


List of Available Output-Handlers

The plug-in provides the following default output handlers that are mapped to all the following default comparison-strategies in the outputhandlers property. You can remove any or all of the output handlers. The output-handlers should be declared with the below property as below:

outputhandlers.list=plaintextemail,jsonfilestore,chartoutputhtml,histogramoutputhtml

Plain Text Email Report Output Handler

The handler generates a plain text email comparison report. The report contains the metrics for all the  strategies that are mapped that are in the properties file.


JSON File Output Handler

The handler generates a JSON report of the average value of the metrics comparison between current build and benchmark build as a percentage.


Chart Output Handler

The handler generates a Chart Output of comparison of performance metrics between benchmark build and current build.

The output is stored in <current_build>/workspace/chartOutput folder. The output html files are in the chartOutput/output folder. The folder contains one file for each comparison-strategy. For Example, If one comparison strategy has six metric paths that are identified, then the output file contains six charts.


Histogram Output Handler

The handler generates build to build comparison in histogram style, with the average value of each metric for each build, default it render for 10 builds.

Comparison Strategy to Output-Handler Mapping

The mapping lets you receive the output from only those comparison-strategies which are mapped to output-handler using the outputHandlers property. The communication between a comparison-strategy and output-handler happens using the Data Structure T within StrategyResult Entity.Map the comparison-strategy and output-handler using the same data format. If the mapping is incorrect, it is not processed by Output-Handler and plug-in raises a warning.

The default implementation of T is DefaultStrategyResult provided with the plug-in. All the default strategies and output-handlers communicate using the StrategyResult<DefaultStrategyResult> Entity.

Jenkins Attributes in APM Team Center

With the help of Jenkins plug-in, you can view the following Jenkins attributes as Custom attributes to each application vertex:

  • currentBuildNumber
  • benchmarkBuildNumber
  • buildStatus
  • loadGeneratorStartTime
  • loadGeneratorEndTime
  • loadGeneratorName

Extending the plugin 

Extending the plug-in lets you add new sub commands to the existing plug-in to perform new tasks. Please refer the Extending the CA APM Jenkins Plugin 

Troubleshoot the plug-in

This section discusses the error message that you see on your screen, symptom, and solution to resolve the error.

Comparator Plugin Execution Completed with failures

Symptom:

The jenkins job fails and detailed specific error is printed on the Jenkins build console.

Solution:

Review the comparison-runner.log file that is generated in <current_build>/workspace directory.


Message printed on Jenkins Console Output

Input Properties file defined in parameters does not exist, please check

Symptom:

Jenkins plug-in run fails because the specific configuration file does not exist.

Solution:

Place the file in the specified location as printed in the error message and run the plug-in.

Comparison Strategy handler for <strategy_name> is not defined

Symptom:

There is no value against the property omparator which is the comparison strategy handler.

Solution:

Provide the complete qualified class name of the comparison strategy handler.

No Agent Specifier(s) defined for <strategy_name>

Symptom:

Values are not specified against agentspecifier for a given comparison strategy.

Solution:

Provide at least one agent-specifier regular expression, at least .*

No metric specifier defined for <strategy_name>

Symptom:

Value is not specified against metricspecifier for a given comparison strategy.

Solution:

Provide a qualified correct regular expression for metric specifier using the following syntax:

Business Segment\|thieves\|[^|]+\|Health:Average Response Time \(ms\)

Warning: No output handler(s) mapped to <strategy_name>

Symptom:

The given comparison strategy is unmapped to any output-handler, so its output is not used anywhere.

Solution:

Either comment this comparison-strategy or assign a relevant output-handler to this comparison-strategy.

No output-handler(s) defined in the configuration, hence exiting Exiting means will stop running the plugin further

Symptom:

No output handlers are mentioned against the outputhandlers list.

Solution:

Provide at least one output-handler, which is mapped to at least one comparison-strategy.

class is not present in the classpath

Symptom:

The specified class is not present inside a jar in the extensions directory that is defined in the performance-comparator.properties file.

Solution:

Check if the specified class is present in the configured extensions directory.

Error in executing comparison strategy <strategy_name>

Symptom:

Detailed reason is present in comparison-runner.log file present in workspace current build directory.

Solution:

Check for detailed error and act appropriately.

Error in executing Output strategy <strategy_name>

Symptom:

Detailed reason is present in comparison-runner.log file present in the <current_build>/workspace directory.

Solution:

Check for detailed error and act appropriately.


Version history

Version 2.0-beta-0 (January 25, 2019)

  • Initial release for 2.x
ArchivesGet past versions
Labels