×
Find plugins

BTC DSL for Pipeline
2.4.0Minimum Jenkins requirement: 1.625.3ID: btc-embeddedplatform


Description

The Jenkins Automation Plugin for BTC EmbeddedPlatform provides easy to use steps for Jenkins Pipeline enabling you to execute test workflow steps from Jenkins. The BTC EmbeddedPlatform HTML reports become available automatically on the Jenkins job page and a JUnit XML report is generated that feeds into Jenkins test trends and status overview dashboards.

The following workflow steps are available for Jenkins Pipeline:

  • Profile Load / Create / Update
  • Import of Test Cases and Stimuli Vectors
  • Import / Export of Tolerances and Input Restrictions
  • Execution of Functional Tests
  • Automatic Vector Generation & Analysis
  • Back-to-Back Test Execution
  • Regression Test Execution
  • Formal Verification
  • Formal Test
  • Configuration of Additional Coverage Goals
  • Export of different reports
  • BTC Migration Suite

For these workflow steps many additional settings can be configured (optional) in order to adapt the automated test runs to your individual project needs. In addition, two utility methods help you to connect to instances of BTC EmbeddedPlatform and to close them if again required. In addition, the Migration Suite use case can be addressed (see section Migration Suite below).

Release Notes

  Toggle Release Notes

Version

Release Notes

Min. EP Version

Max. EP Version

2.4.1

- Fixed a bug that caused Test Execution and Regression Test steps to return the wrong return code.

2.4

2.4

2.4.0

- Support for EmbeddedTester BASE (see parameter "licensingPackage" in the "startup" step for more details).

2.4

2.4

2.3.0

- Added filters for scopes, folders and testcase names. For each of the three a whitelist and blacklist can be defined as a comma separated string.

2.4

2.4

2.2.0

- Added a step to generate and export a model coverage report (RBT & B2B). Requires the Simulink V&V / Simulink Coverage license.

- Test cases (so far only test cases, no SVs, ERs, etc.) that come from a different architecture can now be imported. An automatic scope-mapping is done using the scope name (first match). Smart mapping is only performed if the direct import (which is based on the full scope path) is not possible.

2.4

2.4

2.1.0

- Adaptions for EP 2.4

2.4

2.4

2.0.11

- Added the possibility to select requirement sources as the report source for test execution reports.

2.3

2.3

2.0.10

- Fixed issue: Architecture Update doesn't work if model path changes (relative and absolute)

2.3

2.3

2.0.9

- Added CalibrationHandling as a parameter for the TargetLin profile creation

- Added Export of Tolerances and Input Restrictions to support programmatic modification & re-import

2.3

2.3

2.0.8

- Fixed a bug that prevented combined PLL strings of default and contributed coverage goals (e.g. Statements and Domain Coverage Goals together in one PLL String). 

- Fixed the CSV vector import for test cases and stimuli vectors

- Fixed an issue with backslashes in absolute paths which point to a location inside the workspace

- For Domain Coverage and Range Violation goals users can now set a scopePath of "*" to consider all scopes. Not settings a scope path will still default to the toplevel scope.

- Added the parameter "useCase" for the Code Analysis Report Step (RBT / B2B)

- Removed obsolete parameter "inputRestrictions" from the Vector Generation step. Input Restrictions can be defined through their own step.

- Users can now select the use case for the codeAnalysisReport step (RBT / B2B)

- The codeAnalysisReport now exports a csv file ("BTCCoverageOverview_B2B.csv" / "BTCCoverageOverview_RBT.csv") to the report directory which contains overall coverage percentage (Statement, Decision and MCDC). This CSV file can be used by other Jenkins plugins to display the coverage.

2.3

2.3

2.0.7

- Fixed missing parameters (reuseExistingCode, pilConfig, pilTimeout)

- Added parameters for environmentXmlPath (TL Profile Creation), testMode (grey box / black box) and startupScriptPath

2.3

2.3

2.0.3

- Adapted to EP 2.3

- A Failed Profile Creation / Profile load will now always throw an exception to break the build

- In case of failures during profile creation profile messages will be exported and made available in Jenkins (if possible)

- Fixed a bug that prevented the use of some settings for the wrapUp step.

- Fixed a bug that caused a specified PLL to be ignored.

- Jenkins: HPI plugin ("btc-embeddedplatform-plugin") is now available in the official Jenkins plugin repository

2.3

2.3

2.0.1

- The Vector Generation step now supports dummy toplevels. If the toplevel subsystem is a dummy scope (no C-function available) then the vector generation will be done on the direct children of the toplevel.

- Domain Coverage and Range Violation goals can now be added to the profile and considered during vector generation (requires additional plugins)

- Fixed a bug that caused execution records to not be available for debugging in the migration suite use case.

- archiveArtifacts and stash commands used by the btc-embeddedplatform plugin are now only called if needed and were changed to be more specific

- Fixed an issue that could cause existing profiles to be loaded in the migration suite scenario. From now on the migrationSuite will always create a new profile.

- Profile Creation can now be invoked explicitly.

- The Code Analysis Report can now be created explicitly. It was formerly created by the Vector Generation step. That's still possible but not enabled by default (controlled by property "createReport" in the btc.vectorGeneration step).

2.2p2

2.2

1.8.4

- Fixed a bug that caused the Jenkins Use Case to checkout more than one EmbeddedTester license.

2.2p2

2.2

1.8.3

- Jenkins Pipeline: Wrapping BTC steps into a "dir" closure now works as expected and relative paths are resolved according to the respective directory.

- Fixed a bug that caused the plugin to not start if only EmbeddedTester is installed. Dependencies (e.g. to EmbeddedValidator features) are now only resolved when they are required (e.g. by the btc.formalVerification step).

2.2p2

2.2

1.8.1

- Fixed a bug that occurred when using plugins which add additional information to the reports.

- Added the option to set the depth of the CV engine to infinite (0).

- Adapted for EP 2.2p2 due to licensing interface changes.

2.2p2

2.2

1.8.0

- User Defined Coverage Goals will now only be addressed by the vector generation if the license is available.

- Added two properties for vector generation. Users can now change the engines (ATG, CV, ATG+CV) and set a timeout per coverage goal.

- The export of Debug Environments can now be invoked for Test Execution, Regression Tests and Back-to-Back Tests

- A new step has been added for Formal Test Execution. This requires existing formal specifications in your profile as a prerequisite.

2.2p2

2.2

1.7.3

- Changed default goals addressed by the vector generation to match the changed default in the UI (all goals).

2.2

2.2p1

1.7.2

- Errors during Test Execution are now correctly added to the Jenkins job's console log.

- TargetLink Subsystem can now be selected. This is mandatory if more than one toplevel TargetLink subsystem exists.

- You can now choose to disable the publishing of HTML reports, the archiving of *.epp files and the publishing of test results if you choose to control this on your own.

- The BTC Pipeline Steps no longer create a visible pipeline stage because Jenkins Pipeline does provide options to handle wrapped stages. You add still add the stages in your Jenkinsfile (see https://jenkins.io/doc/book/pipeline/#stage).

- Adaptions of the migration suite use case which required by changes in EP 2.2.

- Added documentation for the Migration Suite Use Case

2.2

2.2p1

1.7.1

- Simulink only use case now uses Additional Model Info as expected.

2.2

2.2p1

1.7.0

- Changed default setting of profile creation property "saveProfileAfterEachStep" from true to false

- Fixed SL Architecture Import

- Adapted for BTC EmbeddedPlatform 2.2

2.2

2.2p1

1.6.0

- Added additional architecture import (Simulink, Merged Architecture).

- Added import steps for Input Restrictions and Tolerances.

- PIL: Worst Case Execution Time and Maximum Stack Size are now added to the Test Automation Report if PIL is used.

2.1

2.1

1.5.1

The license locations (license files, servers) can now also be defined explicitly within the workflow step "profileLoad". If not specified explicitly, the license locations will still be retrieved from the registry (via FlexLM) in the way they have been configured in the EP license dialog.

Vector Import step now correctly treats vectors as test cases in case of kind = tc.

2.1

2.1

1.5.0

Added integer return values for BTC Pipeline steps to indicate the status of each step and support more advanced handling inside the jenkins pipeline script. The return values and their meaning are described in the sections of each step (see section Workflow Steps). Added a timeout property to the startup method.

Added high level Migration Suite workflow to check if two configurations of the same model have the same structural behavior. This can also be distributed across multiple slaves (e.g. in case of different operating systems being involved). Additional section "Migration Suite" has been added to the documentation.

2.1

2.1

1.4.0

Adpted for EP 2.1. Improved performance due to faster application startup. Added Test Composer format to VectorImport step.

2.1

2.1

1.3.2

Added re-use existing code option for target link use case.

2.0.3

2.0.3

1.3.1

Added Vector Import step for csv or xlsx in Standard BTC Embedded Systems format. Improved handling of some situations in overall report. Minor additional changes for reporting.

2.0.3

2.0.3

1.3.0

Added Formal Verification Use Case to enable proof execution as an automated test step including the formal verification report.

2.0.3

2.0.3

1.2.3

Added Test Automation Report which provides an overview of the executed steps and contains links to the respective step reports (Test Execution, Back-to-Back, Code Analysis)

2.0.3

2.0.3

1.2.2

Functional Tests and Regression Tests will now use the correct default behavior regarding the execution config (as described in the documentation).

2.0.3

2.0.3

1.2.1

BTC EmbeddedPlatform installation path is now automatically retrieved from the Windows Registry by the BTC Plugin in Jenkins during the tool startup phase.

2.0.3

2.0.3

1.2.0

Added "BTC DSL for Pipeline" Plugin for Jenkins that allows easy usage of BTC EmbeddedPlatform workflow steps in the Jenkins Pipeline.

2.0.3

2.0.3

1.1.6

TargetLink Profile Creation no longer requires a start up script by design. Concurrent EmbeddedPlatform instances will now use an isolated matlab instance to prevent unintended cross-usage.

2.0.3

2.0.3

1.1.4

Added Parameter to control whether robustness issues are added to the JUnit XML report as "failed tests".

2.0.3

2.0.3

1.1.3

Fixed issue with matlab connection settings. Additionally, stack traces will now be returned via http response in case of exceptions.

2.0.3

2.0.3

1.1.2

Added Ability to set the host compiler. Removed dependecies outside of EmbeddedTester (e.g. EmbeddedValidator related)

2.0.3

2.0.3

1.1.1

Debugging changes.

2.0.3

2.0.3

1.1.0

Added lots of parameters to support more use cases.

2.0.3

2.0.3

1.0.0

Initial Release

2.0.3

2.0.3

Prerequisites

This plugin only works in combination with BTC EmbeddedPlatform which needs to be installed and licensed separately.

Jenkins Pipeline

Overview

Integrating test runs with BTC EmbeddedPlatform in your Jenkins workflows combines the automation and traceability concepts and results in great benefits:

  1. The automated workflows scale for multiple components / projects with low configuration effort
  2. You are easily able to trace changes made to your system under test from the Source Code Management to the integrated product and recognize test failures early in the process
  3. The pipeline visualization intuitively shows how much time each phase of the testing process takes
  4. The Jenkins Automation Plugin produces an XML report in the JUnit format that can be analyzed by Jenkins to provide test status trends over multiple executions and projects
  5. Comprehensive HTML Reports from BTC EmbeddedPlatform are available directly from the Jenkins job page

    Jenkins' Content Security Policy can prevent the reports from being displayed properly. See Configuring Content Security Policy for further details.

  6. Relevant artifacts like the test profile are accessible for easy debugging and analysis

Licensing for Jenkins Integration

In addition to the basic license requirements that depend on the chosen workflow steps which require EmbeddedTester or EmbeddedValidator the Jenkins Automation use case requires the "Test Automation Server" floating network license (ET_AUTOMATION_SERVER).

Since v2.4.0 it's possible to run ET_BASE use cases (Architecture Import, Test Case Import, Test Execution, Reporting) with an EmbeddedTester BASE installation and the license ET_AUTOMATION_SERVER_BASE.

Configuration

In a Jenkins Pipeline the configuration of a job can be defined as simple groovy code which can be versioned alongside the main source files of the component. A full documentation of the Jenkins Pipeline can be found here. The following example shows of how BTC EmbeddedPlatform can be automated from Jenkins via the BTC DSL for Pipeline Plugin. The Plugin needs to be installed in Jenkins and dedicated BTC methods to create a test automation workflow.

Pipeline Example
node {
    // checkout changes from SCM
    checkout scm
 
    // start EmbeddedPlatform and connect to it
    btc.startup {}
 
    // load / create / update a profile
    btc.profileLoad {
        profilePath = "profile.epp"
        tlModelPath = "powerwindow_tl_v01.slx"
        tlScriptPath = "start.m"
        matlabVersion = "2017b"
        updateRequired = true
    }
 
    // execute functional tests
    btc.rbtExecution {}
 
    // generate stimuli vectors
    btc.vectorGeneration {
        pll = "STM, D, MCDC"
        createReport = true
    }
 
    // execute back-to-back test MIL vs. SIL
    btc.backToBack {
        reference = "TL MIL"
        comparison = "SIL"
    }
 
    // close EmbeddedPlatform and store reports
    btc.wrapUp {}
}

Workflow Steps

Step “startup”

DSL Command: btc.startup {...}

Description

Method to connect to BTC EmbeddedPlatform with a specified port. If BTC EmbeddedPlatform is not available it is started and the method waits until it is available. The following optional settings are available:

Property

Description

Example Value(s)

port

Port used to connect to EmbeddedPlatform.

(default: 29267)

1234, 29268, 8073

timeout

Timeout in seconds before the attempt to connect to EmbeddedPlatform is cancelled. This timeout should consider the worst case CPU & IO performance which influences the tool startup.

(default: 120)

40, 60, 120

licensingPackage

Name of the licensing package to use, e.g. to use a EmbeddedTester BASE.

(default: ET_COMPLETE)

ET_BASE

Possible Return values

Return Value

Description

200

Started a new instance of BTC EmbeddedPlatform and successfully connected to it.

201

Successfully connected to an already running instance of BTC EmbeddedPlatform.

400

Timeout while connecting to BTC EmbeddedPlatform (either manually specified or 120 seconds).

500

Unexpected Error

Jenkins will always connect to the active version of EmbeddedPlatform since many tasks will only work with the version that is integrated into Matlab. Please ensure that the correct EP version is active by choosing Activate BTC EmbeddedPlatform in your start menu for the desired version and also ensure that the Jenkins Automation Plugin is installed for this version of EmbeddedPlatform.

Step “profileLoad”

DSL Command: btc.profileLoad {...}

Description

Opens the profile if the specified profile exists, otherwise creates a new profile. A profile update is only performed if this is required. Profile Creation requires either a TargetLink model or C-Code in combination with a CodeModel.xml architecture description.

The “profileLoad” step or any of the "profileCreate" steps are a mandatory starting point for all automation workflows.

Property

Description

Example Value(s)

profilePath

Path of the profile. If it does not exist, it will be created. The path can be absolute or relative to the jenkins job's workspace.

mandatory

"profile.epp"

tlModelPath

Path of the TargetLink model. The path can be absolute or relative to the jenkins job's workspace.

"model.slx"

tlScriptPath

Path of the model init script. The path can be absolute or relative to the jenkins job's workspace.

"init.m"

tlSubsystem

Name of the Subsystem representing the TL top-level subsystem for the analysis. Note: Argument is mandatory if there is more than one top-level system in the model.

"Controller"

environmentXmlPath

Path to the XML file with additional include paths, etc.. The path can be absolute or relative to the jenkins job's workspace.

"Environment.xml"

startupScriptPath

Path to a Startup Script which can be used to initialize matlab (e.g. toolchain startup, etc.). The path can be absolute or relative to the jenkins job's workspace.

The script must provide a function of the same name which takes at least one input argument (the model path) and returns exactly one output value (an exit code; 0 indicating success).

"startup_toolchain.m"

codeModelPath

Path of the hand code description file. The path can be absolute or relative to the jenkins job's workspace.

"CodeModel.xml"

compilerShortName

Short name of the compiler that should be used (C-Code Use Case). Fallback will be an already selected compiler or, if undefined, the first one that is found.

"MSSDK71", "MSVC140", "MinGW64"

slModelPath

Path of the Simulink model. The path can be absolute or relative to the jenkins job's workspace.

"slModel.slx"

slScriptPath

Path of the model init script for the Simulink model. The path can be absolute or relative to the jenkins job's workspace.

"init.m"

addModelInfoPath

Path to the XML file with additional model info for SL use case. The path can be absolute or relative to the jenkins job's workspace.

"AddGenModelInfo.xml"

pilConfig

Name of the PIL configuration to use. This config must exist in TargetLink. Setting a PIL Config will activate PIL in the profile and enable you to choose "PIL" as an execution config.

"default EVM"

pilTimeout

Timeout in seconds for the download process to the PIL board.

60, 120

(default: 60)

calibrationHandling

The calibration handling controls how calibrations are recognized during architecture import.

"EXPLICIT PARAM", "LIMITED BLOCKSET", "OFF"

(default: "EXPLICIT PARAM")

testMode

The test mode controls whether local displayables will be available for testing (GREY BOX) or not (BLACK BOX).

"GREY BOX", "BLACK BOX"

reuseExistingCode

Boolean flag that controls if EmbeddedPlatform will use existing generated code from TargetLink. Requires the Code and the linking information in the data dictionary to be available.

true, false

(default: false)

matlabVersion

Controls which matlab version will be used by the tool.

String containing the release version (e.g. “2016b”), optionally followed by “32-bit” or “64-bit”. The version and 32/64-bit part should be separated by a space character.

“2010a 32-bit”

“2013b”,

“2016b 64-bit”

exportPath

Path to a folder where reports shall be stored. The path can be absolute or relative to the jenkins job's workspace.

"reports” (default)

updateRequired

Boolean flag that controls whether or not the profile is being update after loading.

true, false

(default: false)

saveProfileAfterEachStep

Boolean flag that controls whether or not the profile is being saved after each step.

true, false

(default: false)

logFilePath

Path for the log file. The path can be absolute or relative to the jenkins job's workspace.

"log.txt" (default)

licenseLocationString

String containing the license locations in the order of their priority. Multiple locations are to be separated by a semicolon. If not specified explicitly, the license locations will still be retrieved from the registry (via FlexLM) in the way they have been configured in the EP license dialog.

"C:\Licenses\EP21_30.01.2019.lic;@192.168.0.1"

"9000@myserver.com"

Possible Return values

Return Value

Description

200

Successfully loaded an existing profile.

201

Successfully loaded an existing profile and performed an architecture update (see updateRequired property above).

202

Successfully created a new profile.

400

Error during profile creation. Throws an exception because further testing is not possible.

500

Unexpected Error. Throws an exception because further testing is not possible.

Step “profileCreateTL”

DSL Command: btc.profileCreateTL {...}

Description

Creates a new profile for a TargetLink model.

The listed properties only show the TargetLink specific properties. Each of the properties listed in the "profileLoad" step also apply here.

Property

Description

Example Value(s)

profilePath

Path of the profile. If it does not exist, it will be created. The path can be absolute or relative to the jenkins job's workspace.

mandatory

"profile.epp"

tlModelPath

Path of the TargetLink model. The path can be absolute or relative to the jenkins job's workspace.

mandatory for TL use case

"model.slx"

tlScriptPath

Path of the model init script. The path can be absolute or relative to the jenkins job's workspace.

"init.m"

tlSubsystem

Name of the Subsystem representing the TL top-level subsystem for the analysis. Note: Argument is mandatory if there is more than one top-level system in the model.

"Controller"

environmentXmlPath

Path to the XML file with additional include paths, etc.. The path can be absolute or relative to the jenkins job's workspace.

"Environment.xml"

reuseExistingCode

Boolean flag that controls if EmbeddedPlatform will use existing generated code from TargetLink. Requires the Code and the linking information in the data dictionary to be available.

true, false

(default: false)

Step “profileCreateSL”

DSL Command: btc.profileCreateSL {...}

Description

Creates a new profile for a Simulink model.

The listed properties only show the Simulink specific properties. Each of the properties listed in the "profileLoad" step also apply here.

Property

Description

Example Value(s)

profilePath

Path of the profile. If it does not exist, it will be created. The path can be absolute or relative to the jenkins job's workspace.

mandatory

"profile.epp"

slModelPath

Path of the TargetLink model. The path can be absolute or relative to the jenkins job's workspace.

mandatory for TL use case

"model.slx"

slScriptPath

Path of the model init script. The path can be absolute or relative to the jenkins job's workspace.

"init.m"

addModelInfoPath

Path to the XML file with additional model info for SL use case. The path can be absolute or relative to the jenkins job's workspace.

mandatory for SL use case

"AddGenModelInfo.xml"

Step “profileCreateC”

DSL Command: btc.profileCreateC {...}

Description

Creates a new profile for supported ansi C-Code.

The listed properties only show the C-Code specific properties. Each of the properties listed in the "profileLoad" step also apply here.

Property

Description

Example Value(s)

profilePath

Path of the profile. If it does not exist, it will be created. The path can be absolute or relative to the jenkins job's workspace.

mandatory

"profile.epp"

codeModelPath

Path of the hand code description file. The path can be absolute or relative to the jenkins job's workspace.

mandatory for hand code use case

"CodeModel.xml"

compilerShortName

Short name of the compiler that should be used (C-Code Use Case). Fallback will be an already selected compiler or, if undefined, the first one that is found.

mandatory for hand code use case

"MSSDK71", "MSVC140", "MinGW64"

Step “vectorImport”

DSL Command: btc.vectorImport {...}

Description

Imports test cases or stimuli vectors from the specified location. The following settings are available:

Property

Description

Example Value(s)

importDir

The directory that contains the vectors to import. The path can be absolute or relative to the jenkins job's workspace.

mandatory

“io\vectors", "E:\data\ImportExport"

vectorFormat

String to specify the format of the vector import files in Standard BTC EmbeddedPlatform style.

(default: EXCEL)

"CSV", "EXCEL", "TC"

vectorKind

A String that defines the type of the vectors to import. Can be "TC" (= Test Case) or "SV" (= Stimuli Vector).

(default: TC)

"TC", "SV"

Possible Return values

Return Value

Description

200

Successfully imported all vectors.

300

No valid vectors were found in the importDir.

400

Error during vector import.

500

Unexpected Error

Step “toleranceImport”

DSL Command: btc.toleranceImport {...}

Description

Imports tolerance settings from the specified file. The following options are available:

Possible Return values

Property

Description

Example Value(s)

path

The file that contains the tolerance settings. The path can be absolute or relative to the jenkins job's workspace.

mandatory

“io\tolerances.xml", "E:\data\tolerances.xml"

useCase

String to specify the use case for the Tolerances (Back-to-Back or Requirements-Based Testing).

(default: B2B)

"B2B", "RBT"

Possible Return values

Return Value

Description

200

Successfully imported the tolerance settings.

400

No path specified.

401

The file at specified path does not exist.

402

The specified useCase is invalid.

500

Unexpected Error

Step “toleranceExport”

DSL Command: btc.toleranceExport {...}

Description

Exports tolerance settings to the specified file. The following options are available:

Possible Return values

Property

Description

Example Value(s)

path

The tolerance settings file. The path can be absolute or relative to the jenkins job's workspace.

mandatory

“io\tolerances.xml", "E:\data\tolerances.xml"

useCase

String to specify the use case for the Tolerances (Back-to-Back or Requirements-Based Testing).

(default: B2B)

"B2B", "RBT"

Possible Return values

Return Value

Description

200

Successfully exported the tolerance settings.

400

No path specified.

402

The specified useCase is invalid.

500

Unexpected Error

Step “inputRestrictionsImport”

DSL Command: btc.inputRestrictionsImport {...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Imports Input Restrictions from the specified file. The following options are available:

Possible Return values

Property

Description

Example Value(s)

path

The file that contains the Input Restrictions. The path can be absolute or relative to the jenkins job's workspace.

mandatory

“io\inputrestrictions.xml", "E:\data\inputrestrictions.xml"

Possible Return values

Return Value

Description

200

Successfully imported the tolerance settings.

400

No path specified.

401

The file at specified path does not exist.

500

Unexpected Error

Step “inputRestrictionsExport”

DSL Command: btc.inputRestrictionsExport {...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Exports Input Restrictions to the specified file. The following options are available:

Possible Return values

Property

Description

Example Value(s)

path

The Input Restrictions xml file. The path can be absolute or relative to the jenkins job's workspace.

mandatory

“io\inputrestrictions.xml", "E:\data\inputrestrictions.xml"

Possible Return values

Return Value

Description

200

Successfully exported the tolerance settings.

400

No path specified.

500

Unexpected Error

Step “rbtExecution”

DSL Command: btc.rbtExecution {...}

Description

Executes all functional test cases in the profile. A Test Execution Report will be exported to the “exportDir” specified in the “profileLoad” step. The following optional settings are available:

Property

Description

Example Value(s)

executionConfigString

Execution configs for the Test Execution (comma separated list)

(default: all available configs)

“TL MIL, SIL, PIL”

“SIL, PIL”

“SL MIL”

debugConfigString

Debug configs for the debug export (comma separated list). There will be a debug export for each scope that contains all failed test cases.

(default: no debug export)

"TL MIL"

"SIL (Generic)"

"SIL (Visual Studio 2013)"

reportSource

String that specified if the report is based on scopes or requirement sources. Setting the report source to "REQUIREMENT" has no effect if no requirements are available in the profile.

(default: SCOPE)

"SCOPE"

"REQUIREMENT"

scopesWhitelist

Comma separated String with scopes to include. If this string is not empty, only scopes that are listed here will be considered.

(default: "" - empty String: all scopes will be considered)

"toplevel"

"toplevel, subA, subB"

scopesBlacklist

Comma separated String with scopes to exclude. If this string is not empty, only scopes that are not listed here will be considered.

(default: "" - empty String: no scopes will be excluded)

"toplevel"

"toplevel, subA, subB"

foldersWhitelist

Comma separated String with folders to include. If this string is not empty, only folders that are listed here will be considered.

(default: "" - empty String: all folders will be considered)

"Default Test Cases"

"FolderA, FolderB"

foldersBlacklist

Comma separated String with folders to exclude. If this string is not empty, only folders that are not listed here will be considered.

(default: "" - empty String: no folders will be excluded)

"Default Test Cases"

"FolderA, FolderB"

testCasesWhitelist

Comma separated String with testcases to include. If this string is not empty, only testcases that are listed here will be considered.

(default: "" - empty String: all testcases will be considered)

"tc1"

"tc1, tc2, tc44"

testCasesBlacklist

Comma separated String with testcases to exclude. If this string is not empty, only testcases that are not listed here will be considered.

(default: "" - empty String: no testcases will be excluded)

"tc1"

"tc1, tc2, tc44"

Filtering via White- & Blacklists

You can define whitelists and blacklists for scopes, folders and test cases. Everything will be merged resulting in a filtered set of test cases. Blacklists always have precedence over whitelists (i.e. if something is whitelisted and blacklisted it will be excluded).


Possible Return values

Return Value

Description

200

All test cases passed (status: PASSED)

201

Nothing to Execute (no functional test cases in the profile).

300

There were failed test cases (status: FAILED)

400

There were errors during test case execution (status: ERROR)

500

Unexpected Error

Step “xmlReport”

DSL Command: btc.xmlReport{...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Creates the XML Report and exports it to the "exportDir" specified in the "profileLoad" / "profileCreate" step. Requires BTC Plugin for XMLReports. The following optional settings are available:

Property

Description

Example Value(s)

path

Path to the report (relative paths will be resolved to the exportDir).

(default: "BTCXmlReport_<USECASE>.xml")

true, false

useCase

Controls for which use case the coverage is reported.

(default: B2B)

"B2B", "RBT"

Possible Return values

Return Value

Description

200

Success

500

Unexpected Error

Step “codeAnalysisReport”

DSL Command: btc.codeAnalysisReport{...}

Required License

EmbeddedTester (ET_COMPLETE) for B2B use case.

Description

Creates the Code Analysis Report and exports it to the "exportDir" specified in the "profileLoad" / "profileCreate" step. The following optional settings are available:

Property

Description

Example Value(s)

reportName

The filename (String) for the resulting html file.

(default: "report.html")

"report.html", "BTCCodeCoverage.html"

includeSourceCode

Boolean flag controlling whether the annotated source code will be included in the Code Analysis Report.

(default: false)

true, false

useCase

Controls for which use case the coverage is reported.

(default: B2B)

"B2B", "RBT"

Possible Return values

Return Value

Description

200

Success

500

Unexpected Error

In addition, this step creates a CSV file "BTCCoverageOverview_USECASE.csv" (with USECASE being B2B or RBT) which can be used by other Jenkins Plugins like the Plot Plugin to report coverage.

Example content of the CSV File:

Statement Coverage, Decision Coverage, MC/DC Coverage
100.0, 90.0, 91.98


Example of plots created by the Plot Plugin:

plot csvFileName: 'plot-b2b-codecoverage.csv', csvSeries: [[displayTableFlag: false, exclusionValues: '', file: "reports/BTCCoverageOverview_B2B.csv", inclusionFlag: 'OFF', url: '']], group: 'BTC Code Coverage Overview', style: 'line', title: 'B2B Code Coverage (Structural)', yaxis: 'Coverage Percentage'

Step “modelCoverageReport”

DSL Command: btc.modelCoverageReport{...}

Required License

EmbeddedTester (ET_COMPLETE) for B2B use case

Simulink Coverage (formerly V&V)

Description

Creates the Model Coverage Report and exports it to the "exportDir" specified in the "profileLoad" / "profileCreate" step. The following optional settings are available:

Property

Description

Example Value(s)

executionConfig

The execution config for the MIL execution used for model coverage measurement.

(default: first available MIL execution config, arbitrary if more than one exists)

"TL MIL", "SL MIL"

reportName

The filename (String) for the resulting html file.

(default: "report.html")

"report.html", "BTCCodeCoverage.html"

useCase

Controls for which use case the coverage is reported.

(default: RBT)

"B2B", "RBT"

Possible Return values

Return Value

Description

200

Success

500

Unexpected Error


Step “formalTest”

DSL Command: btc.formalTest{...}

Required License

EmbeddedTester (ET_BASE) + Formal Test Add-On

Description

Executes a Formal Test based on all formal requirements in the profile. A Formal Test Report will be exported to the “exportDir” specified in the “profileLoad” step (and will be linked in the overview report). The following optional settings are available:

Property

Description

Example Value(s)

executionConfigString

Execution configs on which the Formal Test should run (comma separated list)

(default: all available configs)

“TL MIL, SIL, PIL”

“SIL, PIL”

“SL MIL”

debugConfigString

Debug configs for the debug export (comma separated list). There will be a debug export for each scope that contains all failed test cases.

(default: no debug export)

"TL MIL"

"SIL (Generic)"

"SIL (Visual Studio 2013)"

Possible Return values

Return Value

Description

200

All test cases passed (status: PASSED / FULLFILLED)

201

Nothing to Execute (no formal requirements in the profile).

300

There were violations (status: FAILED / VIOLATED)

400

There were errors during test case execution (status: ERROR)

500

Unexpected Error


Step “rangeViolationGoals”

DSL Command: btc.rangeViolationGoals{...}

Required License

EmbeddedTester (ET_COMPLETE)

Required Plugin

Plugin RangeViolationGoals

Description

Adds Range Violation Goals to the profile which contribute to the Code Analysis Report and can be considered during vector generation (pll: "RVG"). The following optional settings are available:

Property

Description

Example Value(s)

scopePath

Hierarchy path to the targeted scope / subsystem. Leave empty to target the toplevel. Use "*" to target all scopes.

(default: toplevel subsystem)

"Toplevel/SubA", "*"

rvXmlPath

Path to an xml file containing Range Violation specs.

"RangeViolationGoals.xml"

considerOutputs

Boolean flag controlling whether the goals should be created for Outputs.

(default: true)

true, false

considerLocals

Boolean flag controlling whether the goals should be created for local displayables.

(default: true)

true, false

checkRangeSpecification

Boolean flag controlling whether the goals should only be created if a signal has Min/Max values other than the data type range.

(default: true)

true, false

Possible Return values

Return Value

Description

200

Success

400

Range Violation Goals plugin not installed

500

Unexpected Error

Step “domainCoverageGoals”

DSL Command: btc.domainCoverageGoals{...}

Required License

EmbeddedTester (ET_COMPLETE)

Required Plugin

Plugin DomainCoverageGoals

Description

Adds Domain Coverage Goals to the profile which contribute to the Code Analysis Report and can be considered during vector generation (pll: "DCG"). The following optional settings are available:

Property

Description

Example Value(s)

scopePath

Hierarchy path to the targeted scope / subsystem. Leave empty to target the toplevel. Use "*" to target all scopes.

(default: toplevel subsystem)

"Toplevel/SubA", "*"

dcXmlPath

Path to an xml file containing Domain Coverage specs.

"DomainCoverageGoals.xml"

raster

String to specify a raster in %. Domain Coverage Goals will be created for equal according to the raster.

(default: 25)

"10", "25", "30"

Possible Return values

Return Value

Description

200

Success

400

Domain Coverage Goals plugin not installed

500

Unexpected Error

Step “vectorGeneration”

DSL Command: btc.vectorGeneration{...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Executes the engines for analysis and stimuli vector generation for structural coverage. The following optional settings are available:

Property

Description

Example Value(s)

pll

Semicolon separated list of PLLs used to set the goals for automatic stimuli vector generation.

(default: all goals will be analyzed)

“STM; D”, “STM:3”, … (see Back-to-Back & Vector Generation User Guide for more details about PLLs)

engine

Engine to be used for vector generation (guided random, model checker, both)

(default: "ATG+CV", combined hierachical approach)

“ATG+CV", "ATG", "CV" (see Back-to-Back & Vector Generation User Guide for more details about engines)

globalTimeout

Global timeout in seconds. 0 means no timeout.

(default: 600)

600

scopeTimeout

Scope timeout in seconds. 0 means no timeout.

(default: 300)

300

perPropertyTimeout

Timeout per coverage property in seconds. 0 means no timeout.

(default: 60)

60

considerSubscopes

Boolean flag controlling whether or not to consider coverage goals from subscopes.

(default: true)

true, false

recheckUnreachable

Boolean flag controlling whether or not to recheck already calculated unreachable results.

(default: false)

true, false

depthCv

Controls the maximum depth for the CV engine. Set to 0 for infinite depth.

(default: 10)

0, 10, 20, 50

depthAtg

Controls the maximum depth for the ATG engine. Must be greater than 0.

(default: 20)

10, 20, 50

loopUnroll

Number of loop interations to unroll for unpredictable loops.

(default: 50)

10, 20, 50

robustnessTestFailure

Boolean flag controlling whether or not robustness issues are added to the JUnit XML Report as "failed tests".

(default: false)

true, false

createReport

Boolean flag controlling whether or not the Code Analysis Report is created by this step. The report can be created explicitly in its own step which is why you might want to tweak this setting.

(default: false)

true, false

Possible Return values

Return Value

Description

200

Successfully generated vectors and reached all selected coverage goals (see PLL property). No robustness goals were covered.

300

Ran into timeouts before completely analyzing all selected goals (see PLL property). No robustness goals were covered.

41x

Covered Robustness Goal: Downcast

4x1

Covered Robustness Goal: Division by Zero

500

Unexpected Error

Step “backToBack”

DSL Command: btc.backToBack {...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Executes a Back-to-Back Test between the specified reference and comparison configuration (e.g. TL MIL vs. SIL). This step requires stimuli vectors or functional test cases in the profile. A Back-to-Back Test Report will be exported to the “exportDir” specified in the “profileLoad” step.

In automated scenarios the effort for manual reviews of frequently executed Back-to-Back tests can become quite high. The BTC plugin "ApplyFailedAccepted" deals with this challenge by applying your manually accepted deviations to all future Back-to-Back Tests as long as the deviating values are equal. For more information, please contact support@btc-es.de.


The following optional settings are available:

Property

Description

Example Value(s)

reference

Execution config for the Back-to-Back test reference simulation.

TL MIL, SIL, PIL, SL MIL

(default: “TL MIL”)

comparison

Execution config for the Back-to-Back test comparison simulation.

TL MIL, SIL, PIL, SL MIL

(default: “SIL”)

debugConfigString

Debug configs for the debug export (comma separated list). There will be a debug export for each scope that contains deviations which includes all vectors which lead to a deviation.

(default: no debug export)

"TL MIL"

"SIL (Visual Studio 2013)"

Possible Return values

Return Value

Description

200

Back-to-Back Test passed (status: PASSED)

201

Back-to-Back Test has accepted failures (status: FAILED ACCEPTED)

300

There were deviations between the reference and comparison architecture (status: FAILED)

400

There were errors during the execution (status: ERROR)

500

Unexpected Error

Step “regressionTest”

DSL Command: btc.regressionTest {...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Executes a Regression Test between the current SUT and old Execution Records that have been saved. If no saved Execution Records are available, the vectors will only be executed on the current SUT and the Execution Records will be stored for a later Regression Test. This requires stimuli vectors or functional test cases in the profile.  A Regression Test Report will be exported to the “exportDir” specified in the “profileLoad” step. The following optional settings are available:

Property

Description

Example Value(s)

executionConfigString

Execution configs for the simulation on the current SUT (comma separated list).

(default: SIL)

“TL MIL, SIL, PIL”, “SL MIL”,

“SIL”

debugConfigString

Debug configs for the debug export (comma separated list). There will be a debug export for each scope that contains deviations which includes all vectors which lead to a deviation.

(default: no debug export)

"TL MIL"

"SIL (Visual Studio 2013)"

Possible Return values

Return Value

Description

200

Regression Test passed (status: PASSED)

201

Nothing to compare. Simulation results stored for later Regression Tests.

300

There were deviations between the old and the new version (status: FAILED)

400

There were errors during the execution (status: ERROR)

500

Unexpected Error

Step “formalVerification”

DSL Command: btc.formalVerification {...}

Required License

EmbeddedValidator (EV)

Description

Executes all existing proofs in the profile and generates a Formal Verification Report. The following optional settings are available:

Property

Description

Example Value(s)

loopUnroll

If the code that is to be analyzed contains loops without explicit maximum number of iterations, e.g. a while(true) loop, these loops need to be unrolled. The given number provides the number of iterations for these loops that is used for the analysis. This unrolling is done in such a way that a "fulfilled" or "fulfilled (n steps)" result can only be obtained if this limit is not exceeded by any possible execution. Conversely, a trace that violates this limit can be found as a witness trace despite not strictly violating the formal requirement itself. This is indicated in the termination reason as "Loop unroll limit (n) exceeded".

(default: 32)

10, 30, 50

memoryLimit

Memory Limit (MB)

The maximum memory footprint to be used by the analysis tools of the EmbeddedPlatform. If an analysis cannot be completed within these limits, this may lead to termination reason "Memory limit (n MB) exceeded"

(default: unlimited)

0 (= unlimited), 1024, 3456

timeLimit

Time Limit (Seconds)

The maximum duration the proof execution may take (excluding some of the pre- and postprocessing tasks, which in general are less time intensive than the actual model checking). This limit should be used especially whenever the proof execution is left unattended and multiple proofs are to be executed in batch. Otherwise, a single proof execution may consume most of the run time and no results would be obtained for the other proofs.

(default: unlimited)

0 (= unlimited), 60, 1000

searchDepth

Search depth (Steps)

the number of executions of the scope under test (i.e. how many execution steps may a counter example be long). This limit corresponds to the term "unwinding depth" employed in bounded model checking. Again, if no such limit is provided, the search for a counter example may in the worst case spend large amounts of time looking for longer and longer counter examples.

(default: unlimited)

0 (= unlimited), 10, 50

Possible Return values

Return Value

Description

200

All proofs are fulfilled (status: FULFILLED)

300

There was a violation (status: VIOLATED)

301

Unknown (status: UNKNOWN)

400

BTC EmbeddedValidator package is not installed

500

Unexpected Error

Step “wrapUp”

DSL Command: btc.wrapUp {...}

Description

Publishes HTML reports and the JUnit XML report to Jenkins and closes BTC EmbeddedPlatform. The following optional settings are available:

Property

Description

Example Value(s)

archiveProfiles

Boolean flag controlling whether BTC EmbeddedPlatform profiles are archived by Jenkins to be available on the Job Page. You can disable this and control the "archiveArtifacts" option yourself.

(default: true)

true, false

publishReports

Boolean flag controlling whether the BTC EmbeddedPlatform reports are published in Jenkins to be available on the Job Page. You can disable this and control the "publishHTML" option yourself.

(default: true)

true, false

publishResults

Boolean flag controlling whether BTC EmbeddedPlatform test results (JUnit XML) are published in Jenkins to be available on the Job Page and for further aggregations. You can disable this and control the "junit" option yourself.

(default: true)

true, false


BTC Migration Suite

The BTC Migration Suite allows you to perform a fully automatic regression test between different Matlab or TargetLink versions. This makes it easy to document that the change of a tool version in a particular project does not influence the behavior of software components on model and code level.

Required License

EmbeddedTester (ET_COMPLETE)

Step “migrationSource”

DSL Command: btc.migrationSource {...}

Description

Creates a profile on the source configuration (e.g. old Matlab / TargetLink version), generates vectors for full coverage and exports the simulation results.

Step “migrationTarget”

DSL Command: btc.migrationTarget {...}

Description

Creates a profile on the target configuration (e.g. newMatlab / TargetLink version), imports the simulation results from the source config and runs a regression test.


For both steps (migrationSource and migrationTarget) the following parameters are mandatory:

Property

Description

Example Value(s)

tlModelPath

Path of the TargetLink model. The path can be absolute or relative to the jenkins job's workspace.

"tlModel.slx", "model.mdl"

matlabVersion

Controls which matlab version will be used by the tool.

String containing the release version (e.g. “2016b”), optionally followed by “32-bit” or “64-bit”. The version and 32/64-bit part should be separated by a space character.

“2010a 32-bit”

“2016b”

In Addition, you can add all other the parameters from the steps btc.profileLoad and btc.vectorGeneration if required.

Migration Suite Example: Jenkinsfile

The following example shows a Jenkinsfile for a complete migration of a TargetLink model. The labels that are specified for the node describe the resource requirements for each configuration and allow Jenkins to provide a suitable agent. Any data produced by the migrationSource step which is needed by the migrationTarget step will be automatically handled by Jenkins. This way, the agent for the target config (which is obviously a different one machine due to the different Windows versions) will have access to the simulation results from the source config required for the regression test.

Migration Suite Example

Migration Suite Example
node ('Win7 && TL41 && ML2015b') {

    checkout scm
 
    stage ("Migration Source") {
        btc.migrationSource {
            matlabVersion = "2015b"
            tlModelPath = "source/model.mdl"
            tlScriptPath = "start.m"
        }
    }
}
 
node ('Win10 && TL43 && ML2017b') {

    checkout scm
 
    stage ("Migration Target") {
        btc.migrationTarget {
            matlabVersion = "2017b"
            tlModelPath = "target/model.mdl"
            tlScriptPath = "start.m"
        }
    }
}

Adding the BTC Plugin to Jenkins

In order to use the convenient pipeline syntax described above you need to add the BTC Plugin to Jenkins. This is very easy and can be done with the following steps:

  1. On your Jenkins web UI go to Jenkins > Manage Jenkins > Manage Plugins
  2. Click on the Available tab and search for btc-embeddedplatform
  3. Select the plugin btc-embeddedplatform and install it
    1. Once the plugin is installed new versions which are available will appear in the Updates section
    2. Updating a plugin usually requires you to restart your Jenkins master

The BTC Pipeline plugin is now available in Jenkins and you can benefit from the convenient BTC pipeline methods described in the sections above. Enjoy!

If your Jenkins master can't access the internet you can manually download the plugin here and upload it to the server via the advanced section of the "Manage Plugins" page.



ArchivesGet past versions
Labels
This plugin has no labels