Railflow for TestRail

Jenkins is one of the most popular CI applications and is used by thousands of companies all over the world. Jenkins is used to define the application build and test processes. QA teams also use Jenkins to run/schedule functional tests across a variety of tools and frameworks. However, the results of these tests can only be viewed in Jenkins, and it is not really possible to aggregate results across multiple jobs, retain long history, and delegate test failures to your team.

By using the Railflow Jenkins Plugin, you can easily integrate Jenkins testing jobs with TestRail and automatically export all testing reports to TestRail. Aggregating result from all your Jenkins jobs into TestRail allows teams to look at test trends, auto-assign failures via Railflow automation, create link between Jenkins and TestRail, and much more.

Railflow Jenkins plugin is an open source product from version 2.9 onwards. As such, it is now available in the Jenkins plugin directory. However you still need a valid license is required to use the Railflow CLI tools.

• Jenkins version 2.204.1 or newer

1. Navigate to the Manage Jenkins | Plugins | Available plugins page of the Jenkins UI to see the list of available plugins.
2. Select the Railflow for TestRail plugin and click Install.

To configure plugin, navigate to Manage Jenkins | Configure System page of the Jenkins UI and scroll down to the Railflow Global Settings section:

Railflow provides two license activation models. If your network allows outbound call to our api endpoint: https://api.railflow.io, then option 1: License Key (online activation) will work. If you are unable to reach our api endpoint, pick option 2: License File (offline activation).

1. License Key (online activation): Select License Key option and copy/paste the license key from your email and click Activate License

2. License File (offline activation): If your Jenkins instance does not have outbound internet access, you may use this option to upload the license activation from your email. The license file has a .skm extension.

Railflow plugin configuration section allows you to centrally defined one or multiple TestRail servers. Once configured, they can be easily referenced in the Jenkins jobs.

If you are using Jenkins UI based (freestyle job) jobs to run some test frameworks/tools, you can use the Railflow plugin's UI controls to easily integrate with TestRail by following these steps.

Add and configure the Jenkins job's post-build action: Railflow Plugin: TestRail Test Results Processor. Railflow's post-build action allows you to specify the test framework, test results location, and various other TestRail configurations.

For additional TestRail export functionality, check out the Advanced section.

If you are using a Jenkinsfile to run some test frameworks/tools, you can access the same plugin functionality within the Jenkinsfile and easily integrate with TestRail by following these steps

pipeline {
    agent {
        label "linux"
    }
    stages {
       stage('Build') {
          steps {
             echo '********BUILD STEP**********'
             git branch: 'master', credentialsId: '*****', url: 'https://...junit-demo.git'
             sh 'mvn test'
          }
       }
    }
    post {
     always {
     echo '********UPLOAD RESULT INTO TESTRAIL**********'

    railflow(
        testRailServerName: 'TestRail',
        testRailProjectName: 'Railflow Demo',
        failIfUploadFailed: true, 
        overriddenUserName: 'userName', 
        overriddenPassword: 'password',
        jobConfigurations: [[configurationNames: '''OS/Linux
            Browser/Chrome''', 
        milestonePath: 'M1/M2', 
        resultFilePattern: '**/surefire-reports/*.xml',
        uploadMode: 'CREATE_NO_UPDATE',
        searchMode: 'Path',
        testCaseCustomFields: '''Required text field=Hello from Jenkins
            estimate=30s''', 
        testCasePath: 'Railflow/Website', 
        testCasePriority: 'High', 
        testCaseType: 'Automated', 
        testPlanName: 'Test plan', 
        testReportFormat: 'JUNIT', 
        testResultCustomFields: '''Custom field=Results from Jenkins
            version=2.0''', 
        testRunName: '${JOB_NAME}-${BUILD_NUMBER}'
        smartTestFailureAssignment: 'user1@abc.com, user2@abc.com', 
        testCaseTemplate: 'template1', 
        caseSearchField: 'case search field name', 
        tagsFieldName: 'tags field name',
        uploadMode: 'CREATE_NO_UPDATE', 
        disableGrouping: true, 
        closeRun: true, 
        closePlan: true, 
        fullCaseNamesAllowed: true ]])
      }
    }
}

Both the Jenkinsfile and Freestyle job depend on the Railflow Plugin: TestRail Test Results Processor post-build action to process test results and post them to TestRail. This action provides users a host of options to integrate Jenkins with TestRail in a variety of ways and across any testing tool and framework. The reference below describes all the options and their usage.

You can add multiple Raillflow post-build actions using Add More button. This may be needed if you have multiple reports to process and want to map each report to different places in TestRail.