SCM Koji

Is an powerfull jenkisn SCM plugin, allwoing you to pull from koji-based build systems, like https://koji.fedoraproject.org/koji/.
Next t the plugin itself, the project contains also dymy file hosting server with koji-like XML-RPC api and one custom api.

• Jenkins SCM Koji Plugin
• Fake Koji
  • Naming Convention
  • SCP Upload
  • XML-RPC API
  • Curl API
  • HTTP File Listing
  • HTTP View
• Future goals

• Koji top URLs
  • URL of Kojihub, Brewhub or Fake Koji
  • this is the URL of xml-rpc server, from which plugin gets builds' information
  • can contain more than URLs separated by space, first URL found first served
• Koji download URLs
  • RPMs or archives get downloaded from these URLs
  • can contain multiple URLs separated by space, first URL found first served
• Package name
  • package name to watch
  • can contain multiple names separated by space
• Package arch
  • only builds for specified arch will be used
  • can be empty
  • can contain multiple archs separated by coma
• Package tag
  • only builds containing specified tag
  • may be empty to match all the builds
  • supports glob pattern
• Excluding NVRs
  • glob patterns for NVRs not to download.
  • example: {*debug*, *src*}
• Download directory
  • name of directory inside Jenkins' workspace to download RPMs/archives
  • if empty, the RPMs/archives will be downloaded directly into workspace
• Max previous builds
  • max number of previous builds to check, in case there's more of them
• Clean download directory
  • removes all the content of the download directory before the download
• Create subdirectory for each NVR
  • creates subdirectiory under the download directory where the name of subdirectory is going to be the name of the NVR of the RPM package

After Jenkins jobs is finished, the plugin generates a page with results information. This page shows what specific build was used(name, version, release, NVR, tags), what RPMs or archives were downloaded, links and hash sums to this build can be also found here. If available, sources of build are included.

Fake Koji serves as a storage for builds along with logs and source snapshots.

Fake Koji uses Koji's NVRA (name, version, release, architecture):

• name represents java product (java-10-openjdk, java-11-openjdk)
• version represents tag (jdk.10.46, jdk.11.23)
• release represents:
  • repository i.e. (shenandoah, ember)
  • number of changesets since last tag
  • keywords (static, slowdebug, fastdebug, zero, openj9)
• architecture can be:
  • src containing source snapshot
  • real platform (x86_64, i686, aarch64, win10_x64, rhel7_aarch64, fedora28_x64, centos7_i386)
• data stores build log for each architecture and pull/incoming logs for src

scp ./dir/file user@host:nvra

scp ./dir/nvra user@host

scp ./dir/nvra user@host:/any_path/differentNvra

scp .dir/nvra user@host:/any_path/nvra

The uploaded file will be stored in user@host:/path_to_builds/n/v/r/a/. Note that in the last two cases the any_path, where the NVRA should be stored, is ignored.

scp user@host:nvra ./dir/

scp user@host:nvra ./dir/differentNvra

The downloaded file: ./dir/nvra.

Works similarly:

scp ./dir/nvra1 ./dir/nvra2 user@host

scp user@host:nvra1 user@host:nvra2 ./dir

scp ./dir/logFile user@host:nvra/log

scp ./dir/logFile user@host:nvra/log/newFile

The logFile will be stored in user@host:/path_to_builds/n/v/r/data/logs/a/

scp user@host:nvra/log/logFile ./dir

scp user@host:nvra/log/logFile ./dir/newLogFile

The downloaded file: ./dir/logFile

Works the same as single upload/download:

scp logFile1 logFile2 user@host:nvra/log

scp user@host:nvra1/log/logFile1 user@host:nvra2/log/logFile2 ./dir

Having nvra1 and nvra2 in ./dir, then

scp -r ./dir user@host

will upload nvras to user@host:/path_to_builds/n1/v1/r1/a1 and user@host:/path_to_builds/n2/v2/r2/a2, respectively.

Having logFile1 and logFile2 in ./dir, then

scp -r ./dir user@host:nvra/logs

will upload both log files to user@host:/path_to_builds/n/v/r/data/logs/a/.

Having data1 and data2 in ./dir, then

scp -r ./dir user@host:nvra/data

will upload both files to user@host:/path_to_builds/n/v/r/data/

NVRA:

Recursive download of nvra works same as normal download. Multiple tarballs for one nvra is unlikely to be supported.

Logs:

scp -r user@host:nvra/logs ./dir

will download all logs for given nvra.

scp -r user@host:nvra/data/logs ./dir

and

scp -r user@host:nvra/data ./dir

are currently broken.

Part of Fake Koji is xml-rpc server compatible with Koji and Brew. You can get build information using these methods:

• getPackageId
  • parameter: package name
  • returns the package id if given package
• ListBuilds
  • parameter: package id
  • returns all builds of given package
• listTags
  • parameter: build id
  • returns tags of given build
  • tags tell us whether build is done on a cetain architecture or not
• listRPMs
  • parameter: build id, architectures
  • returns all build's rpms of given architectures
• listArchives
  • parameter: build id, architectures
  • archives represent .tar.gz, .zip and .msi files
  • returns build's archives of given architectures

You can get various information about Fake Koji's configuration and storage using its curl api.

curl host/get?option:argument

Alternatively you can use url bar in your browser.

• allProducts
  • lists all available products
• allProjects
  • lists all available projects
• dport
  • download port
• expectedArchesOfNvr
  • returns list of architectures the nvra is supposed to be built on
  • argument: nvra (java-10-openjdk-jdk.10.46-0.static.x86_64)
• expectedArchesOfProject
  • returns list of architectures the project is supposed to be built on
  • argument: project (java-10-openjdk-updates)
• productOfNvra
  • returns product of given nvra
  • argument: nvra
• productOfProject
  • returns product of given project
  • argument: project
• projectOfNvra
  • returns project of given nvra
  • argument: nvra
• projectsOfProduct
  • list of projects of given product
  • argument: product (java-10-openjdk)
• repos
  • path to folder containing repositories
• root
  • path to folder containing builds
• uport
  • scp upload port
• viewlport
  • view port
• xport
  • XML-RPC port

Here you can browse Fake Koji's folder structure. There are three sections with same content but different sorting. The first one uses version sorting(last version on top), the second uses latest modified sorting(the latest modified file on top) and the last one uses latest modified directory content.

└── name
    └── version
        └── release
            ├── arch1
            │   └── name-version-release-arch1.tar.gz
            ├── arch2
            │   └── name-version-release-arch2.tar.gz
            ├── arch3
            │   └── name-version-release-arch3.tar.gz
            ├── data
            │   └── logs
            │       ├── arch1
            │       ├── arch2
            │       ├── arch3
            │       └── src
            └── src
                └── name-version-release-src.tar.gz

This is the frontend of fake Koji. Here you can see the latest successful builds of every project. Projects are divided by their product. To see all builds of a project by clicking on project's details.

• Project/Repository management
• Implementation of HTTPS
• JSON API

This plugin is multimodule lib, plugin and external service. Thus the autorelease do not work as expected. To release, you have to cd koji-scm submodule which contans koji-scm-plugin artifactId and here run manually JAVA_TOOL_OPTIONS="-Dmaven.test.skip=true" mvn release:prepare and JAVA_TOOL_OPTIONS="-Dmaven.test.skip=true" mvn release:perform. During preapare, always adjsut all modules, not jsut dependent ones (choice of 0). The skipped tests expected you toi test it properly before, as there are some lenghty ones. The `JAVA_TOOL_OPTIONS is used to workaround the forked processes.

Sometimes, you may need to "fake" the build bit, and to change project version to future not-snapshot one (the upcoming release), and future snapshot-one, so they are in local m2 repos. then revert the change, and proceed as described above.

Note that only plugin goes to official repos. Include the services and other artfacts on github release page, after tag is done.

After release, you may need to fix your poms. Verify that all pom.xml files have correct version and are sane (eg: https://github.com/jenkinsci/koji-scm-plugin/commit/668512eaf998537f29587f8c7bdc7ec91a0d9e96 , but can be muhc worse)