Tech - 09/02/2017

First Jenkins Area Meetup in Munich @ PAYBACK

On Monday, 6th February 2017 we had our first Jenkins Area Meetup in Munich. We were quite surprised how many local experts are around here. They even came from Ingolstadt just to join the JAM Kickoff. One of the main goals for us was to find out what the attendees are interested in. Besides the obvious social networking part to share and learn about Jenkins technologies, we asked what they would love to hear about in further JAM Sessions. Here are the results:


People were also highly interested in some specific workshops for Jenkins Job DSL and our own Framework “V-DSL”. Please stay tuned until we can release it into the wild under an Open-Source licence. To get a sneak peak about what we did, check out the slides and presentation on Rule Jenkins with Configuration as Code in the meanwhile. Another topic was advice about how to build a rock stable Jenkins environment.

Startup Jenkins with specific JVM options and System Properties

-Xms4096m -Xmx4096m
-XX:NewSize=2048m -XX:MaxNewSize=2048m
-XX:ParallelGCThreads=4 -XX:ConcGCThreads=4

This is the base setup you should go with. 4 gigs of RAM should be enough for any use case, nothing more. If you are interested in the details here, check out another great Jenkins talk: So, You Want to Build the World’s Biggest Jenkins Cluster? Keep in mind, that if you encounter significant performance decrease, activate JMX in order to keep an eye on your JVM running Jenkins. To generally enhance your overall Jenkins experience is setting proper System Properties. Check out the complete list to see what is possible. For example:


The most significant one we stumbled upon was a fix for a major groovy memory leak. This is fixed in groovy version 2.4.8, but unfortunately, Jenkins core runs 2.4.7 in the current LTS version (v2.32.2). We recommend to set the following startup parameter as a workaround if you are excessively running groovy scripts (e.g. DSL or pipeline)


Have full control on other system settings with Post-initialization scripts

The simplest example can be found in the documentation, but this opens up a lot of other possibilities too. For example we heavily struggled with Jenkins SECURITY-170, where arbitrary build parameters were disabled. Either you enable them back one by one, or you are aware of the risks and enable them back again and migrate affected jobs calmly. Same thing with Jenkins SECURITY-95. If you struggle with displaying issues with e.g. internally scripted tooling for Jenkins expand the Context Security Policy settings accordingly, or disable them. Keep in mind that simply disabling these security features on an internet facing Jenkins environment is a bad idea obviously.

 // SECURITY-170
 System.setProperty('hudson.model.ParametersAction.keepUndefinedParameters', 'true')
 System.setProperty('hudson.model.DirectoryBrowserSupport.CSP', '')

Update on Static Code Analysis Plug-ins

For the second talk, we had special guest Prof. Dr. Ullrich Hafner from the University of applied sciences Munich, who is the maintainer of the Static Code Analysis Plugin. He started developing this plugin round about 10 years ago and gave us a small forecast what might come up in the next 10 years, depending on personal time and engagement of his students working on bachelor and master thesises on this topic. Upcoming in one of the next releases is for example an improved recognition of warnings based on Abstract Syntax Tree comparison. Also a more modern warnings report view in Jenkins is planned. Please report any bugs on Jenkins bugtracker and do not hesitate to raise pull requests.

Follow up
A big “Thank you” to all who joined the JAM Kickoff. Many interesting talks took place especially after the keynotes and we are looking forward to host the next Meetup soon. Join the JAM Meetup and follow us on Twitter.

Christian & Markus

Tech - 05/02/2017

Join us at the first Jenkins Area Meetup in Munich!

JenkinsSwagThere are a lot of tech companies in our city and many of them are using Jenkins as their automation server. We want to gather all Jenkins enthusiasts and start this Meetup series with some interesting talks. You will see, how to configure a rock solid build server and get a glance at an ongoing open source project for a Jenkins Configuration As Code Testing Library.

Also plugin development will be covered: Almost every software project uses any kind of static code analysis and, of course, there exist a lot of Jenkins plugins for this purpose. Did you know that they are all developed here in Munich? You will have a chance to meet the guy on Monday, so don’t forget to sign up here.

Jenkins doesn’t just cover topics around continuous integration, you can do continuous delivery with it as well. To round things up, we will show you how to automate your deployments, for example with Spinnaker.

Tech - 24/01/2017

Apache Kafka Meetup with Jay Kreps and Michael Noll

Just use Kafka, they said. It will be easy, they said.

At PAYBACK Global we introduced Kafka in 2016 to enable asynchronous communication between our services in order to decouple our components and improve performance. What started as a simple idea quickly turned into an ambitious project with several hurdles. The concept of a messaging service might seem easy at first, but the more services you have and the more requirements (e.g. delivery semantics) you add, the more complex it gets in the real world. Who’d have thought that?

Our first approach of using Kafka:


We identified the following topics which still needed to be addressed:

  • How do we assure backwards and upwards compatibility?
    • How to implement?
    • How to test?
  • Do we need versioning for messages? If so, how?
  • Do we need to transport part of the request context along with actual messages/events, like a correlation id?
  • How do we deal with resilience and other non-functional requirements

If you want to find out how we answered these questions and which other pitfalls we encountered on our way join us on Wednesday 25th 2017 at the Apache Kafka Meetup with Jay Kreps and Michael Noll.

Tech - 19/09/2016

Jenkins World 2016 – Becoming a true Jenkins master

14375273_10210465135573444_494122255_oThe final day was all about how to become a true Jenkins master. The „Open Source Hub“, a booth filled with Jenkins experts and contributors, continuously showed live demos about new plugins and projects they wrote. Here I finally found the inspiration to adopt a Jenkins Plugin that is highly used within PAYBACK, but is no longer maintained. The other sessions gave a detailed insight on how to effectively prevent Jenkins instances to fail, how to setup Jenkins to deal with hundreds of team members located around the world, and how to create huge Jenkins clusters. Speaking of which, they also demoed us the worlds biggest Jenkins cluster with the astonishing amount of 2000 masters and nearly 8000 executors. Stephen Connolly,a well known Jenkins core committer and technical lead for CloudBees Jenkins Operations Center, showed us how we can do that too.


He gave some great ripping-off-the-state-of-californiainsights how we can determine the right setup size of what we really need and where scaling with Jenkins is possible. Even the Developer Infrastructure group at Google was there to
present their journey deploying a large-scale Jenkins installation. In between sessions, there were a lot of fun activities you could participate in. CloudBees managed to get the CommitStrip team to Jenkins World, creating some live drawings and even a a giant collaborative mural painting. Thomas, the writer and Etienne, the cartoonist, teamed up with a few Jenkins contributors to design a 5m x 2m mural which was painted by the attendees. Finally CloudBees invited us for a „Farewell Get-Together“ for all international attendees in an original California style casino to say goodbye and plan what we will accomplish until next year.

Tech - 16/09/2016

Jenkins World 2016 – It’s Jenkins time in California!

unbenanntFrom September 13th until September 15th 2016, the Jenkins World took place in Santa Clara, California! It is the biggest international conference concerning Jenkins, one of the most famous continuous integration tools. On three days, users, experts and continuous delivery thought leaders had the opportunity to learn, explore and network face-to-face, as well as to help shape the future of Jenkins. Of course PAYBACK could not miss an event like this, as we are also using the tool on an advanced level. Our colleague and DevOps Engineer Christian attended and even held a talk about Ruling Jenkins with Configuration as Code! Read what he experienced during these three exiting days.

The first day was all about workshops where you could get some insights into microservices, Jenkins pipeline plugin or continuous delivery in the context of DevOps. Additionally you could get prepared to become a certified Jenkins Engineer. Fortunately, I already got this certificate in the beginning of the year by participating in one of the first tests offered. Afterwards there was a great networking event, hosted at the best place you can possibly imagine, the Computer History Museum! Over there we were not only provided with some great American barbecue, you also could get back to your childhood, by walking through the exhibition halls filled with great ancient Technologies. Enjoying a couple of drinks, we also could share some insights on how companies are using Jenkins, typical issues you encounter and how to deal with them. Finally I could get some answers for very specific questions that came up all over the year by some other experts. The great talks also inspired me to finally setup a Jenkins Area Meetup Group in Munich. I struggled with this thought for a couple of months now, since there are a lot of tech companies in Munich, and almost all of them use Jenkins. I’m pretty sure we can find some other Jenkins engineers that will be interested, and as far as I know PAYBACK is a great partner for hosting meetup-events anyway 🙂

On the second day the main event finally arrived and Kohsuke Kawaguchi, creator of the Jenkins project, did the Kickoff-Talk. He was giving us a look back at where Jenkins originally started and where we are today, also giving away a sneak peek at a couple of nice new features that will be available for Jenkins in the near future. Afterwards Cloudbees CEO and founder Sacha Labourey took over to announce DevOps Express and showing us the The Four Quadrants of DevOps Maturity. After these interesting keynotes, we started to discuss more specific subjects like secure container development pipelines, continuous security or continuous testing. Actually, I got a lot (!) of feedback regarding my talk, as it seems I rather hit the nerve of what a lot of other Jenkins engineers are struggling with, too. And as I imagined, they were looking for a way to deal with it in a way, PAYBACK is now finally able to do. Afterwards, there was also a chance to have a closer look at the future of Continuous Delivery with Jenkins by using the Pipeline Plugin and a brand new user interface experience for Jenkins called Blue Ocean. In the end I had some great round up talks about today’s sessions at the “After Dark DevOps Express Networking Reception” and also got some new ideas for hacking our way through the DevOps challenges.


Tech - 12/08/2016

PAYBACK taking part in Amex hackathon #growthHACK16

As being a IMG_3703part of the American Express (Amex) group, we are of course participating in the annually internal hackathon. Last year our colleagues made it to the final round and pitched their ideas in New York among other participants from all over the world.

To those who haven’t heard about hackathons (also known as a hack day or codefest) yet – it is an event in which computer programmers and others involved in software development, including graphic designers, interface designers and project managers, collaborate intensively on software projects.

This year it is IMG_3810all about #growth! In 13 Amex locations worldwide teams are designing and implementing their ideas concerning growth for our PAYBACK or Plenti business based on mobile, location based services, augmented reality, web, real time and big data submitted by the teams.

In Munich we were the first to start hacking last Wednesday, August 10th. 33 innovators grouped in 6 cross-functional teams, including PO, DEV, TEST and OPS. The teams had awesome names like LearnFastActFast, The Crowds, MuMo, SmellsLikeTeamSpirit, Catch em all and MVP.

More to follow, as the jury still has to decide which idea they liked best.


Tech - 11/08/2016

PAYBACK Global Scrum Model

quadAgile Development has gained tremendous success in the last years. Almost every company has transformed or at least adapted partly to agile driven development; in most cases using the Scrum methodology. Starting with Scrum in a single, co-located team environment is quite easy. The challenges start when more than just one team is working on a single project or product. These challenges include the management of dependencies among the teams, aligned architecture changes and collaboration in general. The situation becomes even more delicate when the teams are not co-located. This could mean that they are in different locations and sometimes even in different time zones.


PAYBACK Global decided early on to add offshore teams to support the development in certain areas. During the last six years we have gained a lot experience in working together with dispersed scrum teams. There are many important factors if you consider scrum across distributed teams and, in particular, with offshore teams. In this article I will focus only on the team setup. The setup of a scrum team is usually simple: Product Owner, Scrum Master and the Scrum Team. In our scaled environment with many distributed teams, which are working with an offshore partner, the setup is quite unique. Additionally, the following preconditions had a huge impact on our team setup:

  • Product ownership remains at PAYBACK (headquarters). The unique selling point of PAYBACK and the entire LP group is the comprehensive knowledge of how to run a loyalty program. Further, we need to deal with different stakeholders in different countries, which is easier to do at a centralized location.
  • Technology ownership remains at PAYBACK. With the experience of PAYBACK Germany, PAYBACK Poland, and the Miles and More program (developed by our sister Loyalty Partner Solutions), we have gained a lot of experience in building and operating high-load transactional systems.
  • Quality ownership also remains at PAYBACK. As we use behavior driven development, QAs, developers and product owners are working closely together from the very beginning to define the requirement and agile acceptance test scenarios.

These preconditions resulted in new roles to emphasize these ownerships. The picture shows how our offshore scrum teams are organized. Each team consists of developers, QAs and a Scrum Master. These teams are located in India. The corresponding Product Owner (PO) is located in Munich. Three additional roles in Munich complete our “extended” scrum team. The Technology Owner (TO) is responsible for creating the technical concept of new features or change requests. They also coach the scrum team in new technologies or frameworks and they review the code developed by the scrum team to ensure the proper fit into the current architecture and alignment to the coding rules. The Quality Owner ensures the proper testing, identifies the testing scenarios and decides together with TO and PO which tests are needed on which level. He also supports the teams regarding the implementation of the testing scenarios.

In the last years we experienced the need for another role to address all operational needs early. By default, this happens during the initial definition of a new feature. Now we are able to identify necessary infrastructure or monitoring changes at a very early stage. Development and Operations are working closely together during the entire lifecycle of each feature. Some call this role DevOps Engineer, we call it Operation Owner. All four roles together form the QUAD. The responsibility of the QUAD starts with the grooming of upcoming features, but does not end there. It also helps with the definition of test scenarios, supports the team, and assures a smooth rollout to production.


Tech - 08/07/2016

PAYBACK visits DevOpsDays Amsterdam 2016

Afte20160629_091608r visiting conferences like DevOpsDays, your colleagues are usually curious about how it was. Well, DevOps Amsterdam 2016 was great!! Day 1 started with workshops all day long. It is a pity that you have to choose one out of 5 – 6 parallel workshops, as there are many interesting topics on the agenda. One of our favorites was “creating high-performance teams with DevOps behavior” from Dave van Herpen and Robert den Broeder.

The program with talks, ignites (limited to 5 minutes and 20 seconds per slide) and open-space-sessions started at day 2. There were a lot of interesting technical presentations, but non-tech-talks as well, like “What we’re learning about burnout and how a DevOps culture can help” from Ken Mugrage (Thoughtworks) or the opening keynote from Erica Baker (Slack) about diversity.


The audience honored20160701_095343 those non-tech-talks at least as much as the technical topics and people talked about them for the rest of the day. One of the technical hot-topics was “Docker”, which was mentioned in nearly every talk.

Another great aspect of DevOpsDays were the open space discussions. They always give good insights how other companies are dealing with the difficult topics. You are sitting together with some experts and talk about topics like “Information overload” or “container monitoring” and how others handle them.

For those who are not that familiar with the sponsoring of DevOps days: there are not only Gold, Silver and Bronze Sponsors, but also BBQ & Beer Sponsors, which was of course highly appreciated 😉

You can find more details about the event here.

Torben, Artem and Hans-Peter

Tech - 07/07/2016

Java & JMX – How to (finally) control your ports

When using JMX in Java you may struggle with an additional ephemeral port. This port cannot be controlled by a system property or some other config settings. When you’re searching for this issue in the web, you’ll find this bugreport, showing that it’s a known issue that Java opens an additional random port, which is not configurable when JMX is used. But in some cases you want to set the port to avoid sporadic NetBindExceptions. The more high ports are bound on a server, the more likely are NetBindExceptions.

A possible solution could be the following: Create your own LocalRMIServerSocketFactory class and set the same package as the „original“ LocalRMIServerSocketFactory. Use a custom system property to configure a new port and set the old given port as default value.

public class LocalRMIServerSocketFactory implements RMIServerSocketFactory { [....]
 public ServerSocket createServerSocket(final int port) throws IOException {
 int newPort = port;
 try {
 newPort = Integer.valueOf(System.getProperty("custom.jmxlocal.rmi.port", String.valueOf(port)))
 System.out.println("custom.jmxlocal.rmi.port: " + newPort);
 } catch (final NumberFormatException nfe) {
System.err.println("custom.jmxlocal.rmi.port: " + nfe.getMessage());
 return new ServerSocket(newPort) {

Now just create a jar file containing only the newly implemented class. Copy this jar file into the jdk1.8.0_xx/jre/lib/endorsed directory and add your new system property as argument. Now you are able to control the port easily.

java \ \ \ \ \

Tech - 01/07/2016

Global Scrum Gathering in Bengaluru

Iphone 109The PAYBACK Global development teams have been using Agile methods for 6 years. All of our teams (near shore & off shore) are involved in the same agile processes and face the same challenges when practicing distributed Scrum. Cultures, Collaboration and Dependencies are just some of those challenges. Over the past few years we have mastered most of the challenges we faced and have implemented our own collaboration model across the teams. Our models have evolved as we have progressed and improved continuously.

I was invited to share my insights to the processes and experiences in a distributed scaled agile environment at the Scrum Gathering in Bengaluru (India). Scrum Gatherings take place 3 times a year around the globe to share experiences among the Scrum Alliance. It was a great opportunity to meet Scrum Masters, Developers, Testers as well as Project Managers dealing with challenges in similar environments. I really enjoyed the discussions.

Don’t miss my next talk at the AgileWorld in Munich on 11st of July. I will be speaking about “Building & launching a worldwide loyalty platform“.