Bartosz Frąckowiak

Technical Leader / Software Consultant

Branches and code merging at scale — automated branch merging

Intro

One day we decided that we don’t want to waste any more time on tasks that can be automated and we decided to start with automation of our merging process.

Today’s development teams use various branching models. Ranging from the straightforward ones, such as the trunk-based model, up to a completely autonomous continuous deployment process. Many consider continuous deployment to be a utopia for large projects and only achievable for small, startup projects.

We realized that with a sophisticated — multiple development loops — model, such as the one we use, automation is a necessity, not an option. As an example, at any particular moment, we might have a standard feature development loop living next to an emergency bug fixing loop (that needs to go to production asap) and a standard bug fixing loop (that can wait for two or more weeks) in addition. For various reasons, each loop is going to be deployed at some unknown time in the future. Every single loop usually comes with a dedicated branch and then developers create short-living branches that are eventually merged into this dedicated branch.

Pretty complex with multiple teams working towards one common regular deployment, isn’t it? Well, it was, until we decided to solve it with an internal tool — The Automerger. It’s a tool that we created during our “branching story” at the EcoVadis Engineering team and in this article, we’ll show you how it helped us…

Problem

Let’s examine the situation where the development team has three active branches. A common situation with GitFlow branching pattern:

Master branch is representing the current production state.

The first working branch is DevelopDevelopbranch contains all the features that are going to be included in a release in the future; moreover, it doesn’t matter when this release happens.

The second branch, let’s name it Release-8 is going to be deployed to production very soon. No feature development on this branch is happening at this precise moment. Release-8 is currently in the stabilization phase. Let’s think about it as it is in QA hands. Only bug fixes can go there.

What’s the dilemma? Let’s imagine a situation where one bugfix goes to Release-8 and does not go to develop. Next Release-9 would reintroduce bugs. A situation like that is unquestionably not desired. Yep, you can ask developers to put changes into two branches at the time, once for the Release-8 and second time to develop. Sure, but a human tends to make mistakes, someone can forget. Develop can be a way ahead of Release-8, which makes it hard to put the same change on two branches. A developer can simply omit this duty. That’s life, what can we do about it? Automate it.

This is where the Automerger tool comes in place. It is intended to detect branches assigned to the development loops and perform in-place merges between them systematically. Whenever it finds any conflicts during the merge, it notifies an appropriate team that user action is required.

Standalone tools

How can Automerger help my team/teams?

If your teams are in a position where you perform merges that are required by the branching strategy, or there are technical merges that have to be made periodically for any reason — Automerger comes into play. It takes the duty of your team and allows them to concentrate on delivering value to your company rather than completing boring merges. Do not lose time on tasks that can be done automatically.

What is Automerger?

Technically, it is a tool built with Cake. Sources can be found here: GitHub. It makes extensive use of the git command line output. Please feel free to fork the repo (or contribute to it via pull requests) and put your changes there. We will be delighted if you notify us 😉

Flow. How does it work inside?

One of the goals of the tool was to keep it as simple as possible. As an input, it takes two required parameters: source branch, target branch and executes the following steps:

  1. Create a temporary branch based on the source branch. The naming of the temporary branch can also be customized if you need.
  2. Merge to the temporary branch target branch.
  3. Parse output, confirm whether there are conflicts or not. If so, notify a particular developer to perform a manual merge procedure.
  4. Usually, when there is no conflict on the temporary branch, the script performs Cake task “Default”.
  5. After properly finish the default target, the script merges temp branch to the destination branch.
  6. Push changes to the origin.

There is one external dependency inside the specified flow. I mean point (4) with the execution of the “Default” target from the main Cake script. Its purpose is to validate the codebase after the merge. It is up to the script to run a build of the whole solution with unit test execution. It is necessary to validate the condition of the codebase at this point.

The Automerger is supposed to be an automatic tool, as few manual interventions as possible should be needed. Based on that assumption, there is a need for an automatic way to validate the correctness of performed merge-operations. The easiest way to achieve that is by building all projects with as many checks as possible. For example unit, integration or acceptance test. Of course, including more checks does not come for free — it increases the execution time significantly. Bear in mind that usually there are more than one of the executions of the Automerge script at a particular time.

The platform, only .Net?

The Automerger execution log

The Cake is Roslyn and .Net based. It allows you to write scripts using plain C#.

Automerger is C# based. But, this visible coupling to the .Net platform is not a blocker. The Automerger works with different platforms. At this moment at our place, the EcoVadis team has seven separate instances of the Automerger running simultaneously. Only two of them are for .Net based projects. The following five projects are JavaScript based frontend projects using React. It was achieved thanks to adjusting one of the steps performed by the script. Following point (4) from the flow described above, it is possible to adjust the script to perform a proper build of code based using JavaScript or Java, for example. You simply need to provide a command-line external tool that performs validation of code state.

Convention over configuration

Usually, the branching model defines the naming of branches that are created during development. The Automerger takes advantage of such conventions. Based on branch names, the Automerger knows which branches should be merged to which. Having branches named:

master, develop, release/release-8.0, release/release-8.1, release /release-9.0

The script parses names and sort branches in order:

  1. Master
  2. Release-9.0
  3. Release-8.1
  4. Release-8.0
  5. Develop

Depending on how many active release branches there are, it performs merging as follows Master -> Release-9.0 -> Develop. Introducing a new branch that follows the convention is automatically picked up by the tool. No additional configuration needed.

Limitations

Automerger requires a way to check whether changes on one branch are already present on the target branch. If you decide to perform proper merges from one branch to another, rather than doing cherrypicks, you are in a safe place. Git checks it for you, based on the hash of each commit. Automerger is based on this assumption. At this moment, if your team decides to go with cherrypicks and moving changes between branches with creating new commits each time. Automerger gets confused. It notifies you about the vast changes list and probably many conflicts.

Scalability and maintainability

The Automerger is present in the form of a script. Logically it is present as at least two separate files. These files need to be in a repository where the script is working. This limitation makes the Automerger a bit tricky to scale out and maintain in the longer run. Each time you create a new repository, it needs to be provided with those scripts inside. Each repo has a new instance of Automerger files. As a solution for this, you can create a template for your empty repository containing the Automerger files along with .gitignore file.

Getting all together

Automerger in CI/CD

The Automerger achieves full usability when it goes hand to hand with tools like Azure DevOps or any other CI/CD platform. CI part provides the possibility to schedule a build that executes the Automerger periodically. You can find out of the box build definition for VSTS here: GitHub.

The Automerger build configuration

Stats

After two years of using the tool, it proved itself working and merging in around 90%. What it means, loosely nine of ten times the Automerger do merge changes between branches without any merging conflict. Nine of ten times, a set of changes is propagated between branches without a single person knowing. The 10% scenario is a case when the Automerger informs the author about needed intervention.

We do not keep the data from the beginning of time due to data retention policy. Last 7 days of data at the moment when the article was published looks like:

FAQ

Is there a minimum requirement to run Automerger?

As long as you fulfil the requirements for the Cake build tool, you are fine. Minimal requirements do not come directly from Automerger itself. Requirements are more related to your building process itself. Please be sure you have put all required external dependencies in place to have a proper build process. For example, you have a smoke test calling Elasticsearch as a part of artefact verification, please be sure Elasticsearch instances are available from the build machine, etc..

How often do you run the script?

After a time, we have learned that as often as possible synchronization is crucial. Postponing merging of changes increases the risk of a potential merge conflict significantly, also increases the difference between the two branches. It is generally not recommended to perform the huge merge, way better is to have them more and smaller. We have decided to perform script runs every ~3–4 hours per working day. Usually it is 5am, 8am, 12pm, 3pm, 6pm. Hours were chosen empirically. We have found out it is a good balance between the execution time and development speed of the engineering teams and build agent occupation. You may want to pick your hours yourself. I suggest running it as often as possible.

You use Slack notifications, our team is using mail communication.

No problem with that, please feel free to make code changes with the mailing mechanism and share a pull request. I will be more than happy to cooperate. Please, also find that cake building tool provides a wide array of out-of-the-box plugins that can be used. You can find add-ins here: https://cakebuild.net/addins/

We have more than one team, 20 teams with 10 devs each.

No problem. Please find out file: .automergeConfig. It contains a simple mapping between developers and the team. Whenever the Automerger finds a conflict during merge, it looks for the last person that edited a particular file. Then, the script finds the right team and notifies the team — no need to notify everyone in your IT department.

Sure, but I want to notify everyone.

The default configuration will do it then 😉

Does The Automerger produce quite a several branches over time?

It depends on configuration. The script can go both ways. It can leave the branch with conflict for your team to follow later. Depending on how often you execute the script, it creates a new temporary branch each time. It is not recommended, we have found out that it is better to allow The Automerger to clean after itself. It allows work to be more asynchronous. The developer does not need to take care of conflicts before the next Automerger’s run. On the other hand, the Automerger’s build can run as much as it wants to not be blocked by the developer.

Summary

Automation is crucial. Do not let your team or yourself find a comfortable place, repeating the same task day by day. Building a tool for task automation allows you to move on and focus on tasks that bring company value. The Automerger tackles one of these day-by-day struggles. In this particular case, it is all about repeatable merges.

Technical Leader — identity disorder

https://medium.com/ecovadis-engineering/technical-leader-identity-disorder

Hi, my name is Bartosz Frąckowiak, for over three years now I’m the so-called Technical Leader at EcoVadis and I would like to tell you my story 🙂

A technical leader can be just a simple position in the corporate hierarchy. It can be a safe position reserved for a person technically based with some leadership skills. Opposite to team leaders who generally should have strong leadership skills and technical base, not necessarily be a guru in programming stack. The question you should ask yourself is what do you expect from your career?

One of the ingredients is your attitude. I have never wanted to make this title just a corporate position to me. The footer in corporate mail means nothing to me. After three years of being a technical leader at EcoVadis, I’ve learnt a lot about what it’s like to be one of them (in this particular environment). I have ups and downs, one day I would never change this position to a different one, the next day I was almost ready to quit and move on. I would like to break down this position into smaller pieces for you. Let’s start with splitting just the name “technical leader”. It has two words in it. The first part is strictly “technical”, the second is “leader”. I will try to address these two parts inside this article, explain what it means to me. What I do to be the best possible technical leader.

“Technical”

The technical part is all about widely known Engineering craftsmanship. The whole category is extremely wide, let’s split it even more.

Self-development

The first one is the self-development in the context of engineering knowledge. So simple and yet so complex at the same moment. The technical leader needs to join the technical mastery journey no matter what he/she knows already. A technical leader is responsible for analyzing new approaches, new shiny bits and bobs that come to our engineering market. Follow trends, join conferences, etc. As the market is going fast in various directions, it is a tough nut to crack.

Usually, not experienced developers are highly motivated. Discovering the engineering world just makes them even happier to go further and further. Somewhere on the road, the developer finds out that it is almost impossible to be a specialist in every branch of engineering.

After some time you find it spiritually releasing to decide not to try to be a specialist in everything, because it is almost impossible. Just find your thing or two, that drives you, makes you sleepless thru the night, makes you searching for the next piece of information about a particular topic. On other topics be up-to-date, more or less. Do you like data science? It’s fine not to know specification differences between open API specification version 3.0 and 2.0. Just simply know it’s there, you will deep dive into details when needed.

“Let him who would move the world first move himself.” ― Socrates

Using knowledge

The second chapter is using engineering knowledge in day-to-day workspace. Making the right decisions and leading the team to the chosen common goal — which should describe where the IT department wants to be in the long run. Technical leader’s responsibility is to make sure the goal itself should correlate to a high-level shape of the software architecture. Architecture simple as possible at the same time meeting all the needs of the business. Easy to develop and easy to maintain in the long run.

Finding this architecture is a never-ending story. The company evolves and software should follow the development as well.

“Clean code is not written by following a set of rules. You don’t become a software craftsman by learning a list of heuristics. Professionalism and craftsmanship come from values that drive disciplines.” ― Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship

Being a medium

The third category is being the technical medium/connector between the architect, other teams and the team itself. At EcoVadis at the moment we have eight teams with seven developers each. This is not space when lack of proper communication can be accepted. Discussing with other technical leaders and architect takes place twice per week.

Being a connector between the technical leader’s community and the team is an important task which needs to be done every day. Making sure the team knows where we are heading. It’s crucial to the whole IT environment to know what decision has been made and why the technical leader’s community took this approach. In the long run, communication lets the team understand the goal where we are heading, therefore, be more independent in making decisions and at the end be more effective.

“The single biggest problem in communication is the illusion that it has taken place.” ― George Bernard Shaw

“Leader”

Each technical leader has a “leader” inside position title. Leader means leadership for the team. How to be the best one for the team? Here I will present to you tangible ideas and practices that are introduced inside my team. I am sure that thanks to those, my team has improved a lot during these three years. Are those rules golden bullet suited for every team in every corporation? For sure, not… Simply looking from a statistical point of view, those will not fit every environment. I hope I can inspire you a bit and encourage discussion.

Dev knowledge sharing

The first idea is about sharing knowledge in a safe space. I have booked one hour per week, usually at the end of the week when everyone is thinking about the weekend already. Idea is to create a safe space to share any topic you want.

One rule only applies, you have to talk about engineering. There is no constraint that the topic has to be work-related or has to suit the project that the team is working on. Simply, you have something to share? Come and talk about it. What’s more, there are no rules about the presentation itself. If a developer wants to have a high-end presentation it is appreciated, if a developer presents an idea from scratch of paper that’s fine as well.

The idea found interest quickly and became a permanent part of the team’s life. We had meetings where all of us were reviewing pull requests with some new approach as well as meeting about machine learning algorithms. From TDD to ideas on how to optimize our team efficiency. From coding conventions to project managing strategies. Anything developers are interested in is suited for the local dev knowledge sharing.

After three years of hosting these meetings, I found out that the secret here is feeling safe inside the team. Each presenter received feedback during the meeting knowing that feedback is only for his good and self-improvement. There are no mean jokes, nobody criticizes that the topic isn’t important in any context. It is simply time for a developer to shine and be proud of his knowledge.

These meetings evolved from local dev knowledge sharing to company-wide meetings. We keep both, local teams meetings stays the same — safe place as it was designed before. There is also a place to share the knowledge with the bigger crowd — to the whole IT department. Sure, it requires a little bit more preparation than just a piece of paper and an idea. Nice presentation is one of the unwritten requirements. The audience is multinational, so it also requires globalization for your presentation. Globally hosted knowledge sharing meetings are a much bigger deal than the local one. Both meetings are held hand by hand. In the end, the outcome is to improve team knowledge with no additional cost.

“If you have knowledge, let others light their candles in it.” ― Margaret Fuller

Meetups

Starting from local dev knowledge sharing idea, we have moved even one step further. We hosted a meetup. The first meetup was about the approach to one of the projects we were involved in. We had space and time to learn (learn the hard way) TDD inside DDD.

We decided to share our journey with the local community. The event took place at our office, it was in the workshop formula and it was specially designed to guide participants to the same mistakes we made. We showed how we handle challenges and what we had learned. More about the event itself can be found in the article.

Technical excellence meetings

On the other side, inside the team, we have team members NOT resources. Each team member is a human being with different backgrounds, different needs and different goals.

As a technical leader, there is enough space to show myself useful. The idea is to independently develop each team member’s skills based on his/her needs. The concept name is “technical excellence”. There are two kinds of meetings during the month, the first meeting is quick and it takes up to half an hour. The goal of the meeting is to quickly find an area that will be later developed. Ideas usually come from the developer. He/she knows better what he/she is interested in (well, sometimes it is not the case as some people need a small hint). This meeting is a planning and negotiation meeting for the next one that is coming. My role in these meetings is to ensure that the idea is valid at the basic level and try to find common ground between developer needs and company goals.

With having some plans in place, there is a proper longer meeting usually around two hours once or twice per month. This relatively small time is completely for the developer. We can do whatever he/she wants as long as it brings educational value. During these three years, we had meetings about: blockchain, testing, many topics related to DDD or specific technologies inside Azure features. During the meeting, we can either research the topic itself, work on problems related to this topic or even do a presentation that will be shown later on the IT department knowledge sharing meeting.

The ones that touched me the most are about self-improvement. Majority of my team members found themselves in a place where they want to take a “step forward”. The “step” that will distinguish them from every other backend developer. Those meetings are the most heart touching, the hardest ones, but also the most inspiring and rewarding ones. I would recommend it to each technical leader, try to talk not only about facts and technology. Find a way to inspire team members, even a little bit.

The success of those meetings is based on properly planned time for the developer way ahead. Each of us has some topic that he/she wants to handle when the time allows. The truth is that there will be no free time, what’s more, you will have less and less “free time” as your career evolves. Those meetings address the problem, the time frame can not be moved, every participant needs to be prepared. The meeting must take place.

Every year there is a time of appraisals, the time when your performance gets reviewed. What is important to me, there are feedback forms that let others put opinions about me as a technical leader. In the past, I found a lot of mentions about technical excellence in the feedback forms’ and most of them were very positive. For myself, it is simply a win and a way to go.

Along with those meetings, I found two things that make me wonder a bit. Since each meeting was with myself and another team member, I had to be always prepared for every topic. This made a bit of pressure on my calendar. The idea I want to test is to follow the same meeting pattern, but team members would do the meeting to themselves. I would just watch at the very beginning and help until the point when I will be no longer needed. (At this moment I am part of the team containing very experienced developers, the idea might not work with the less experienced team)

The second matter that bothers me a bit is pure egoistic. Each team member gets a few hours per month to develop himself, but what about me? 🙂 Answer is simple: I do technical excellences with myself (sounds like a split personality). I simply book some time during the month for myself and learn what I’m interested in the most, at the moment hype is on deep learning.

“Be a yardstick of quality. Some people aren’t used to an environment where excellence is expected.” ― Steve Jobs

Pair programming

Again, simple as that and powerful as that. Pair programming in any form that suits developer and situation. Starting from just simple sitting together and being a rubber duck for the partner, to hardcore pure pair programming with the whole team inside the dedicated room. Having not just one additional pair of an eye but three members watching. Live coding session with a large screen presentation and discussion about specific problems.

Which method should we choose? The rule of thumb is to pick “small” (sitting together and switching keyboards from time to time) pair programming to smaller problems, like bugs, before planned tasks or user stories and “the whole team is watching” meeting for solutions that will affect the future work of everyone. Usually, meetings about core decisions made at the beginning of the project or changes in the direction where the team was heading.

A lesson learned here for technical leaders is to be prepared to monitor the involvement of others during “everyone is watching” meetings. “Tool” is so simple and so powerful, in terms of team building and knowledge sharing is one of the best in my opinion. Be careful not to fall into the trap of being “The Devil’s Advocate” which causes frustration. Usually stepping back and taking a huge breath is the cure. After all, being pragmatic and empathic is crucial here.

http://geek-and-poke.com/

Open source initiative

Teams at EcoVadis are backend based. Usually, we have a 5:2 ratio of the backend to frontend developers inside the team. Team members produce source code daily. Most of the code is usually not visible to the end-user, therefore, it is harder to appreciate the craftsmanship of individuals.

Here, the idea was to share our concepts to the external world. Share it with the IT community in the form of YouTube videos, source codes on GitHub and articles on medium.com exactly like this one. Each IT member can express himself/herself to the community about the topic he/she had mastered during nine to five work. Every content is signed by the developer’s name, not anonymous content. The sharing idea brings both sides clear benefits. Developers build a brand for their names as well as build a brand for the EcoVadis as a company with a strong engineering culture.

“Open source is a development methodology; free software is a social movement.” ― Richard Stallman

Power of small changes

All the tools/methods described here are working for my team. What I’ve found during the few years of being a technical leader is that the team evolution is faster than you think. Each member works hard to become a better version of himself/herself not only during work hours but also after the nine to five period. To keep everyone interested there is a constant need for small improvements, small changes that make the technical leader’s toolset fresh and joyful for everyone. If you find yourself in the same corporate position as mine, please share your experiences with the set of tools that you have developed over time.

“Great things are not done by impulse, but by a series of small things brought together. “― George Eliot