Merge Duty#

All code changes to Firefox land in the mozilla-central repository

  • The nightly releases are built from that repo twice a day.

  • DevEdition and Beta releases are built from the beta repository

  • Extended Support Releases follow-up from the relevant ESR repo, such as mozilla-esr115

  • Release and Release Candidates are built from mozilla-release repository

How are those repositories kept in sync? That’s MergeDuty and is part of the releaseduty responsibility.

Overview of Procedure#

MergeDuty consists of multiple separate days of work. Each day you must perform several sequential tasks. The days are spread out over nearly three weeks, with three major days of activity:

Historical context of this procedure:

Originally, the m-c -> m-b was done a week after m-b -> m-r. Starting at Firefox 57, Release Management wanted to ship DevEdition b1 week before the planned mozilla-beta merge day. This meant Releng had to merge both repos at the same time. With 71.0, we’re back to the initial workflow with merging m-b -> m-r in the first week and then m-c -> m-b in the follow-up week.

Do the prep work a week before the merge#

Do migration no-op trial runs#

Doing a no-op trial run of each migration has one major benefit these days: you ensure that the migrations themselves work prior to Merge day.

General steps#

  1. Go to Treeherder.

  2. Select the repo depending on the merge you want to perform (central, beta or the ESR one).

  3. On the latest push, click on the down arrow at the top right corner.

  4. Select “Custom push action…”

  5. Choose merge-automation from the dropdown and paste the payload provided in the sections below.

  6. In Treeherder, you’ll see a new push show up in Treeherder in the repo you will be merging to. It can take a few minutes for the push and task to appear.

  7. Click on the merge or bump tasks (not the Gecko decision task). A job details panel will pop up and from there you’ll find a link to the diff file in the artifacts tab. Note: There will be a cron job that kicks off another bump task with same th name, only one of them will contain the diff.

mozilla-beta->mozilla-release migration no-op trial run#

  1. Follow the general steps hopping on beta

  2. Insert the following payload and click trigger.

force-dry-run: true
behavior: beta-to-release
push: true

mozilla-central->mozilla-beta migration no-op trial run#

  1. Follow the general steps hopping on central

  2. Insert the following payload and click trigger.

force-dry-run: true
behavior: central-to-beta
push: true

mozilla-esr bump no-op trial run#

mozilla-esr branches evolve over time: Today (August 2023), mozilla-esr115 is the current esr, and that is used in the discussion below; in the future, you may need to substitute a different esr version number.

  1. Follow the general steps hopping on esr115

  2. Insert the following payload and click trigger.

force-dry-run: true
behavior: bump-esr115
push: true

Diff should be similar to this esr115 one.

Release Merge Day - part I#

When: Wait for go from relman to release-drivers@mozilla.org. Relman might want to do the migration in two steps. Read the email to understand which migration you are suppose to do, and then wait for second email. For date, see Release Scheduling calendar or check with relman

Merge beta to release#

  1. Close mozilla-beta. Check “Remember this change to undo later”. Please enter a good message as the reason for the closure, such as “Mergeduty - closing beta for $VERSION RC week”. Note: You need to be in the treestatus_users group in order to change trees.

  2. Run the m-b -> m-r no-op trial run one more time, and show the diff to another person on releaseduty.

  3. The diff for release should be fairly similar to this, with updated the version change.

  4. Submit a new task with force-dry-run set to false:

force-dry-run: false
behavior: beta-to-release
push: true
warning:

It’s not unlikely for the push to take between 10-20 minutes to complete.

warning:

If an issue comes up during this phase, you may not be able to run this command (or the no-op one) correctly. You may need to publicly backout some tags/changesets to get back in a known state.

  1. Upon successful run, mozilla-release should get a version bump and branding changes consisting of a commit like this and two new tags. The first tag should be in the form FIREFOX_RELEASE_xxx_END - where the xxx is the major Gecko version that Release had prior to the merge. The other tag should be in the form FIREFOX_RELEASE_yyy_BASE - where the yyy is the major Gecko version that Release now has.

  2. At the same time mozilla-beta should get a tag in the form FIREFOX_RELEASE_xxx_BASE - where the xxx is the major Gecko version that Beta had prior to the merge. (This should be the exact same tag and revision as the second one that you saw in the Release repo in step 1.)

  3. Verify changesets are visible on hg pushlog and Treeherder. It may take a couple of minutes to appear.

warning:

The decision task of the resulting pushlog in the mozilla-release might fail in the first place with a timeout. A rerun might solve the problem which can be caused by an unlucky slow instance.

Reply to relman migrations are complete#

Reply to the migration request with the template:

This is now complete:
* mozilla-beta is merged to mozilla-release
* mozilla-release is tagged and version bumped to XX.Y
* beta will stay closed until next week

Release Merge Day - part II - a week after Merge day#

When: Wait for go from relman to release-drivers@mozilla.org. For date, see Release Scheduling calendar or check with relman

Merge central to beta#

  1. Run the m-c -> m-b no-op trial run one more time, and show the diff to another person on releaseduty.

  2. The diff generated by the task should be fairly similar to this.

  3. Submit a new task with force-dry-run set to false:

force-dry-run: false
behavior: central-to-beta
push: true
warning:

It’s not unlikely for the push to take between 10-20 minutes to complete.

  1. Upon a successful run, mozilla-beta should get a version bump and branding changes consisting of a commit like this and two new tags. One tag should be in the form FIREFOX_BETA_xxx_END - where xxx is the major Gecko version that Beta had prior to the merge. The other tag should be in the form FIREFOX_BETA_yyy_BASE - where yyy is the major Gecko version that Beta now has.

    Click the first HG revision link (left side under date and timestamp) for the merge push to verify this.

  2. Verify that browser/locales/l10n-changesets.json has revisions, not default, and/or verify that the merge task has l10n-bump in the logs. You’ll need to click on the second HG revision link (commit message will be something like "no bug - Bumping Firefox |10n...") to verify this. The diff should look like this

  3. At the same time mozilla-central should get a new tag in the form FIREFOX_BETA_xxx_BASE - where xxx is also the major Gecko version that mozilla-central had prior to the version bump. (This should be the exact same tag and revision as the second one that you saw in the Beta repo in step 1.) It’s worth noting that we do not create FIREFOX_NIGHTLY_yyy_BASE tags, as we do for Beta & Release repositories.

  4. Verify changesets are visible on hg pushlog and Treeherder. It may take a couple of minutes to appear.

warning:

The decision task of the resulting pushlog in the mozilla-beta might fail in the first place with a timeout. A rerun might solve the problem which can be caused by an unlucky slow instance.

warning:

The merge day automation may not be idempotent. The merge automation task may fail and auto-retry (because of a worker shutdown, for instance). If the task retries after updating the state of the repo, it will update the state of the repo again, pushing repeated commits.

Re-opening the tree(s)#

Restore mozilla-beta tree to its previous state (approval-required) so that l10n bumper can run.

Tag central and bump versions#

What happens: A new tag is needed to specify the end of the nightly cycle. Then clobber and bump versions in mozilla-central as instructions depict.

  1. Follow the general steps

  2. Insert the following payload and click submit.

force-dry-run: false
push: true
behavior: bump-central

  1. Upon successful run, mozilla-central should get a version bump consisting of a commit like this and a new tag in the form FIREFOX_NIGHTLY_xxx_END - where xxx is the major Gecko version that mozilla-central had prior to the version bump.

  2. Verify changesets are visible on hg pushlog and Treeherder. It may take a couple of minutes to appear.

Bump ESR version#

Note: You could have one ESR to bump, or two. If you are not sure, ask.

Run the bump-esr no-op trial run one more time, and show the diff to another person on releaseduty.

Diff should be similar to this one.

Push your changes generated by the no-op trial run:

  1. Follow the general steps

  2. Insert the following payload and click submit.

force-dry-run: false
push: true
behavior: bump-esr115

Note The esr version is currently hardcoded to the action; If necessary, an action for other esr versions can be added to taskcluster/ci/config.yml.

  1. Upon successful run, mozilla-esr${VERSION} should get a commit like this.

  2. Verify new changesets popped on https://hg.mozilla.org/releases/mozilla-esr115/pushloghtml

Reply to relman central bump completed#

Reply to the migration request with the template:

This is now complete:
* mozilla-central is merged to mozilla-beta
* mozilla-central is tagged and version bumped to XXX.0a1
* mozilla-beta is tagged and version bumped to YYY.0b1
* mozilla-esr115 is version bumped to ZZZ.A.0esr
* newly triggered nightlies will pick the version change on cron-based schedule

Hint: verify current versions

Update wiki versions#

  1. Edit the new values manually: (ok to update a day early)

Do a manual check of product-details after the Nightly version bump#

When Nightly builds with a new version number are uploaded to the Mozilla archive, the Nightly version in ShipIt is automatically updated. When the version is updated, a message is sent to #releaseduty:

taskcluster-firefoxci
sheriffduty: ciduty: releaseduty: Updated firefox nightly version to `130.0a1`.

The product-details API should be updated automatically, but sometimes messages are dropped, or intermittent errors keep the API from being updated. It is a good idea to check https://product-details.mozilla.org/1.0/firefox_versions.json and make sure FIREFOX_NIGHTLY is set to the current value. If it is not, manually trigger a product-details rebuild from the ShipIt admin UI.

Release Merge Day - part III - release day#

Historical issues#

The merge day automation may not be idempotent#

The merge automation task may fail and auto-retry (because of a worker shutdown, for instance). If the task retries after updating the state of the repo, it will update the state of the repo again, pushing repeated commits.