Merge Duty¶
All code changes to Firefox land in the mozilla-central repository
The
nightly
releases are built from that repo twice a day.DevEdition and Beta releases are built from the beta repository
Extended Support Releases follow-up from the relevant ESR repo, such as mozilla-esr91
Release and Release Candidates are built from mozilla-release repository
How are those repositories kept in sync? That’s MergeDuty
and is
part of the releaseduty
responsibility.
Overview of Procedure¶
MergeDuty
consists of multiple separate days of work. Each day you
must perform several sequential tasks. The days are spread out over
nearly three weeks, with three major days of activity:
Do the prep work a week before the merge
On Merge day:
A week after Merge day, bump mozilla-central:
Historical context of this procedure:
Originally, the m-c
-> m-b
was done a week after m-b
->
m-r
. Starting at Firefox 57
, Release Management wanted to ship
DevEdition b1
week before the planned mozilla-beta merge day. This
meant Releng had to merge both repos at the same time. With 71.0, we’re
back to the initial workflow with merging m-b
-> m-r
in the
first week and then m-c
-> m-b
in the follow-up week.
Do the prep work a week before the merge¶
Assign to yourself the migration bug¶
A tracking migration bug should’ve already been filed - please assign that to yourself. If there isn’t one (e.g. bug 1694412), please contact the last releaseduty owner and file one. You can find more of this in Release owners.
Do migration no-op trial runs¶
Doing a no-op trial run of each migration has one major benefit these days: you ensure that the migrations themselves work prior to Merge day.
General steps¶
Go to Treeherder.
Select the repo depending on the merge you want to perform (central, beta or the ESR one).
On the latest push, click on the down arrow at the top right corner.
Select “Custom push action…”
Choose
merge-automation
In Treeherder, you’ll see a new push show up in Treeherder in the repo you will be merging to. It can take a few minutes for the push and task to appear.
Click on the merge or bump tasks (not the Gecko decision task). A job details panel will pop up and from there you’ll find a link to the diff file in the artifacts tab. Note: There will be a cron job that kicks off another bump task with same th name, only one of them will contain the diff.
mozilla-beta->mozilla-release migration no-op trial run¶
Follow the general steps hopping on beta
Insert the following payload and click submit.
force-dry-run: true
behavior: beta-to-release
push: true
mozilla-central->mozilla-beta migration no-op trial run¶
Follow the general steps hopping on central
Insert the following payload and click submit.
force-dry-run: true
behavior: central-to-beta
push: true
mozilla-esr bump no-op trial run¶
mozilla-esr branches evolve over time: Today (October 2021), mozilla-esr91 is the current esr, and that is used in the discussion below; in the future, you may need to substitute a different esr version number.
Follow the general steps hopping on esr91
Insert the following payload and click submit.
force-dry-run: true
behavior: bump-esr91
push: true
Diff should be similar to this esr78 one.
Sanity check no blocking migration bugs¶
Make sure the bug that tracks the migration has no blocking items.
Release Merge Day - part I¶
When: Wait for go from relman to release-drivers@mozilla.org. Relman might want to do the migration in two steps. Read the email to understand which migration you are suppose to do, and then wait for second email. For date, see Release Scheduling calendar or check with relman
Merge beta to release¶
Close mozilla-beta. Check “Remember this change to undo later”. Please enter a good message as the reason for the closure, such as “Mergeduty - closing beta for $VERSION RC week”.
Run the
m-b -> m-r
no-op trial run one more time, and show the diff to another person on releaseduty.The diff for
release
should be fairly similar to this, with updated the version change.Submit a new task with
force-dry-run
set to false:
force-dry-run: false
behavior: beta-to-release
push: true
- warning
It’s not unlikely for the push to take between 10-20 minutes to complete.
- warning
If an issue comes up during this phase, you may not be able to run this command (or the no-op one) correctly. You may need to publicly backout some tags/changesets to get back in a known state.
Upon successful run,
mozilla-release
should get a version bump and branding changes consisting of acommit
like this and atag
like thisIn the same time
mozilla-beta
should get a tag like thisVerify changesets are visible on hg pushlog and Treeherder. It may take a couple of minutes to appear.
- warning
The decision task of the resulting pushlog in the
mozilla-release
might fail in the first place with a timeout. A rerun might solve the problem which can be caused by an unlucky slow instance.
Release Merge Day - part II - a week after Merge day¶
When: Wait for go from relman to release-drivers@mozilla.org. For date, see Release Scheduling calendar or check with relman
Merge central to beta¶
Run the
m-c -> m-b
no-op trial run one more time, and show the diff to another person on releaseduty.The diff generated by the task should be fairly similar to this.
Submit a new task with
force-dry-run
set to false:
force-dry-run: false
behavior: central-to-beta
push: true
- warning
It’s not unlikely for the push to take between 10-20 minutes to complete.
Upon a successful run,
mozilla-beta
should get a version bump and branding changes consisting of acommit
like this and atag
like this. Click the first HG revision link (left side under date and timestamp) for the merge push to verify this.Verify that
browser/locales/l10n-changesets.json
has revisions, notdefault
, and/or verify that the merge task has l10n-bump in the logs. You’ll need to click on the second HG revision link (commit message will be something like"no bug - Bumping Firefox |10n..."
) to verify this. The diff should look like thisIn the same time
mozilla-central
should get a tag like thisVerify changesets are visible on hg pushlog and Treeherder. It may take a couple of minutes to appear.
- warning
The decision task of the resulting pushlog in the
mozilla-beta
might fail in the first place with a timeout. A rerun might solve the problem which can be caused by an unlucky slow instance.- warning
The merge day automation may not be idempotent. The merge automation task may fail and auto-retry (because of a worker shutdown, for instance). If the task retries after updating the state of the repo, it will update the state of the repo again, pushing repeated commits.
Re-opening the tree(s)¶
Restore mozilla-beta tree to its previous state (approval-required) so that l10n bumper can run.
Tag central and bump versions¶
What happens: A new tag is needed to specify the end of the nightly
cycle. Then clobber and bump versions in mozilla-central
as
instructions depict.
Follow the general steps
Insert the following payload and click submit.
force-dry-run: false
push: true
behavior: bump-central
Upon successful run,
mozilla-central
should get a version bump consisting of acommit
like this and atag
like thisVerify changesets are visible on hg pushlog and Treeherder. It may take a couple of minutes to appear.
Bump ESR version¶
Note: You could have one ESR to bump, or two. If you are not sure, ask.
Run the bump-esr no-op trial run one more time, and show the diff to another person on releaseduty.
Diff should be similar to this one.
Push your changes generated by the no-op trial run:
Follow the general steps
Insert the following payload and click submit.
force-dry-run: false
push: true
behavior: bump-esr91
Note This is currently set to esr91
; the defaults can be
overridden in-tree in taskcluster/ci/config.yml
or specified here
using an action payload such as:
force-dry-run: false
push: true
behavior: bump-esr
to-branch: esr91
to-repo: https://hg.mozilla.org/releases/mozilla-esr91
Upon successful run,
mozilla-esr${VERSION}
should get acommit
like this.Verify new changesets popped on https://hg.mozilla.org/releases/mozilla-esr91/pushloghtml
Update wiki versions¶
Edit the new values manually: (ok to update a day early)
Next release date. This can be found in the release calendar <https://wiki.mozilla.org/Release_Management/Calendar>. This updates
Update the releng changelog¶
Edit the changelog by adding a new section for the current Nightly release. You can also delete the previous section if it is empty.
Bump Nightly version and release dates in ShipIt¶
ShipIt currently hard-codes the version of Nightly that’s being released, as well as the release dates.
It doesn’t get automatically updated because it would need to know when a new nightly was available, not just when the version had been updated in-tree. Everything up to merging this pull request can be done early, but the PR must not be merged before the first nightly has been built and published with the new version. However, it does need to be merged and deployed to production <https://github.com/mozilla-releng/shipit#production> as soon as reasonably possible after nightly is built. Note: the product details API needs to be manually updated by pressing the gear icon in shipit
git clone git@github.com:mozilla-releng/shipit.git
git checkout -b nightly_version_bump_${version}
Edit FIREFOX_NIGHTLY’s major version in https://github.com/mozilla-releng/shipit/blob/f3d45d1dd1cc08cc466865f7d39305f1b2edbcf7/api/src/shipit_api/common/config.py#L49
Edit the LAST and NEXT known dates (all 6 of them) at https://github.com/mozilla-releng/shipit/blob/f3d45d1dd1cc08cc466865f7d39305f1b2edbcf7/api/src/shipit_api/common/config.py#L54-L59
Commit, and submit a pull request
Attention! Thunderbird follows the same procedure as Firefox so they usually follow-up with a similar PR in the same day. In order to minimize the number of deployments to production, please make sure to deploy both PRs at the same time. If there hasn’t been a PR up-to-review yet, please ping/CC https://github.com/jfx2006 in your PR (or ping @rjl in #releaseduty to make them aware.
Merge the pull request(s) after a new nightly version has been pushed to CDNs: once the new nightly version appears at https://archive.mozilla.org/pub/firefox/nightly/latest-mozilla-central/
Push ShipIt
main
branch toproduction
Monitor the versions info page. If it does not automatically update, you may need to click
Rebuild product-details
in the ShipIt UI.
Close migration bug, file one for the next release¶
Once release is out of the door on Tuesday, close the existing bug tracking this release, from initial step and clone that bug into a similar one, tracking the next release. Please CC all the RelEng team. One can find the next release date in Release owners.
Historical issues¶
The merge day automation may not be idempotent¶
The merge automation task may fail and auto-retry (because of a worker shutdown, for instance). If the task retries after updating the state of the repo, it will update the state of the repo again, pushing repeated commits.