885456 - Schedule full-stack emulator b2g unittests on m-c, inbound and try

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

They've already been scheduled on cedar. It looks like crashtests, reftests and xpcshell tests are already green and can be immediately scheduled on the new build for all branches (and subsequently removed from the ics_armv7a_gecko build).

Mochitests have some issues I'm currently investigating, and Mn tests have some intermittent failures.

Aki, would you prefer I filed separate bugs per test suite or just keep track of everything here?

Comment 1

[Aki Sasaki (not active)]

I'm ok tracking things here.  I would like to see the results from https://tbpl.mozilla.org/?tree=Cedar&rev=1be607d22e17 so we can decide what to do with debug this first pass.

Comment 2

[Aki Sasaki (not active)]

Attachment: (configs) full emulator test rollout, part 1

This is complex enough that I need to do a builder list dump & compare.
Also, our mozilla-tests/b2g_config.py has gotten seriously ugly.  Hopefully as we get rid of the old emulator builds it'll get less so.

Comment 3

[Aki Sasaki (not active)]

Attachment: builder list diff

This is the diff in builders.

This may be more far-reaching than we'd like, or it may normalize things in a way we wanted in the first place.  I'm not sure I know.

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 766116 [details] [diff] [review]
(configs) full emulator test rollout, part 1

Also see the builder diff -- that's what I used to do a short eyeball verification.

Comment 5

[Phil Ringnalda (:philor)]

Looking at Cedar, the right builder diff would only be adding emulator_vm crashtest-1 and -2 and -3, emulator_vm xpcshell, and emulator (without the vm) reftest-1 through -10. We can't run reftests on VMs yet, but since we can run crashtests and xpcshell on VMs, we don't have any need to run those on hardware.

Comment 6

[Andrew Halberstadt [:ahal]]

Comment on attachment 766171 [details] [diff] [review]
builder list diff

Review of attachment 766171 [details] [diff] [review]:
-----------------------------------------------------------------

What philor said. I think we only want to run each suite on either a VM or physical hardware, but not both. I believe R is the only suite not capable of running on the VM's at the moment so:

C, X on b2g_emulator_vm
R on b2g_emulator

::: clean_builder
@@ -3755,4 @@
> -b2g_emulator cedar opt test crashtest-1 ScriptFactory
> -b2g_emulator cedar opt test crashtest-2 ScriptFactory
> -b2g_emulator cedar opt test crashtest-3 ScriptFactory
> -b2g_emulator cedar opt test marionette-webapi ScriptFactory

Hard to tell from the diff, but is this removing the marionette-webapi on b2g_emulator from cedar? Or will it still be running on the VM? We still need to fix some test failures for this suite on cedar.

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 766116 [details] [diff] [review]
(configs) full emulator test rollout, part 1

Review of attachment 766116 [details] [diff] [review]:
-----------------------------------------------------------------

> +b2g_emulator_vm birch opt test crashtest-1 ScriptFactory

This doesn't work yet. See bug 818968.

::: mozilla-tests/b2g_config.py
@@ +279,5 @@
>          'unittest-env': {'DISPLAY': ':0'},
>          'enable_opt_unittests': True,
>          'enable_debug_unittests': False,
>          'fedora-b2g': {
> +            'opt_unittest_suites': MOCHITEST + MARIONETTE,

this change would affect all branches, including b2g18*, on which we don't use Ubuntu yet (trains...) I'm not sure if we want to switch to Ubuntu with this patch.

Comment 8

[Aki Sasaki (not active)]

Ok.  I was trying to do the right thing and not make our configs fuglier, but it looks like our requirements won't allow for that.

So: can someone tell me specifically what I should turn off and what I should turn on, or should I try to figure it out from the above multiple comments and most likely get it wrong again?

Comment 9

[Andrew Halberstadt [:ahal]]

So the latest Cedar push made all of the crashtests and reftests (which were previously green) orange. I don't know why, but I think you should hold off on this bug for a little while.

Sorry, it is super confusing with the two emulator builds + vm/hw + opt/debug. I just wanted to switch the suites over incrementally so we could avoid bustage like what just happened. But looks like we'll need to spend some more time on them now on Cedar.

Comment 10

[Aki Sasaki (not active)]

Probably due to https://bugzilla.mozilla.org/show_bug.cgi?id=885802#c5 .

Comment 11

[Andrew Halberstadt [:ahal]]

(In reply to Rail Aliiev [:rail] from comment #7)
> > +b2g_emulator_vm birch opt test crashtest-1 ScriptFactory
> 
> This doesn't work yet. See bug 818968.

It looks like crashtests work, it's just reftests that don't. So...

Suites to enable:
b2g_emulator_vm opt - C, X
b2g_emulator opt - R

Suites to disable:
ics_armv7a_gecko_vm opt - C, X
ics_armv7a_gecko opt - R (also C, X on cedar)

I haven't seen b2g_emulator results for debug yet, so let's leave that as is.

Comment 12

[Andrew Halberstadt [:ahal]]

...and now cedar reftests/crashtests are busted for both the old and new emulator builds

Comment 13

[Aki Sasaki (not active)]

Do we know what happened? It's odd to me that old emulator tests are busted, but m-c isn't.

Comment 14

[Andrew Halberstadt [:ahal]]

No, I'm not sure. It looks like bug 882888 except not intermittent. I'll see if I can find some time to look at it tomorrow.

Comment 15

[Andrew Halberstadt [:ahal]]

I think I fixed the OnReftestLoad problem, so when bug 882888 lands we can go ahead with this. The mochitests got fixed with bug 885454 and are also now all green. To clarify:

Step 1 - enable suites:
B2G Emu opt - R, Mn
B2G Emu (VM) opt - M, C, X

Step 2 - disable suites:
B2G Arm opt - All tests and builds!

Notes:
* Ideally we want to run Mn tests on the (VM) pool, but there are still some intermittent oranges there. If we land on the hardware pool first we'd be able to nuke the ics_armv7a_gecko builds wholesale. If you'd prefer to have them running on the (VM) pool right away we could disable the tests or look into the failures first.
* We might want to do step 1 before step 2 just to be safe, though I'll leave that up to you.

Comment 16

[Andrew Halberstadt [:ahal]]

Also just to clarify, this is only for m-c based branches. The b2g18 branches should remain as is.

Comment 17

[Aki Sasaki (not active)]

Attachment: as requested

Thanks Andrew!

Comment 18

[Aki Sasaki (not active)]

Attachment: builder list diff

I'm guessing the cedar bit is just a re-ordering.

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 768668 [details] [diff] [review]
as requested

https://hg.mozilla.org/build/buildbot-configs/rev/82f84bdf4492

Comment 20

[Aki Sasaki (not active)]

Reconfiged.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

Any reason birch was left out on this? As the main integration branch for b2g, I would think running these builds and tests there would be of high importance.

Comment 22

[Aki Sasaki (not active)]

Weird, I didn't special case birch at all.

Comment 23

[Ryan VanderMeulen [:RyanVM]]

I see them running now. Sorry for the false alarm.

Comment 24

[Andrew Halberstadt [:ahal]]

This has been running for awhile now without problem. I'll file a new bug for turning off the old builds.

Comment 25

[Armen [:armenzg]]

I see this in buildbot-configs:
1092 # emulator hacks.  See bug 885456
1093 # MERGE DAY This will someday ride trains...
1094 for branch in BRANCHES.keys():
1095     if branch in ('mozilla-aurora', 'mozilla-beta', 'mozilla-release',
1096                   'mozilla-esr17', 'mozilla-b2g18_v1_0_0',
1097                   'mozilla-b2g18_v1_0_1'):
1098         if 'emulator' in BRANCHES[branch]['platforms']:
1099             del BRANCHES[branch]['platforms']['emulator']

When is the code suppose to be riding the trains?

Comment 26

[Aki Sasaki (not active)]

(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #25)
> I see this in buildbot-configs:
> 1092 # emulator hacks.  See bug 885456
> 1093 # MERGE DAY This will someday ride trains...
> 1094 for branch in BRANCHES.keys():
> 1095     if branch in ('mozilla-aurora', 'mozilla-beta', 'mozilla-release',
> 1096                   'mozilla-esr17', 'mozilla-b2g18_v1_0_0',
> 1097                   'mozilla-b2g18_v1_0_1'):
> 1098         if 'emulator' in BRANCHES[branch]['platforms']:
> 1099             del BRANCHES[branch]['platforms']['emulator']
> 
> When is the code suppose to be riding the trains?

Probably now, now that b2g26 is riding the trains.