Background: 3×3 at LinkedIn
At LinkedIn, we pride ourselves on our 3×3 system: the notion that we should be able to ship code to production three times a day, with no more than three hours between releases, so that our members experience the latest and greatest of our platform. With 3×3, we’re shipping more code than ever to our members, which in turn reduces the amount of time we have to manually test our releases. In order for us to confidently ship to production three times a day, we need to be able to heavily rely on the health of our automated tests, as well as on our test infrastructure as a whole.
SPAs and test (in)stability
LinkedIn.com is built as a SPA, and if your site is built using Ember, you’re in the same boat. Building LinkedIn as a SPA offers many benefits to our members, such as faster page loads between routes and fewer round trips between the client and the server. On the other hand, developing SPAs comes with unique challenges, such as managing memory and asynchrony. In a traditional non-SPA, the browser gives us a clean slate on each page reload. We do not have this luxury in a SPA environment. If care is not taken when writing a SPA, it’s easy to end up with asynchronous code, such as xhr requests and setTimeouts executing after the components that initiated these calls have been destroyed. This can lead to nasty side effects, a poor user experience, and test instability. More on this later.
Testing asynchronous code in Ember: A crash course
Let’s talk about the wait helper (recently renamed to settled), Ember’s solution to testing asynchronous code. The wait helper, at the most basic level, is a utility function that returns a promise that will resolve when all asynchrony in the application has been completed. In Ember, an asynchronous timer is created with a call to Ember.run.* methods (e.g., Ember.run.later). Each of these methods call setTimeout under the hood, hence the name “asynchronous timer.” Because the wait helper pauses the execution of the test runner while at least one async timer exists, it’s a great way to ensure that all async code has been completed prior to running your assertions.
Is using the wait helper causing test timeouts? You may have a leak.
This section makes references to “leaking timers,” a timer that’s set up at some point during the application’s lifecycle but not torn down when the application is destroyed.
At LinkedIn, there was a point in time when a lot of our tests were timing out, and we weren’t sure what the cause was. We eventually noticed that many of the tests timing out were using the wait helper, which pauses the test runner until all asynchrony has finished. Had the same set of tests been timing out consistently, we could have safely assumed that the application code being run by the tests in question was likely leaking async timers. Unfortunately, though, different tests were failing between multiple executions of our test runner, which led us to believe that we likely had timers leaking between tests. From here, we needed to come up with a way to a) prove that we had async timers leaking in our test suite; and b) locate the source of said leaks in order to remove them from our application code.
Identifying leaking timers in Ember
Within the Ember.run namespace, Ember exposes a handy method called hasScheduledTimers, which returns a boolean (true if there are any running async timers). As seen in the code below, combining hasScheduledTimers with QUnit’s testDone method makes for a convenient way to confirm async leaks in tests, as well as to pinpoint the source of the leaks.