Speeding up UI tests with CodeceptJS and Playwright

At Canva, we empower the world to design by providing a powerful and intuitive user interface that anyone can use. However, with 75 million people using Canva each month and more than 350 frontend engineers making 160+ changes per day, we must keep our frontend working as expected.

One way we ensure stability is by writing browser-functional tests that mimic how a user interacts with Canva. These functional tests are run for every change and must pass in our continuous integration (CI) pipelines before the changes are accepted into the Canva codebase.

In this blog post, we'll recount our Test Platform team's experience migrating browser-functional tests from Chimp, WebdriverIO, and Saucelabs to CodeceptJS and Playwright. We'll share our journey, the challenges we encountered, how we resolved them, and the lessons we learned.

Problems with our original framework

In 2017, we began rebuilding Canva's frontend architecture so that development could scale to handle 1000s of engineers working together. We decided to use a testing framework called Chimp 1.0 to write frontend functional tests using Cucumber, Webdriverio, and Typescript. Chimp 1.0 had some features that our team liked, such as watch mode, which enabled tests to run automatically when we made a local change.

While Chimp worked for us for several years, Chimp fell behind on updates and fixes for Cucumber and Webdriverio. The old Cucumber version had performance issues and processed all support files, even if you only wanted to run one test. As the number of tests grew from 100 to 1000, the test start-up time went from seconds to several minutes. We explored upgrading to the latest Cucumber and Webdriverio versions. However, we couldn't make the newer Webdriverio version scale in our extensive Typescript codebase because of the retranspilation speed (this has since been fixed). We were also on a fork of the project because of fixes needed for Chimp's Saucelabs support.

Our existing framework setup quickly gained negative sentiment amongst engineers. It reduced productivity and lowered quality because engineers were deferring writing tests because of its slowness. It was time to address this problem.

What did we do?

Because upgrading the internal components didn't work out, we switched to a whole new framework.

The new framework needed to:

• Support Typescript so that it was consistent with our frontend codebase.
• Have active development accepting external contributions so that it's less likely to become outdated.
• Support our existing test setup (Cucumber) while we migrated to the new framework.
• Support using drivers and libraries such as Playwright for web tests and Appium for web and mobile tests.
• Support testing on multiple browsers and devices, including using cloud-based browser testing services such as Saucelabs, Browerstack, or LambdaTest.
• Support headless browsers such as those provided by Playwright.

CodeceptJS fulfilled all our requirements and has good documentation and a great community.

A detailed look at our journey

We started our journey a year ago by planning out what the migration journey would look like. We had to customize our new framework so that it worked seamlessly with existing tests and to abstract away most of the complexities so that it supported running tests on both frameworks. We spent a good amount of time tailoring the framework set up and then updating the tests to work with the new framework.

How we set up our new framework

Chimp-like API (with pseudo-locators)

One of the guiding principles of the migration was keeping code changes to a minimum. We first decided to create a wrapper around the CodeceptJS API that resembled the Chimp API to support both test frameworks. This would allow us to inject the correct framework during runtime, depending on which framework we wanted to run.

Trying to get CodeceptJS to behave like Chimp proved harder than anticipated. There were fundamental differences in some APIs, and often there wasn't a matching API available in CodeceptJS. One example was how CodeceptJS used selectors.

WebdriverIO allowed the use of multiple selector strategies which involved chaining find element commands such as browser.$('.editor').$('.canvas').$('.element'). However, CodeceptJS only supported one level of chaining, which didn't satisfy our needs. To work around this, we introduced a pseudo-element object when using the find element command that internally kept track of the selectors. When actions were triggered, we would join the underlying selectors in the pseudo-element into one selector, such as browser.$('.editor .canvas .element').

Supporting sync and async APIs with codemods

The largest issue we faced was that the Chimp API was synchronous while the CodeceptJS API was asynchronous. Chimp used Fibers to make asynchronous actions synchronous but Fibers' authors announced they would no longer maintain this feature. All the other major test frameworks used asynchronous APIs too. So regardless of which framework we chose, we needed to change our tests to use asynchronous interactions.

Supporting both frameworks meant making our existing framework continue working while we started adding asyncs and awaits to the codebase. Unfortunately, adding asyncs and awaits caused the Cucumber steps in the old Cucumber version to run out of order, and there was no fix for it.

We then explored the possibility of using codemods at compile time to add asyncs and awaits. Using codemods was our preferred option because it meant we didn't have to update our test code, but it proved too difficult to apply the correct transformation for all scenarios to work. Array functions, such as forEach or filter, are examples that require complex transformations. These functions don't work with asyncs and awaits, and changing them from a function into a for-loop is not a straightforward transformation in most cases.

Instead, we did the opposite and wrote a codemod to remove asyncs and awaits to work for our existing framework. These codemods worked well but had some minor downsides, such as slightly skewed stack traces and slightly slower execution times, but were manageable.

This still meant we needed to add asyncs and awaits to all our tests. We'll talk about this challenge later in this post.

Watch mode

CodeceptJS doesn't provide watch mode out-of-the-box but this was a necessity for us. Almost all our engineers rely on watch mode to develop frontend tests as it provides a faster and seamless developer experience. So we used ts-node-dev to watch for any code changes and programmatically invoke our tests if any changes are made.

Supporting co-existing type declarations for both frameworks

Supporting two frameworks also caused clashes in Typescript typings. In our tests, we used a browser object to access the Chimp API and the CodeceptJS API, but only one of these APIs was available at a time. So we created a script to switch between the typing declarations whenever an engineer needed it. Enabling typing was important because it allowed engineers to access autocomplete that helped migrate missing API implementations and tests. In addition, this would help engineers become familiar with the new framework.

Improving Safari test runs

Playwright allows Webkit testing without the need for cloud-based cross-browser testing services. Using Playwright in our CI pipelines halved our CI Webkit test run times and increased our test coverage by 30%. We encountered many unique challenges configuring our CI agents to work with Playwright that we'll cover separately in a future blog post.

How we migrated our existing tests

Running both frameworks in CI

During the initial months of the migration, we ran all tests using the old and new test frameworks in parallel in CI to compare the test runs and catch any unforeseen problems. After we migrated enough tests and we had confidence with the builds, we started running migrated tests only in the new framework.

Migrating the actual test code

As mentioned previously, we needed to add asyncs and awaits to all the tests. Making a function asynchronous also meant any calling functions needed to become asynchronous. In the example below, adding an await to this.element.isExisting() in Button.getLabel() meant we also needed to add async and await in the step definition function that was calling it. We had many occurrences in our codebase ranging between two and five levels deep, and they all had to be updated.

// Chimp

getLabel() {
 const hasLabel = this.element.isExisting(styles.label);
 …
}

// CodeceptJS

async getLabel() {
 const hasLabel = await this.element.isExisting(styles.label);
 …
}

// Chimp

Then(/^the button label is "([^"]*)"$/, function(this: Scope, label: string) {
 expect(this.button.getLabel()).toBe(label);
});

// CodeceptJS

Then(/^the button label is "([^"]*)"$/, async function(this: Scope, label: string) {
 expect(await this.button.getLabel()).toBe(label);
});

While the change was straightforward, changing thousands of lines would be time-consuming.

Automating the code transformation

We took inspiration from our initial attempts using codemods to add asyncs and awaits during compile-time and created two scripts: one to add awaits and another to circle back and add asyncs.

To add awaits, we customized the eslint no-floating-promise rule to find asynchronous functions that were not awaited and wrote a fix function specific to our scenarios that would safely add awaits.

To add asyncs, we used ts-morph, a Typescript AST, to transform function expressions and arrow functions to asynchronous functions if they contained the keyword await.

We ran the scripts alternatively until all the asyncs and awaits were added. It wasn't the most elegant solution and needed some manual tweaking, but it saved a lot of time.

We finally made it!

Even though we aimed to automate the migration as much as possible, we knew we couldn't migrate all the tests within the Test Platform team. So we reached out to other teams through alignment requests and worked with each team's QA representative to keep teams accountable. Reaching out to other teams worked well, but the nature of alignment requests meant migration took months longer than expected because teams needed to plan the migration into their roadmap. The last test was migrated in mid-August, eight months after the migration started.

And we went further by reducing start-up times

When working with Cucumber, you must provide a configuration file detailing where to find feature files and their respective step definitions files. The default values **/*.feature and **/steps.ts would work quickly for most cases, but we had accumulated over 400 feature files and 600 step definitions files. For a developer running one scenario, it would take at minimum 90 seconds in Chimp and 45 seconds in CodeceptJS to start a test run on top-end Mac laptops. It was a painful experience waiting between changes.

To reduce the start-up wait times, we programmatically generated the Cucumber configuration file to contain only the feature files and steps definition files required for the test. First, we used the Gherkin AST to read the feature file steps that we wanted to run. Second, we used ts-morph to extract all step regexes from step definitions files and matched the regexes with the feature file steps mentioned previously. Using this method reduced the start-up time to between 6 and 10 seconds. This was a significant reduction in wait times for local execution, but we also saw faster executions in CI builds.

Additionally, we introduced a cache mechanism that would search the step definitions used from the previous run before searching all the step definitions. Using this cache meant searching only about 20 step definition files instead of 600. In most cases, a developer would want to run the same test multiple times, which worked great and reduced the start-up time to between 2 and 4 seconds.

Lessons learned

Keep your tooling updated

Setting aside time and resources to keep your tooling updated prevents large and complex updates in the future. While some brave engineers attempted to solve the problem as a side project, it ultimately needed more hands and dedicated time. Special shout out to Gustavo Henke_ ,_ George Crabtree and Joscha Feth.

Align with teams early and have a dedicated point of contact

When introducing outside work to teams, be mindful that each team handles new work differently, and this means allowing teams generous time to complete work. In our situation, we saw some teams migrating all their tests at once over several days while others slowly migrated their tests over several weeks. It's important to have a dedicated team contact who understands their team's context, can communicate any changes, and keep their team accountable.

Visualizing progress

During the migration we collected data on the number of tests remaining to be migrated and we created dashboards to visualise our progress. The dashboards were incredibly helpful when communicating with teams and understanding when teams needed assistance. We found that many teams were motivated by seeing their progress and would celebrate when they finished migrating their tests.

Acknowledgements

Kudos to the members of the Test Platform team — Srini Ramasamy, Benjamin Sejas, Ray Ru, Henry Pan, Klym Malakhov and Niko Georgievskiy — who worked on building the infrastructure to migrate to CodeceptJS. Special thanks to Joscha Feth, Gustavo Henke and George Crabtree for their early efforts in migrating our test framework. Thank you to all the QAs for working with your teams to finish off the migration.

Interested in UI testing infrastructure? Join Us!