Testing Search Engine Optimization Scalably without Breaking a Sweat

Search Engine Optimization (SEO) helps a website rank better in search results on search engines.

SEO can also be considered as a set of best practices to ensure your web pages are positioned or ranked higher in organic search results. Search is the best possible way for users to discover your web pages and it's completely free of charge. So having an excellent SEO strategy and ensuring there are no SEO regressions is quite critical.

SEO serves a very important function to Canva: getting Canva discovered and used for design by users around the globe. The magic of SEO is usually planned upfront and is done by a majority of people in an engineering team including content researchers, content writers, software developers, software testers, product managers, and many others. Each role is equally important.

In this post, we'll focus on the testing side of things when it comes to SEO, and how we approach SEO testing at Canva.

First, let us take a look at what organic search is.

What is organic search?

When you're interested in something and you search for it on the Internet, it's usually through a search engine such as Google, Bing, Baidu, Yahoo, and a few others. The returned search results can be categorized into 2 parts:

• Ads
• Organic

Ads can be useful in helping users navigate to your website and in generating more traffic. However, you need to pay for these Ads and the costs can grow significantly.

On the other hand, organic search results are completely free so you don't have to pay for them. These search results are based on the algorithms and ranking factors determined by the search engine. To satisfy the needs of the search engines and to ensure your web pages rank higher and are always consistently at the top, a LOT of effort needs to be put in, so there is a cost in terms of time and effort.

This is different to Paid marketing or Ads which belong to the SEM (Search Engine Marketing) territory.

A combination of ads and organic search is necessary for any web page to generate decent traffic as a lot of people do click on ads on mobile devices due to a lack of real estate but organic search results win in the longer run, as organic traffic increases and becomes more important.

Why is SEO important to Canva?

Canva's overarching mission is to empower the world to design — where every person with access to the Internet is using Canva regularly to unlock their creativity and meet their design needs. We need to be at the top of our SEO game and help users satisfy their intent by showcasing solutions at Canva.

For example, if a user wants to design a business card (intent), the first option they see is our business cards product page which will help them design their very own customized business card (solution) and eventually also help them print it (additional solution). The user being able to see our business cards product page as the first organic result is all the hard work of our SEO and Engineering teams and it helps Canva tremendously.

Organic results drive a lot of traffic to Canva and are the second-highest source of traffic. The main focus is usually on Google but can differ depending on the locale (e.g. Baidu in China).

How we test for SEO at Canva

There are different methods to test for SEO at Canva but they can broadly be classified into 2 groups: manual and automated. Both methods help us in identifying SEO issues and preventing them in the future.

Since SEO's benefits can only be reaped in the long term, any SEO issues found can negatively impact rankings and have a greater negative impact on Canva's vision. The impact of SEO issues will not immediately be noticeable until it's too late.

Manual SEO testing

SEO testing is as important as functional testing. There isn't much of a visual aspect to it but it's very important for pages that want to be ranked higher for search results. Manual SEO testing is done at different stages in Canva.

Quality Assurance (QA) Kickoffs

QA kickoffs are usually led by the QA engineers in collaboration with developers and sometimes product managers. As part of our QA kickoffs, we iron out scenarios that might affect SEO negatively. If the feature that's being developed changes the URL of our pages, we think about how we can prevent potential issues such as pages not being indexed or canonical URLs not being self-referenced. If the feature that's being developed introduces new assets on the page, we think about how we can prevent regressions to our CVW (Core Web Vital) metrics such as LCP (Largest Contentful Paint), FID (First Input Delay), CLS (Cumulative Layout Shift).

Thinking about these issues even before development begins gives us the edge when it comes to SEO testing. We can plan and prevent SEO blockers from the start.

Developer testing

Developers are more confident testing their code in terms of functionality as well as SEO by using the testing notes from QA kickoffs. Manual SEO testing is done by the developers themselves when functionally testing a feature at Canva. Any issues that arise when testing locally are fixed immediately.

Exploratory testing

QAs regularly do exploratory testing at Canva to ensure we have a quality product in the market. SEO testing is something that's considered by QAs while doing exploratory testing (usually if the pages that they are testing are optimized).

As a user, you can access Canva logged in (authenticated) or out (anonymous). Some teams work on features that are only accessible to logged-in users and hence SEO testing is out of scope for such teams. However, SEO testing is a top priority for teams that manage logged-out pages which are usually optimized.

Things that we test when doing exploratory SEO testing are

Disabling JavaScript (JS)

• It is important to check how a page renders without JS as this is what search engine crawlers would see
• Ensuring all the content on the page is still visible with JS disabled is one of the checks performed during exploratory SEO testing
• Nowadays, modern crawlers can render the page with JS, however the downside of that is that it consumes more crawl budget and is not ideal

Canonicals

• Canonicals play an important role in SEO as they help search engines determine the primary page to rank if multiple versions of a page exist
• When performing exploratory SEO testing, it is important to ensure that canonicals are present and self-referencing.
• For dynamic URLs (user-generated pages with URL parameters e.g. ?query= for search generated pages), these are usually canonicalized or have canonical tags pointing to their primary page

robots.txt

• robots.txt is perhaps the most important file when it comes to SEO testing as it directs search engine crawlers to accessible pages (pages that should be crawled) of your website and tells crawlers which pages are not accessible (pages that should not be crawled)
• During exploratory testing, we usually ensure that the right pages are allowed or disallowed for bot crawling in the robots.txt file

hreflang

• As Canva is available in 100+ languages, testing for localisation and hreflang rules is very important. hreflangattributes help search engines determine the relationship between web pages in alternate languages.
• As part of our testing, we ensure that hreflang attributes:
• are present
• are pointing to all supported locales for that page
• for the main page always have hreflang="x-default"

Lighthouse

• Lighthouse is an open-source tool from Google that helps you quickly uncover performance and SEO issues
• We use Lighthouse to quickly determine if basic SEO checks pass, and usually expect the score to be 100. Admittedly a perfect score is hard to achieve and not always necessary as there could be diminishing returns once Core Web Vitals and performance scores go high enough.

Test Parties

At Canva, we usually have test parties before making a feature available to end users. Test parties are a great way for the team to come together and test a feature thoroughly in an effective time-based manner. Manual SEO testing is also considered when running test parties. The different types of exploratory testing mentioned in the section above are also usually conducted during the test party

Post launch

We have monitoring in place to help us determine if the feature we've launched has had negative implications for SEO. Dashboards alert us if there's a drop in traffic or if unrelated pages are being ranked on Google.

The SEO dashboards also help us determine who's crawling our web pages including Google and other crawlers.

Automated SEO testing

Apart from Manual SEO testing, we have automated tests for SEO. We use different tools such as Deepcrawl and Datadog to help us test for SEO in an automated fashion and alert us if something goes wrong. We review tooling to remain consistent. We are proactive with the changing requirements of our SEO strategies and testing roadmaps. Note that we're not affiliated with any of the tools mentioned below — they're just good software that helps us get the job done in an efficient way.

Deepcrawl

We've been using Deepcrawl for the past 2 years. Not only does it allow us to run automated SEO tests but they also provide technical expertise every month as well as run frequent webinars. We generally use 2 tools from Deepcrawl:

Automation Hub (formerly known as Automator)

Automation Hub is Deepcrawl's tool that can be integrated into CI pipelines. Using this tool, you can set up a test suite and configure it to run heaps of tests with different thresholds to catch SEO issues on your web pages. During the configuration of a test suite, you can decide the starting URL, the type of crawl (list or web), crawl settings, tests, alerts, and CI connection settings.

We've configured multiple test suites to cater to different parts of the product and their test thresholds differ based on the requirements. These test suites are triggered throughout the day based on our release cycle as well as our daily schedule.

We've adopted an approach at Canva that allows us to run daily smoke and regressions SEO tests. After every release, we trigger a set of smoke test suites that run SEO tests against a small set of SEO-optimized pages. If a test fails, we instantly get notified on Slack and the issue is then investigated and triaged by the development team.

Aside from the post-release smoke tests, we also have a set of daily regression tests that are executed regularly. These tests are full-blown and test almost every SEO-optimized page at Canva. The alerting mechanism is the same as above for these tests.

These tests help us identify SEO issues such as

• 3xx
• 4xx
• 5xx
• Canonicals not present
• Canonicals pointing to non-200s
• Canonicals are not self-referencing
• Broken links
• Broken redirects
• Duplicate pages
• Empty pages
• Missing titles
• Unsupported hreflanglinks

Our teams work hard to fix these issues instantly if they are critical. Some are fixed over time and the test thresholds are updated accordingly.

Analytics Hub (formerly known as Deepcrawl Core)

Analytics Hub is Deepcrawl's main product which is a comprehensive cloud-based crawler as well as a monitoring and reporting tool. We use it to run ad-hoc full-blown crawls, analyze reports and observe the trends and patterns of issues occurring over time.

From the dashboard above, we can dive deeper into the Non-200 Pages report and identify exactly what's causing 49 URLs to return a non-200 status code. The reports are quite powerful and provide loads of sorting and filtering functionalities.

From the report above, we can see that all 49 URLs returned a 500 status code. This usually happens when we are not able to serve the page to crawlers. End users usually aren't affected by this.

Each failing URL can be analyzed further to investigate what the issue is by checking the response headers and other useful information.

Using the combination of Automation and Analytics Hub from Deepcrawl helps us sleep peacefully at night knowing that no SEO issues get past us.

Datadog Synthetics

We also use Datadog Synthetic tests or in order words Synthetic Monitoring from Datadog for automated SEO testing. This allows us to ensure our top-performing SEO pages are always up in all parts of the world.

These synthetic tests help us quickly identify if any of our top performing pages go down so that we can get them fixed as soon as possible, and back online.

In-house tools and tests

Apart from the above-mentioned tools, we also have a few in-house test frameworks and tools that help us test for SEO changes directly or indirectly. There are some test frameworks and tools that help check the performance of our pages in relation to Google's CWV metrics while some other test frameworks that help validate if a page is SSRed (server side rendered) properly.

Conclusion

SEO is a long-term game and its effects aren't always immediately apparent. It's best to keep on top of any issues you find to prevent them in the future. An issue happening today may hamper your SEO journey tomorrow.

With the right balance of manual and automated SEO testing, we strive to set a high bar of quality for SEO at Canva.

SEO testing is an innovative space, and there's a lot of potential for more improvements. As our founders often say, we've only achieved 1% of what we know is possible!

Acknowledgements

Kudos to all the people in the Marketplace and SEO groups for always encouraging me to try different things to solve challenging problems. Special thanks to

• Jake and Kollins: for always staying on top of all the SEO issues caught by our frameworks,
• Nikola, Sasha, Belen, and Linda: for sharing their experiences, insights, and feedback when it comes to SEO, and
• Mark and Hansel: for constantly providing guidance and constructive feedback

Massive shoutout to Chris Hew and Paul Tune for suggestions in improving this post.

Interested in applying for a QA role? Apply here!