Case Study: How the Cookie Monster Ate 22% of Our Visibility

Very last year, the team at Homeday — a person of the primary assets tech firms in Germany — made the conclusion to migrate to a new content administration program (CMS). The ambitions of the migration were being, among other factors, enhanced page speed and developing a point out-of-the-art, upcoming-proof internet site with all the necessary characteristics. One particular of the principal motivators for the migration was to permit content editors to work far more freely in building internet pages without the need of the assist of builders. 

Soon after evaluating various CMS choices, we made the decision on Contentful for its modern day technology stack, with a top-quality knowledge for both of those editors and builders. From a specialized viewpoint, Contentful, as a headless CMS, lets us to decide on which rendering technique we want to use. 

We’re presently carrying out the migration in various stages, or waves, to decrease the risk of challenges that have a significant-scale unfavorable impression. During the very first wave, we encountered an concern with our cookie consent, which led to a visibility loss of almost 22% within 5 days. In this posting I am going to describe the issues we were facing in the course of this 1st migration wave and how we fixed them.

Setting up the very first test-wave 

For the to start with check-wave we selected 10 Search engine marketing pages with substantial website traffic but very low conversion charges. We founded an infrastructure for reporting and monitoring those people 10 internet pages: 

• Rank-monitoring for most pertinent keywords 

• Web optimization dashboard (DataStudio, Moz Pro,  SEMRush, Research Console, Google Analytics)

• Standard crawls 

Right after a thorough preparing and testing section, we migrated the initially 10 Search engine optimisation pages to the new CMS in December 2021. Even though a number of worries happened during the tests section (amplified loading occasions, bigger HTML Doc Item Model, and so on.) we made the decision to go are living as we failed to see large blocker and we required to migrate the to start with testwave ahead of xmas. 

Initial performance overview

Really thrilled about obtaining the to start with phase of the migration, we took a appear at the performance of the migrated internet pages on the upcoming working day. 

What we saw future definitely didn’t please us. 

Right away, the visibility of tracked keyword phrases for the migrated pages lessened from 62.35% to 53.59% — we lost 8.76% of visibility in a person working day! 

As a result of this steep fall in rankings, we done yet another considerable spherical of testing. Amid other points we examined for protection/ indexing concerns, if all meta tags were being bundled, structured info, inside inbound links, web page pace and cell friendliness.

Second performance evaluation

All the article content experienced a cache day soon after the migration and the written content was absolutely indexed and getting read through by Google. In addition, we could exclude many migration possibility components (improve of URLs, written content, meta tags, format, and so on.) as sources of mistake, as there hasn’t been any variations.

Visibility of our tracked key terms endured a different drop to 40.60% above the future couple days, generating it a total fall of just about 22% inside of 5 days. This was also plainly demonstrated in comparison to the competitors of the tracked keywords (in this article “approximated traffic”), but the visibility appeared analogous. 

As other migration possibility aspects plus Google updates had been excluded as resources of errors, it certainly experienced to be a technological concern. Too significantly JavaScript, minimal Main World-wide-web Vitals scores, or a larger sized, far more advanced Document Item Model (DOM) could all be potential causes. The DOM represents a website page as objects and nodes so that programming languages like JavaScript can interact with the site and transform for illustration design, composition and written content.

Pursuing the cookie crumbs

We experienced to recognize problems as speedily as achievable and do swift bug-fixing and lessen much more adverse results and visitors drops. We at last obtained the initially genuine trace of which technological explanation could be the cause when just one of our equipment showed us that the quantity of web pages with large external linking, as properly as the quantity of web pages with highest content material dimensions, went up. It is critical that webpages never exceed the highest information sizing as internet pages with a extremely large amount of money of body written content may possibly not be totally indexed. Regarding the high external linking it is significant that all exterior back links are reputable and appropriate for end users. It was suspicious that the number of exterior back links went up just like this.

Both equally metrics ended up disproportionately higher in contrast to the quantity of pages we migrated. But why?

After examining which exterior back links experienced been included to the migrated webpages, we observed that Google was looking through and indexing the cookie consent sort for all migrated internet pages. We carried out a web page look for, examining for the articles of the cookie consent, and noticed our concept verified: 

This led to various challenges: 

1. There was tons of duplicated material designed for each individual site owing to indexing the cookie consent sort. 

2. The content material dimension of the migrated webpages significantly amplified. This is a problem as internet pages with a very significant amount of human body content material may well not be completely indexed. 

3. The amount of external outgoing one-way links significantly elevated. 

4. Our snippets all of a sudden showed a date on the SERPs. This would advise a website or news report, though most articles or blog posts on Homeday are evergreen material. In addition, thanks to the date showing up, the meta description was slice off. 

But why was this going on? In accordance to our support supplier, Cookiebot, research engine crawlers entry websites simulating a entire consent. For this reason, they attain access to all written content and copy from the cookie consent banners are not indexed by the crawler. 

So why was not this the circumstance for the migrated webpages? We crawled and rendered the pages with unique person brokers, but nevertheless couldn’t find a trace of the Cookiebot in the supply code. 

Investigating Google DOMs and looking for a alternative

The migrated web pages are rendered with dynamic details that arrives from Contentful and plugins. The plugins comprise just JavaScript code, and sometimes they occur from a associate. 1 of these plugins was the cookie manager associate, which fetches the cookie consent HTML from outdoors our code foundation. That is why we failed to discover a trace of the cookie consent HTML code in the HTML resource information in the 1st spot. We did see a much larger DOM but traced that again to Nuxt’s default, much more complex, more substantial DOM. Nuxt is a JavaScript framework that we do the job with.

To validate that Google was reading through the copy from the cookie consent banner, we made use of the URL inspection resource of Google Lookup Console. We when compared the DOM of a migrated webpage with the DOM of a non-migrated site. Within just the DOM of a migrated website page, we last but not least discovered the cookie consent content material:

One thing else that obtained our attention were being the JavaScript information loaded on our old pages compared to the information loaded on our migrated webpages. Our website has two scripts for the cookie consent banner, presented by a 3rd bash: one to demonstrate the banner and grab the consent (uc) and just one that imports the banner content material (cd).

• The only script loaded on our previous internet pages was uc.js, which is liable for the cookie consent banner. It is the one script we need in every web page to tackle person consent. It shows the cookie consent banner devoid of indexing the articles and will save the user’s final decision (if they agree or disagree to the utilization of cookies).

• For the migrated webpages, aside from uc.js, there was also a cd.js file loading. If we have a web site, the place we want to demonstrate far more information about our cookies to the consumer and index the cookie info, then we have to use the cd.js. We imagined that both of those documents are dependent on each and every other, which is not appropriate. The uc.js can run alone. The cd.js file was the purpose why the articles of the cookie banner got rendered and indexed.

It took a while to come across it since we imagined the second file was just a pre-prerequisite for the very first 1. We decided that simply just eliminating the loaded cd.js file would be the resolution.

General performance review right after implementing the resolution

The working day we deleted the file, our search term visibility was at 41.70%, which was nonetheless 21% reduce than pre-migration. 

Even so, the working day after deleting the file, our visibility elevated to 50.77%, and the following day it was pretty much back to typical at 60.11%. The believed targeted traffic behaved likewise. What a reduction! 

Conclusion

I can picture that a lot of SEOs have dealt with little challenges like this. It appears to be trivial, but led to a considerable drop in visibility and targeted traffic all through the migration. This is why I recommend migrating in waves and blocking adequate time for investigating technical problems before and right after the migration. Moreover, preserving a close look at the site’s functionality in just the months following the migration is essential. These are surely my key takeaways from this migration wave. We just accomplished the next migration wave in the beginning of May perhaps 2022 and I can condition that so far no significant bugs appeared. We’ll have two much more waves and full the migration ideally efficiently by the close of June 2022.

The overall performance of the migrated web pages is virtually back to regular now, and we will proceed with the following wave.