Overcoming the challenges of JavaScript SEO
PWA Studio
Magento PWAs enables native app-like interactions, web push notifications, and home-screen installation. Yet, there is more. Distribution is much easier with standard web-like linkability, shareability – and no-app-store distribution. From a technical perspective, a PWA is a Single Page Application – a JavaScript app that runs within the browser environment, and therefore doesn't need to be downloaded or installed.
Progressive Web Apps built with Magento PWA Studio are probably one of the best moves for improving your customers' conversion rate and user experience. Using JavaScript as a PWA framework certainly provides a superior user experience, but it also leaves questions about how to create a crawler-friendly JavaScript application. The distribution model for the PWA is the web, for which organic search is vital. As a result, JavaScript SEO the key to success. Knowing that many JavaScript websites have troubles in having search engines fully crawl, render, and index their websites, how to optimize your PWA Studio for SEO becomes a vital question.
This post discusses the JavaScript SEO challenges that come with a storefront built with PWA Studio. I will use our own client PWA ("CFM", www.cloth-face-masks.com) as an example. PWA Studio gives you tools to create a React application (a JS framework) that will read data from a running Magento 2 instance via an UPWARD web server.
Why PWA Studio relies on JavaScript
I believe most web experts agree that JavaScript is the future of the web. Sure, HTML and CSS are and remain the foundation. But the best websites will come with JavaScript too. Why? JavaScript has the unique capacity to update the content of a page dynamically. JavaScript brings interactivity and dynamics to the web that's built on the static foundation of HTML markup.
Example types of content that are commonly generated by JavaScript are:
- Top products
- Pagination
- Internal links
- Reviews
- Comments
- Main content (less common)
- Auditing our PWA Studio storefront
Now, in our example PWA, what elements of our page depend on JavaScript? Well, the PWA is probably more than 90% depending on JavaScript. As a result, it is essential to have an in-depth look at whether Google can crawl, render, and index our PWA.
Before we get to that, let's have a look at how Google sees the website with JavaScript disabled and enabled so that you can understand the impact of the use of JavaScript:
JavaScript disabled
JavaScript enabled
As in fact, the website in reality looks like this:
As you can see, we identified a significant impact on SEO here. Major parts of the homepage are all JavaScript-based, like e.g., the "New Arrivals", "Trending Items" etc. But there is more. JavaScript also adds all relevant internal links:
When we compare the tags (most notably the title, description, meta robots, and canonical tags) before and after JavaScript is rendered, there is no change. That is in itself a good thing. However, please mark that the Page Title, and Description are of inferior quality.
When we compare the source code and the rendered DOM, we can see which chunks of the code are invisible when not indexed with full JS.
In short: The PWA relies heavily on JavaScript, and even when JavaScript is enabled, Googlebot has a tough time properly crawling, rendering and indexing our PWA.
Why does Googlebot have such trouble with JavaScript?
Well, Googlebot is capable of executing JavaScripts and crawl JavaScript-based website. It seems it is more a question of willingness.
Googlebot, and other bots, still prefer HTML websites, as crawling is easy and straightforward. Crawling a JavaScript-based website is more complicated and time-consuming: the Googlebot finds no links in the source code. As explained in the previous section, the links are only injected after executing JavaScript. So Googlebot needs to do a workaround and come back later to run the JavaScripts, and only then can it start rendering and indexing. In short, Googlebot is not a real browser. It may decide not to click a button etc., not to fetch all data. After all, Google has a limited crawl budget for your site.
Google refers to the above process as Googlebot's "two waves". First, it will crawl the site without JavaScript, then days, weeks, or even months later, they will slowly crawl your new pages with their Chrome engine to reveal content that's only available for browsers with JavaScript. Such delay is unacceptable for most website owners.
The solution for our example PWA Studio storefront: serving Googlebot a pre-rendered version of the website
It is clear that for whatever reason, Googlebot doesn’t index the JavaScript content in our case study. Luckily, there is a solution: implement server-side rendering (“SSR”). Why is that beneficial? It guarantees that search engine crawlers will see the fully rendered page, which translates to better SEO. In short, users are served the fully-featured JavaScript website. In parallel, the server will send Googlebot (and/or other search engine bots) a static version of the website.
As of today, there are a couple of options (I am aware of) for implementing SSR with PWA Studio. Let’s have a brief look:
- UPWARD: provides a solution for simple SSR rendering of page data and metadata. In short, UPWARD can reach a graphql endpoint, retrieve some data, and render this data in a mustache template. And voila, the static content from a product or CMS page is there to crawl for the “dumb” crawlers.
- SEOSnap.io: Pre-render with a headless browser crawler.
- Pre-rendering the React app with something like ReactDOMServer because it is a well-known and supported approach. This approach is doable, capable, and fast, but it requires a NodeJS Web server in production.
- Prerender.io: Routing bots to the prerender.io service to serve up pre-rendered HTML pages. The solution works similar to SEOSnap.io, except it uses a SaaS pre-renderer instead of a custom hosted one.
Conclusion
A PWA, built with PWA Studio acts like a native app and provides a better user experience.
Search engines can't smoothly process PWA applications as they are powered by JavaScript.
Server-side rendering is the best solution for serving the PWA Studio JavaScript application to the crawlers in a SEO-friendly way.