What is the best way to conduct an SEO audit using Screaming Frog without disabling JavaScript?

 To conduct an SEO audit using Screaming Frog without disabling JavaScript, you need to ensure that the crawler can render JavaScript and capture the full content of the pages, especially for websites that rely on dynamic loading. Here's a step-by-step guide:

1. Set Up Screaming Frog for JavaScript Rendering

Go to Configuration > Spider > Rendering.

Select “JavaScript” from the "Rendering" dropdown. This ensures that Screaming Frog loads the website with JavaScript, allowing it to crawl dynamic elements such as menus, content sections, and others that might not appear in a basic HTML crawl.

2. Adjust Crawl Settings

Adjust crawl depth and speed if necessary to ensure Screaming Frog can handle JavaScript rendering without overwhelming the server. Go to Configuration > Speed, and tweak the crawl limit or delay if needed.

Respect crawl rules (like robots.txt). Make sure Screaming Frog is respecting robots.txt and other directives to avoid crawling pages that should not be indexed. Configure this under Configuration > Robots.txt settings.

3. Run the Crawl

Start crawling the website as you normally would. Screaming Frog will now load the pages and their JavaScript-rendered content, capturing all elements for a thorough audit.

4. Audit Key Areas

Meta Data: Check for missing or duplicate meta titles, descriptions, and headers. Navigate to the "Page Titles" and "Meta Descriptions" tabs to identify problems.

Content: Review for thin content, duplicate content, or issues with key sections not being rendered properly.

Links & Navigation: Ensure all links are being crawled correctly, especially JavaScript-driven menus and links.

Mobile and Desktop Versions: Review the difference in rendering between mobile and desktop if necessary by configuring a mobile user agent in Configuration > User Agent.

Internal Linking and Structure: Identify any orphaned pages or crawl depth issues. JavaScript-heavy sites may hide internal links, so make sure they are all properly rendered and crawled.

Speed & Performance: Since JavaScript rendering can slow down pages, check for any performance issues that might arise during the audit.

5. Review Crawl Data

After the crawl, you can analyze all the data points Screaming Frog provides, including:

Indexability: Ensure all important pages are indexable.

Broken Links: Identify and fix any 404s or other HTTP errors.

JavaScript Content: Check that all JavaScript-generated content is visible and not blocked by issues like lazy loading errors.

6. Export Reports

Export the data from various sections like HTML, meta tags, content, and links to further analyze them or share with your team for improvements.

This setup allows you to conduct a full SEO audit with Screaming Frog while JavaScript is enabled, ensuring you capture the complete user experience and on-page elements.