Install an audit crawler
Run your own diagnostic crawl and turn findings into a fix list.
An audit crawler runs on your computer or service to inspect your site. It is different from Google's crawler. A small HTTP fetcher checks responses; it does not execute JavaScript or prove that a search engine will index a page.
Your implementation route
Use available Wix SEO/log reports plus Search Console and Analytics. Keep an external uptime check if needed. Advanced event or AI API code requires supported backend functionality or a separate backend; the examples here are portable patterns, not Wix drop-ins.
Platform boundaries & account requirements
Use Wix's native integrations and SEO & GEO tools. Backend examples belong in a supported Wix backend module or an external service, never in a page embed.
Open Wix ↗Authorization to audit the site, a low request rate, a known sitemap and a folder for exported results.
Follow the steps
Check only after completingInstall a desktop crawler
Download Screaming Frog SEO Spider from its official site and install the matching Windows/macOS/Linux package. Review current free/license limits. Start with the production homepage in Spider mode; avoid login-protected content and use a conservative crawl speed.
Set a bounded first audit
Limit scope to your hostname, exclude logout/cart actions and uncontrolled URL parameters, and honor robots rules. For a large site, begin with a sample of priority pages. Use JavaScript rendering only when necessary and supported by your edition; it costs more resources.
Export the actionable findings
Inspect response codes, internal links, missing/duplicate titles, canonicals, directives and sitemap conflicts. Save the crawl/export with the date. A duplicate title is a diagnostic clue; prioritize broken revenue journeys and inaccessible content first.
Fix, then compare
Assign each material issue a URL, owner, change and expected result. Re-run the same configuration after fixing it. The Python alternative below audits an explicit list only, honors robots and has a request timeout; it is deliberately not a full rendering crawler.
Implementation recipes
Expand · choose · copy</>A bounded audit of your own URL listOpen recipe
Save as audit.py, edit ORIGIN and PATHS, then run python audit.py (Windows: py audit.py). Python 3 standard library; no packages needed. It checks at most 20 explicit URLs and does not execute JavaScript.
import csv, time
from urllib.request import Request, urlopen, build_opener, HTTPRedirectHandler
from urllib.error import HTTPError, URLError
from urllib.robotparser import RobotFileParser
from urllib.parse import urljoin, urlsplit
ORIGIN = "https://example.com"
PATHS = ["/", "/services/heat-pump-repair"]
AGENT = "SeoAIPlaybookAudit/1.0"
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None # Record redirects; do not follow out of scope.
opener = build_opener(NoRedirect)
robots = RobotFileParser()
try:
with opener.open(Request(ORIGIN + "/robots.txt",
headers={"User-Agent": AGENT}), timeout=15) as r:
robots.parse(r.read(500000).decode("utf-8", errors="replace").splitlines())
except HTTPError as e:
if e.code == 404:
robots.parse([]) # No robots file.
else:
raise SystemExit("Robots unavailable; inspect it before auditing.")
except URLError:
raise SystemExit("Robots unavailable; inspect it before auditing.")
rows = []
for path in PATHS[:20]:
url = urljoin(ORIGIN, path)
if urlsplit(url).netloc != urlsplit(ORIGIN).netloc:
continue
if not robots.can_fetch(AGENT, url):
rows.append([url, "robots blocked", "", ""])
continue
time.sleep(max(1, robots.crawl_delay(AGENT) or 0))
try:
with opener.open(Request(url, headers={"User-Agent": AGENT}),
timeout=15) as r:
rows.append([url, r.status, r.headers.get("Content-Type", ""),
r.headers.get("X-Robots-Tag", "")])
except HTTPError as e:
rows.append([url, e.code, e.headers.get("Location", ""), ""])
except URLError:
rows.append([url, "network error", "", ""])
with open("audit.csv", "w", newline="", encoding="utf-8") as out:
writer = csv.writer(out)
writer.writerow(["url", "status", "type_or_redirect", "x_robots_tag"])
writer.writerows(rows)
print("Saved audit.csv. Inspect redirects and page content separately.")Review the setup notes and replace example values before using this on your site.Save before/after reports for the same URL set. Verify the repaired URLs in a browser and in Search Console where appropriate; do not treat a clean crawler report as a ranking result.
Stop the crawl if load increases, reduce scope/rate, and restore any page changes separately. The audit itself is read-only.
Read the primary instructions
Guidance checked October 7, 2026. “2027” describes this prototype’s visual concept, not a prediction of platform rules.