How to Find All Current and Archived URLs on an internet site

How to Find All Current and Archived URLs on an internet site

Blog Article

There are lots of causes you could possibly need to seek out the many URLs on a web site, but your exact objective will determine Whatever you’re hunting for. For example, you might want to:

Detect every indexed URL to research issues like cannibalization or index bloat
Accumulate existing and historic URLs Google has seen, especially for internet site migrations
Obtain all 404 URLs to Get better from post-migration errors
In Every single circumstance, one Resource gained’t Present you with almost everything you would like. Regretably, Google Lookup Console isn’t exhaustive, along with a “site:instance.com” look for is limited and tough to extract data from.

In this particular post, I’ll walk you through some applications to construct your URL list and prior to deduplicating the data utilizing a spreadsheet or Jupyter Notebook, based upon your web site’s measurement.

Old sitemaps and crawl exports
For those who’re in search of URLs that disappeared through the Are living web site a short while ago, there’s an opportunity somebody in your crew could have saved a sitemap file or possibly a crawl export before the changes were made. If you haven’t now, check for these documents; they're able to frequently offer what you would like. But, should you’re looking at this, you almost certainly didn't get so Blessed.

Archive.org
Archive.org
Archive.org is a useful Device for Web optimization responsibilities, funded by donations. If you look for a domain and select the “URLs” choice, you could entry approximately ten,000 listed URLs.

On the other hand, There are several constraints:

URL limit: You could only retrieve as much as web designer kuala lumpur 10,000 URLs, which can be insufficient for much larger sites.
Quality: Many URLs may be malformed or reference resource data files (e.g., photographs or scripts).
No export choice: There isn’t a designed-in approach to export the listing.
To bypass The dearth of the export button, use a browser scraping plugin like Dataminer.io. Even so, these constraints imply Archive.org might not supply a complete Answer for larger websites. Also, Archive.org doesn’t reveal whether or not Google indexed a URL—however, if Archive.org discovered it, there’s a good prospect Google did, also.

Moz Professional
Whilst you might normally make use of a connection index to search out exterior web pages linking to you personally, these instruments also explore URLs on your web site in the process.


How you can utilize it:
Export your inbound back links in Moz Professional to obtain a speedy and easy list of concentrate on URLs from the web site. If you’re addressing a massive Site, consider using the Moz API to export information over and above what’s manageable in Excel or Google Sheets.

It’s imperative that you Take note that Moz Pro doesn’t ensure if URLs are indexed or found out by Google. However, considering the fact that most web pages use exactly the same robots.txt rules to Moz’s bots as they do to Google’s, this process frequently operates very well to be a proxy for Googlebot’s discoverability.

Google Search Console
Google Look for Console features several valuable resources for creating your listing of URLs.

Inbound links studies:


Comparable to Moz Professional, the One-way links segment presents exportable lists of goal URLs. Regretably, these exports are capped at 1,000 URLs Every single. You can use filters for distinct web pages, but since filters don’t utilize into the export, you might have to depend on browser scraping resources—restricted to five hundred filtered URLs at any given time. Not best.

Overall performance → Search Results:


This export provides a list of webpages getting research impressions. When the export is proscribed, You can utilize Google Lookup Console API for greater datasets. You can also find cost-free Google Sheets plugins that simplify pulling much more extensive facts.

Indexing → Pages report:


This area provides exports filtered by concern style, though these are typically also minimal in scope.

Google Analytics
Google Analytics
The Engagement → Web pages and Screens default report in GA4 is a superb source for amassing URLs, which has a generous limit of a hundred,000 URLs.


Better yet, you may apply filters to develop diverse URL lists, successfully surpassing the 100k Restrict. As an example, if you would like export only weblog URLs, stick to these techniques:

Action 1: Include a phase to the report

Move two: Click “Make a new section.”


Step three: Define the section with a narrower URL pattern, such as URLs that contains /blog/


Observe: URLs located in Google Analytics might not be discoverable by Googlebot or indexed by Google, but they offer worthwhile insights.

Server log data files
Server or CDN log data files are Maybe the ultimate Instrument at your disposal. These logs seize an exhaustive record of every URL path queried by users, Googlebot, or other bots through the recorded period.

Issues:

Knowledge size: Log documents could be massive, a great number of web sites only keep the final two weeks of information.
Complexity: Analyzing log information might be complicated, but various resources are available to simplify the procedure.
Blend, and fantastic luck
As you’ve gathered URLs from each one of these resources, it’s time to combine them. If your internet site is small enough, use Excel or, for much larger datasets, resources like Google Sheets or Jupyter Notebook. Be certain all URLs are persistently formatted, then deduplicate the listing.

And voilà—you now have an extensive list of recent, outdated, and archived URLs. Fantastic luck!