How To Find All Pages On A Website

Tabular array of Contents

Why you demand to find all the pages on your site

How your content actually gets to be seen

What is crawling and indexing?

Links

Sitemaps

CMS

What is indexing?

Using robots.txt

Using 'noindex'

What are orphan pages?

How do orphan pages come virtually?

How about dead-end pages?

Where do dead-end pages come from?

What are hidden pages?

Should all hidden pages be done away with?

Newsletter sign ups

Pages containing user information

How to detect hidden pages

Using robots.txt

Manually finding them

How to find all the pages on your site

Using your sitemap file

Using your CMS

Using a log

Using Google Analytics

Manually typing into Google's search query

What and so practise yous do with your URL list?

Manual comparison with log data

Using site crawling tools

SEOptimers SEO crawl tool

In decision

Call up about information technology. Why do y'all create a website? For your potential customers or audience to hands find you and for you lot to stand out among the contest, correct? How does your content really become to be seen? Is all the content on your site always seen?

Why you demand to notice all the pages on your website

It is possible that pages containing valuable data that actually needs to be seen, do not get to be seen at all. If this is the case for your website, then you are probably losing out on significant traffic, or even potential customers.

At that place could also be pages that are rarely seen, and when they are, users/visitors/potential customers hit a expressionless-end, as they cannot access other pages. They can only leave. This is every bit merely as bad as those pages that are never seen. Google will begin to note the high bounciness rates and question your site'due south credibility. This volition see your web pages rank lower and lower.

How your content really gets to be seen

For users, visitors or potential customers to meet your content, itch and indexing needs to be done and washed frequently. What is crawling and indexing?

What is crawling and indexing?

For Google to show your content to users/visitors/potential customers, information technology needs to know first that content exists. How this happens is via crawling. This is when search engines search for new content and add it to its database of already existing content.

What makes itch possible?

• Links
• Sitemaps
• Content Management Systems (CMS – Wix, Blogger)

Links:

When yous add a link from an existing page to another new page, for case via ballast text, search engine bots or spiders are able to follow the new page and add it to Google's 'database' for future reference.

Sitemaps:

These are also known as XML Sitemaps. Here, the site owner submits a list of all their pages to the search engine. The webmaster can too include details similar the last date of modification. The pages are then crawled and added to the 'database'. This is all the same non real fourth dimension. Your new pages or content will not be crawled every bit soon every bit you submit your sitemap. Itch may happen after days or weeks.

About sites using a Content Management System (CMS) machine-generate these, so it's a chip of a shortcut. The simply fourth dimension a site might non have the sitemap generated is if you created a website from scratch.

CMS:

If your website is powered by a CMS like Blogger or Wix, the hosting provider (in this case the CMS) is able to 'tell search engines to clamber whatsoever new pages or content on your website.'

Here'south some information to assistance yous with the process:

Adding a sitemap to WordPress

Viewing the sitemap

Where is sitemap for Wix?

Sitemap for Shopify

What is indexing?

Indexing in simple terms is the adding of the crawled pages and content into Google'southward 'database', which is really referred to as Google'southward alphabetize.

Before the content and pages are added to the alphabetize, the search engine bots strive to understand the page and the content therein. They fifty-fifty go alee to catalog files like images and videos.

This is why as a webmaster, on-page SEO comes in handy (page titles, headings, and utilise of alt text, among others). When your page or pages take these aspects, it becomes easier for Google to 'understand' your content, catalog it appropriately and index it correctly.

Using robots.txt

Sometimes, you may not want some pages indexed, or parts of a website. You lot need to give directives to search engine bots. Using such directives also makes crawling and indexing easier, equally there are fewer pages being crawled. Learn more than about robots.txt hither.

Using 'noindex'

You can as well this other directive if in that location are pages that you exercise not want to appear in the search results. Learn more about the noindex.

Earlier yous start calculation noindex, you'll want to identify all of your pages so you can clean upwardly your site and arrive easier for crawlers to clamber and index your site properly.

What are some reasons why you lot need to find all your pages?

What are orphan pages?

An orphan page can be defined as ane that has no links from other pages on your site. This makes it almost impossible for these pages to be found by search engine bots, and in addition past users. If the bots cannot find the page, and then they will not show information technology on search results, which farther reduces the chances of users finding it.

How do orphan pages come about?

Orphan pages may result from an attempt to continue content private, syntax errors, typos, duplicate content or expired content that was not linked. Here are more means:

• Test pages that were used for A/B testing and that were never deactivated
• Landing pages that were based on a season, for example, Christmas, Thanksgiving or Easter
• 'Forgotten' pages every bit a result of site migration

How nearly expressionless-end pages?

Unlike orphan pages, dead-end pages accept links from other pages on the website but do not link to other external sites. Dead-end pages examples include thank you pages, services pages with no call to actions, and "naught plant" pages when users search for something via the search option.

When you have dead-end pages, people who visit them only have ii options: to leave the site or go dorsum to the previous folio. That means that you are losing significant traffic, especially if these pages happen to be 'primary pages' on your website. Worse yet, users are left either frustrated, confused or wondering, 'what'due south next'?

If users get out your site feeling frustrated, confused or with any negative emotions, they are never probable to come up back, merely similar unhappy customers are never likely to buy from a brand again.

Where practise dead-end pages come up from?

Dead end-pages are a result of pages with no calls to action. An example here would be an about page that alludes to the services that your company offers only has no link to those services. Once the reader understands what drives your company, the values you uphold, how the company was founded and the services y'all offer and is already excited, yous need to tell them what to practice next.

A simple call to action push button 'view our services' will practise the job. Make certain that the button when clicked actually opens up to the services folio. You practise non desire the user to be served with a 404, which volition exit him/her frustrated equally well.

What are subconscious pages?

Hidden pages are those that are not attainable via a bill of fare or navigation. Though a visitor may be able to view them, peculiarly through anchor text or inbound links, they can be hard to find.

Pages that fall into the category section are likely to be hidden pages besides, as they are located in the admin panel. The search engine may never be able to admission them, as they exercise not access data stored in databases.

Hidden pages can as well result from pages that were never added to the site's sitemap simply exist on the server.

Should all hidden pages be done abroad with?

Not really. There are hidden pages that are admittedly necessary, and should never be accessible from your navigations. Permit's look at examples:

Newsletter sign ups

You can accept a page that breaks down the benefits of signing up to the newsletter, how oftentimes users should expect to receive it, or a graphic showing the newsletter (or previous newsletter). Remember to include the sign upward link as well.

Pages containing user data

Pages that require users to share their information should definitely exist hidden. Users need to create accounts earlier they tin can access them. Newsletter sign ups can besides be categorized here.

How to find hidden pages

Like nosotros mentioned, you can find hidden pages using all the methods that are used to find orphan or expressionless cease pages. Allow's explore a few more.

Using robots.txt

Subconscious pages are highly likely to be hidden from search engines via robots.txt. To access a site's robots.txt, blazon [domain name]/robots.txt into a browser and enter. Replace 'domain name' with your site's domain name. Look out for entries commencement with 'disallow' or 'nofollow'.

Manually finding them

If yous sell products via your website for example, and suspect that one of your production categories may be hidden, you can manually look for information technology. To do this, copy and paste another products URL and edit it accordingly. If you don't find it, then you lot were right!.

What if you have no thought of what the hidden pages could exist? If you organize your website in directories, yous can add your domainname/binder-name to a site'due south browser and navigate through the pages and sub-directories.

Once you have constitute your hidden pages (and they do not demand to stay subconscious equally discussed to a higher place), yous need to add information technology to your sitemap and submit a crawl request.

How to find all the pages on your site

You need to find all your spider web pages in order to know which ones are dead-end or orphan. Let'southward explore the different means to achieve this:

Using your sitemap file

We have already looked at sitemaps. Your sitemap would come in handy when analyzing all of your web pages. If yous do not have a sitemap, you can use a sitemap generator to generate one for y'all. All you need to do is enter your domain name and the sitemap will be generated for you.

Using your CMS

If your site is powered by a content management organization(CMS) like WordPress, and your sitemap does not contain all the links, it is possible to generate the listing of all your web pages from the CMS. To practise this, use a plugin like Export All URLs.

Using a log

A log of all the pages served to visitors also comes in handy. To access the log, log in to your cPanel, then notice 'raw log files'. Alternatively, request your hosting provider to share it. This way you get to meet the most frequently visited pages, the never visited pages and those with the highest drop off rates. Pages with high bounce rates or no visitors could be expressionless-end or orphan pages.

Using Google Analytics

Here are the steps to follow:

Stride 1: Log in to your Analytics folio.

Step 2: Go to 'beliefs' then 'site content'

Stride 3: Go to 'all pages'

Step 4: Scroll to the bottom and on the right choose 'show rows'

Step 5: Select 500 or g depending on how many pages yous would gauge your site to have

Footstep vi: Curl up and on the top correct choose 'consign'

Step 7: Choose 'export as .xlsx' (excel)

Step 8: Once the excel is exported choose 'dataset ane'

Footstep 9: Sort by 'unique page views'.

Footstep 10: Delete all other rows and columns apart from the one with your URLs

Stride 11: Use this formula on the second column:

=CONCATENATE("http://domain.com,A1)

Step 12: Supervene upon the domain with your site's domain. Elevate the formula and so that it is practical to the other cells every bit well.

Y'all now accept all your URLs.

If you desire to catechumen them to hyperlinks in order to easily click and access them when looking something up, continue to stride thirteen.

Footstep xiii: Use this formula on the third row:

=HYPERLINK(B1)

Drag the formula and so that it is applied to the other cells as well.

Manually typing into Google'due south search query

You tin as well blazon this site: www.abc.com into Google's search query. Supersede 'abc' with your domain name. Yous will become search results with all the URLs that Google has crawled and indexed, including images, links to mentions on other sites, and even hashtags your make can be linked to.

You can and so manually re-create each and paste them onto an excel spreadsheet.

What then practise you do with your URL list?

At this signal, you lot may be wondering what you need to do with your URL list. Let'due south look at the available options:

Manual comparison with log data

I of the options would be to manually compare your URL list with the CMS log and identify the pages that seem to have no traffic at all, or that seem to take the highest bounce rates. You can then use a tool like ours to cheque for inbound and outbound links for each of the pages that you suspect to be orphan or dead end.

Some other arroyo is to download all your URLs as a .xlsx file (excel) and your log too. Compare them side by side (in two columns for example) and then use the 'remove duplicates option' in excel. Follow the step by stride instructions. Past the stop of the procedure, you will have simply orphan and expressionless-end pages left.

The tertiary comparison approach is copying ii information sets – your log and URL list on to Google Sheets. This allows yous to use this formula: =VLOOKUP(A1, A: B,ii,) to await up URLs that are present in your URL list, merely not on your log. The missing pages (rendered as Due north/A) should exist interpreted as orphan pages. Ensure that the log data is on the outset or left cavalcade.

Using site itch tools

The other choice would be to load your URL list onto tools that can perform site crawls, wait for them to crawl the site and and so you copy and paste your URLs onto a spreadsheet before analyzing them one past i, and trying to effigy out which ones are orphan or expressionless end.

These ii options tin can be time-consuming, especially if you have many pages on your site, correct?

Well, how most a tool that not only finds you all your URLs but also allows you to filter them and shows their status (so that you lot know which ones are expressionless end or orphan?).  In other words, if you want a shortcut to finding all of your site's pages  SEOptimer's SEO Crawl Tool.

SEOptimer'south SEO Clamber Tool

This tool allows you to admission all your pages of your site. You lot can get-go by going to "Website Crawls" and enter your website url. Striking "Clamber"

In one case the crawl is finished yous tin can click on "View Study":

Our crawl tool will detect all the pages of your website and listing them in the "Folio Found" section of the clamber.

You can identify "404 Mistake" bug on our "Bug Found" just beneath the "Pages Establish" section:

Our crawlers can identify other issues like finding pages with missing Title, Meta Descriptions, etc. One time you find all of your pages, you can start filtering and piece of work on the problems at mitt.

In determination

In this article we take looked at how to find all the pages on your site and why information technology is of import. We have likewise explored concepts similar orphan and expressionless end pages, as well every bit subconscious pages. We have differentiated each i, how to identify each among your URls. There is no better time to find out whether y'all are losing out due to hidden, orphan or dead-end pages.

Source: https://www.seoptimer.com/blog/find-all-pages-on-a-website/

Posted by: farrellsymeave.blogspot.com

Share this post