Technical SEO SEO All

Website Not Ranking On Google: Fix Crawling And Indexing Before You Chase Ranking

Shanu Quraishi

13 min read | 15/08/2026

Our minds want the result before the work. Every business owner understands this about money, and almost nobody applies it to Google.

One thing every business needs to understand over time is this: what you put in today is what you will be able to take out tomorrow.

For example, if you have saved ₹10,000 in your bank account today, then tomorrow, whenever the need arises, you can withdraw that ₹10,000. There is no scenario in which you have saved nothing, put nothing into your account, and still expect to suddenly withdraw money from it.

Now bring that same logic to search. When a business says it wants to rank its website on Google, or wants to show up on Google — and showing up eventually means ranking — the question that should come back is simple: what did you deposit?

Why Ranking On Google Is The Last Step, Not The First

If we talk about the right tendency, the right practice for getting ranked, then ranking is a very late concern. Ranking is one of the second-last steps, or arguably the last step.

Before it, a lot of things need to happen. If those things are not happening, then eventually you are missing something, and you will never be able to rank at all.

It is exactly like the example above. If you have deposited ₹10,000, you can withdraw it. If you have not deposited anything, you will not be able to withdraw anything — forget about it.

So the real question becomes: what is going wrong? What mistake is being made? Why is our website not visible on Google, and what can we actually do to make it visible?

The first thing to focus on and understand is the process — what is the method, what is the path through which a website ranks, and where does that path begin? What are the steps that lead to ranking?

Coming in at the very start and expecting that ranking will simply arrive, or staying stuck on the single demand of "I need ranking anyhow" — this is the biggest mistake, and honestly the most foolish one too.

If you want the overview of the full journey from crawling to ranking first, read how a search engine works. This article goes deeper into the part where most websites actually get stuck.

Two Different Problems: Ranking On Page 10 vs Not Being Indexed At All

It is not necessary that your website ranks on the first page. Sometimes it is ranking on the 5th page, the 7th page, the 10th page. In that scenario the strategies and the methods will be different.

But suppose the website is not visible at all — not anywhere, not on page 10, 15 or 20, nowhere on Google. In that particular scenario, the use cases and the strategy will be completely different.

Both are real problems and both need to be discussed. But first, let us understand the indexing issue — the case where the website does not show up anywhere at all. What are the things involved there? What should be the strategy? What should you do, what should you not do, and what actually goes wrong? That deserves attention first, because if this layer is broken, nothing above it can work.

How Google Finds Your Website: URL Server, Crawling And Indexing

What Is Google's URL Server?

To explain it from the absolute basics — Google has something called a URL server. The URL server is nothing but a kind of database of the URLs available across the world that Google needs to crawl.

Think of it this way. Suppose there is an officer who goes to his office. If he finds 10, 20, 15 — even 100 files placed on his table, with the instruction that these are the files he has to read today, then he can comfortably read them and check them.

He may finish all of them by evening. If he does not finish by evening, he will keep working on them the next day, continuously, and at some point he will finish them.

But if no files are arriving on his table at all, then the officer has no idea which files he is supposed to look at, which ones he is supposed to read, and what action he needs to take or not take.

Your URL is that file. If it never reaches the table, no one is ignoring you — no one even knows you exist.

What Is Crawling In SEO And What Does Google Crawler Do?

In the same way, Google has a bot whose name is Google Crawler.

The job of the Google Crawler is to go to every single URL of every website and scan them — to crawl them, basically. In Google's language, this technique, this facility, is called crawling.

Why Google Is Not Crawling Your Website

If it has ever happened that crawling has not taken place on your website, then the first thing you should focus on is exactly this: why is crawling not happening? What are the causes behind crawling not happening?

Very often, the website has just been built today. It is a brand new website, so it will take some time. That is fine — it will crawl it eventually.

But if it is genuinely not crawling, then there is a tool inside Google Search Console that you can use. Crawling often happens from there.

Along with that, what you can also do is build backlinks. And sometimes Google itself picks up your website's URL for crawling on its own.

But whichever way the URLs are picked up, at the end of the day they all reach the same place — the URL server. That is where all the URLs the crawler has to crawl are held, exactly like the officer being told which files to work on today.

So the crawler has an exact list, a proper list in the database, of however many URLs it has to scan today — 10 thousand, 20 thousand, 50 thousand, 1 lakh. It goes through them and crawls them.

So what you have to make sure is this: your URL should reach the URL server. Because once it reaches there, the crawler will crawl it.

What Is Indexing In SEO And What Google Checks First

After crawling, what does Google do? If it feels that there is no malware on your website, there is no problem, you are not doing phishing, you are a genuine business, a genuine person, a genuine content creator, and you are doing everything genuinely —

— then it saves a repository, a copy, a document of your business, your website and your things in its own database.

This particular thing is called indexing.

When these two things have happened — indexing being done and crawling being done — that is when you can expect that the ranking process will now begin.

Google Search Console: The Doctor For Your Website

Very often what happens is that we do know our website is not getting indexed. Whether it has been crawled or not, we do not have a very exact idea about that either.

In this, the tool I consider the most important is Google Search Console. It is a very lovely tool, a very good tool — and not only because it will tell you the indexing status of your website, or because it will get your website, that particular URL, indexed.

Google Search Console is much more than that. It is a next-level tool, and it is free. Such a lovely tool, so important, so insightful, being available for free is the genuinely surprising part for me.

It has a lot of use cases. I consider Google Search Console to be the doctor of a website. So let us understand this doctor specifically, and try to understand how it helps in indexing.

How To Check The Real Crawl Status Of A URL

Suppose you have a URL — let us say xyz.com/testing. Now you search for it, and you see that this URL of yours does not rank inside Google. So somewhere you identify that there is an indexing problem.

We have identified the indexing problem — that it is not getting indexed. But going two steps further than that, there is another thing to identify: the matter is not only about indexing.

We will have to look at the actual crawl status. And if it has not been crawled till today, or has not been crawled recently, then we have to confirm where exactly the visibility of your URL is being decided — where it is being confirmed that your URL is not getting indexed.

This matters for ranking as well, because ranking has to happen off the back of it.

How To Submit A URL For Indexing In Google Search Console

There you get a button through which you can submit that URL to the crawler for indexing or for crawling.

When you submit it, it takes approximately one minute for a single URL submission. But after that you get a message telling you that your URL has been added to our URL list — which indirectly is the crawl URL server.

Why Submitting The Same URL Again And Again Does Not Work

Now, you should not submit multiple times. Google itself says this. Because submitting multiple times is not going to change anything.

If there are 10 thousand URLs, your URL gets a number — as in, so many URLs will be looked at, and then this one will be looked at.

Priority is already decided inside Google for every website, according to its crawl budget. What crawl budget is, and how it decides this priority, is a topic that deserves its own explanation — we will cover it in detail separately.

So what you need to understand properly is this: Google knows your URL has been put into a crawl queue, and a number has been assigned to it.

Now if you keep submitting it again and again, the number it holds in the queue is not going to change.

This is what Google says in that very message — that submitting it multiple times will not change its priority and its position in the queue.

So once you have submitted it, after that, be patient. It takes a little time. Two to four days — and sometimes the URL gets indexed within 24 hours as well.

It will get indexed, and then you will be able to do the further work on top of it that moves you towards ranking.

Common Technical Reasons Your URL Is Still Not Indexed

What usually happens, from what I have seen in my own experience, is that sometimes the mistake is not in crawling or indexing at all.

Sometimes the mistake is in the robots.txt file. Sometimes the mistake is in the sitemap.xml file. And sometimes it is a noindex tag that someone applied and forgot about.

Robots.txt Disallow Is Blocking The Crawler

What we sometimes do is keep that particular URL path, or that particular area or section of the website, disallowed inside the robots.txt file.

So the crawler is simply not able to go there. And if it cannot go there, it will not be able to index, it will not be able to crawl.

This is the important part that people miss: submitting the URL in Google Search Console does not bypass robots.txt. The sitemap problem gets bypassed by a direct submission — but if you still have that particular area, section or URL disallowed inside robots.txt, then the crawler will come, read the rule in robots.txt, and leave without crawling at all.

And sometimes the problem is not with a whole section of the website. Sometimes the problem is only this much — that in robots.txt there is a disallow slash. That one slash will go and kill the indexing of the entire website. You will feel there is no problem at all, but in reality there very much is.

So that thing has to be fixed. You have to look at whether it is applied when it should not be, or removed when it should be there. You need to make sure of it.

What robots.txt is, why it is responsible, and what exactly it does — along with what sitemap.xml is — deserves a detailed explanation of its own, and it will get one in a separate article.

Your URL Is Missing From The Sitemap.xml

What you need to understand here is that sitemap.xml at least does this much work: whenever the crawler arrives, the sitemap gives the crawler a list of URLs of that website. It says — if you have come to crawl, here is the website's URL, this one was published at this time, this is its priority.

If you want, crawl this one too. And if you feel there is something worthwhile in it, index it and keep it with you.

Now here is another thing I have seen. Sometimes the URL you are trying to get indexed is not in the sitemap at all.

If you had properly put the URL inside the XML sitemap, then to a large extent it was possible — there were fair chances — that when Google's crawler came and scanned your sitemap, it would have found your URL right there, crawled it on its own, and indexed it.

But somewhere the mistake stayed on your side, or it slipped your mind, and so you were not able to do that particular thing — because of which your URL could not get indexed or could not get crawled.

So when you submit in Google Search Console, you are putting in a direct request — that now this should get indexed.

A Noindex Tag Is Sitting On The Page

Then there is another mistake. A mistake happens, or sometimes people apply something and forget about it.

The noindex tag on every page of every website works at the individual level.

Suppose there are 10 URLs on my website. There I can apply the index tag on 5 URLs, on 7 URLs. And I can apply the noindex tag on two or three URLs.

That noindex tag I am applying is specifically telling Google: come here, do the crawling, but do not index it.

Now if you are making this kind of mistake yourself, and then you expect that Google is interpreting your things in a wrong way somewhere, or that it is not letting your website rank or index — then again, you are somehow making a big mistake. You need to work on that.

The WordPress "Discourage Search Engines" Setting

Sometimes when you build a website in WordPress, right at the beginning there is an option by default: "discourage search engines from indexing this site." There are also plugins involved in this.

Because of that, the noindex tag gets applied on every page — which does not let the website get indexed at all.

This is a very important concern that every person should check once. Before launching the website, or before doing anything further with the website: is there a problem in the tagging — in the labels of indexing, in the metadata?

Fix These URL Errors Before You Chase Indexing And Crawling

Then there is one thing I have noticed a great deal. That link used to exist earlier, and now it does not exist — meaning it is broken. How are you expecting ranking on that?

Or you have moved it now. Earlier the URL was one thing, now it is another. Fine — then it will rank after that change.

Its canonical is not set correctly. It opens without the www, sometimes it opens with the www, sometimes it opens on http.

This URL-level error is what you have to fix first. The redirect loops you have got stuck — you have to fix those.

Then go there and take up indexing, crawling and all these things. Otherwise your website will keep suffering forever.

Google Penalty: Manual Actions And Algorithmic Penalties

Then sometimes another thing happens. You have written the content so badly — you have completed all the technical aspects, but the content is written so badly — or you have essentially played around with Google's policy in some specific way.

At that point Google will come manually and give you a penalty, or its algorithms will give you a penalty.

If you have received a manual penalty, or an automatic penalty, then in Security there is a manual actions section, and you will be given it there.

But if a penalty is coming your way, then you need to work very carefully. As in — why are you even doing the kind of work that gets you a penalty?

You need to keep all these things in mind. You cannot work this way. Reaching the point where it goes as far as a penalty is a very bad signal for any digital marketing professional.

Why did you come into the online space in the first place? You came into the market in order to promote your business, to show the worth of your business.

Now what are you doing? You are spamming — that is why you got the penalty. Otherwise Google is not foolish enough to hand out a penalty to just anyone.

So you need to tackle these things with a lot of maturity.

Conclusion

Ranking is a withdrawal. Crawling, indexing, a clean robots.txt, a correct sitemap.xml, honest noindex tags and fixed URL errors are the deposit.

If your website is not showing on Google anywhere, do not start by asking why you are not ranking. Start by asking which step got skipped — because that question has an answer, and the first one does not.

Check the crawl status. Check whether the URL reached the URL server. Check robots.txt for a disallow. Check whether the URL exists in the sitemap. Check the noindex tag and the WordPress setting. Fix the broken links, the canonical, the www and http versions, and the redirect loops.

Put something in. You will be able to take it out.