Google Web Master Tools reports "Soft 404"
Hi all,
WMT reports many of our Vanilla forum URLs as "Soft 404s".
So far it was has reported about 1,000 URLs as soft 404s. These are a combination of category, discussion and profile pages.
All of these pages exists and returns a 200, so we are not sure why Google classes them as soft 404s.
Could it be something to do with the fact that you can change the URL and it still resolves?
For example, Vanilla returns a 200 even if you change the URL
/community/discussion/1873/i-can-type-anything-here
/community/discussion/1873/is-it-possible
Here is the HTTP header:
Status: 200 OK
Code: 200
Date: Mon, 08 Feb 2016 02:04:57 GMT
Server: Apache
P3P: CP="CAO PSA OUR"
X-Garden-Version: Vanilla 2.2.100.4
X-Frame-Options: SAMEORIGIN
Set-Cookie: Vanilla=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=www.mydomain.com
Vary: Accept-Encoding,User-Agent
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Expires: Mon, 08 Feb 2016 03:04:57 GMT
Connection: close
Content-Type: text/html; charset=utf-8
Comments
I guess Vanilla doesn't add a forward like a 301 if you change the title of a discussion.
Another issue that we have with thousends of links is that our users use mentions quiet often. But they are good in writing usernames wrong. So a regular link to the Admin profile that would be @Admin produces of course a 404 when the user just writes @Admi.
We fix that manually in the Database. What a pain. 6-7000 404s still remain always on our board.
If someone has a smart idea...
Discussions name change does do a redirect. I change titles a lot on my other boards I run for SEO and have not seen any issues.
As for profiles, in my boards I make the robots.txt ignore the profiles to avoid that issue of mistaken mentions.
Have you tried Google's troubleshooting tips?
https://googlewebmastercentral.blogspot.ca/2010/06/crawl-errors-now-reports-soft-404s.html?m=1
As an aside, I just looked and I have only one soft 404. When I fetched and rendered this issue seems to be gone.
Maybe it's a case of Google index and current forum pages not being in sync?
In any case if the concern is impact in search engine visibility or issues, I have not seen any...(my two cents)
Hi Adrian,
I need to hook into regarding not indexing the user profile pages. Have you and the Vanilla team run any tests on larger boards about which way performs better SEO?
Thank you for help,
Stefan
I can't speak to customer data as we would need access to their organic traffic, but we do know Vanilla does perform well with SEO with the current robots.txt we add. I personally would disallow the profile folder altogether, its worked well for me.
Thanks for the replies guys. We are still getting new soft 404s every day. Here is a screenshot (I deleted most and left just 7):
A soft 404 is a page that does not exist, but returns a 200. As mentioned, all of the "reported soft 404s" do exist. I can fetch and render.
We have already disallowed the following (it's been in place for over month):
Disallow: /community/profile/
Disallow: /community/discussion/comment/
Disallow: /community/entry/
Disallow: /community/search/
But the categories and discussions still keep coming up in WMT as soft 404s.
I dunno, it sounds like Google screwing up to me.
I've seen a handful of these on my sites but ignored them because the same thing applied: they are there and working correctly.
If there's no real issue with the forum and no way to ask anyone at Google, I dunno what else you can do.