Posted by Michael Wyszomierski, Search Quality
Last month, several of us with Webmaster Central hit the "good times" jackpot at PubCon Vegas 2007. We realize not all of you could join us, so instead of returning home with fuzzy dice for everyone, we've got souvenir conference notes.
Listening to the Q&A, I was pleased to hear the major search engines agreeing on best practices for many webmaster issues. In fact, the presentations in the duplicate content session were mostly, well, duplicate. When I wasn't sitting in on one of the many valuable sessions, I was chatting with webmasters either at the Google booth, or at Google's "Meet the Engineers" event. It was exciting to hear from so many different webmasters, and to help them with Google-related issues. Here are a few things that were on the minds of webmasters, along with our responses:
Site Verification Files and Meta Tags
Several webmasters asked, "Is it necessary to keep the verification meta tag or HTML file in place to remain a verified owner in Webmaster Tools?" The answer is yes, you should keep your verification file or meta tag live to maintain your status as a verified owner. These verification codes are used to control who has access to the owner-specific tools for your site in Webmaster Tools. To ensure that only current owners of a site are verified, we periodically re-check to see if the verification code is in place, and if it is not, you will get unverified for that site. While we're on the topic:
Site Verification Best Practices
If you have multiple people working on your site with Webmaster Tools, it's a good idea to have each person verify the site with his or her own account, rather than using a shared login. That way, as people come and go, you can control the access appropriately by adding or removing verification files or meta tags for each account.
You may want to keep a list of these verification codes and which owner they are connected to, so you can easily control access later. If you lose track, you can always use the "Manage site verification" option in Webmaster Tools, which allows you to force all site owners to reverify their accounts.
Subdomains vs. Subdirectories
What's the difference between using subdomains and subdirectories? When it comes to Google, there aren't major differences between the two, so when you're making that decision, do what works for you and your visitors. Following PubCon, our very own Matt Cutts outlined many of the key issues in a post on his personal blog. In addition to those considerations, if you use Webmaster Tools (which we hope you do!), keep in mind that you'll automatically be verified for deeper subdirectories of any sites you've verified, but subdomains need to be verified separately.
Underscores vs. Dashes
Webmasters asked about the difference between how Google interprets underscores and dashes in URLs. In general, we break words on punctuation, so if you use punctuation as separators, you're providing Google a useful signal for parsing your URLs. Currently, dashes in URLs are consistently treated as separators while underscores are not. Keep in mind our technology is constantly improving, so this distinction between underscores and dashes may decrease over time. Even without punctuation, there's a good chance we'll be able to figure out that bigleopard.html is about a "big leopard" and not a "bigle opard." While using separators is a good practice, it's likely unnecessary to place a high priority on changing your existing URLs just to convert underscores to dashes.
Keywords in URLs
We were also asked if it is useful to have relevant keywords in URLs. It's always a good idea to be descriptive across your site, with titles, ALT attributes, and yes, even URLs, as they can be useful signals for users and search engines. This can be especially true with image files, which otherwise may not have any text for a search engine to consider. Imagine you've taken a picture of your cat asleep on the sofa. Your digital camera will likely name it something like IMG_2937.jpg. Not exactly the most descriptive name. So unless your cat really looks like an IMG_2937, consider changing the filename to something more relevant, like adorable-kitten.jpg. And, if you have a post about your favorite cat names, it's much easier to guess that a URL ending in my-favorite-cat-names would be the relevant page, rather than a URL ending in postid=8652. For more information regarding issues with how Google understands your content, check out our new content analysis feature in Webmaster Tools, as well as our post on the URL suggestions feature of the new Google Toolbar.
Moving to a new IP address
We got a question about changing a site's IP address, and provided a few steps you can take as a webmaster to make sure things go smoothly. Here's what you can do:
Change the TTL (Time To Live) value of your DNS configuration to something short, like five minutes (300 seconds). This will tell web browsers to re-check the IP address for your site every five minutes.
Copy your content to the new hosting environment, and make sure it is live on the new IP address.
Change your DNS settings so your hostname points to the new IP address.
Check your logs to see when Googlebot starts crawling your site on the new IP address. To make sure it's really Googlebot who's visiting, you can verify Googlebot by following these instructions. You can then log into Webmaster Tools and monitor any crawl errors. Once Googlebot is happily crawling on the new IP address, you should be all set as far as Google is concerned.
To make sure everyone got the message of your move, you may want to keep an eye out for visits to your old IP address before shutting it down.
Proxies
A few webmasters were concerned that proxy services are being indexed with copies of their content. While it's often possible to find duplicate copies of your content in our results if you look hard enough, the original source is most likely going to be ranked higher than a proxy copy. However, if you find this not to be the case, please drop us some URLs in the Webmaster Help Group. There are many Googlers including myself who monitor this group and escalate issues appropriately.
It was great talking with webmasters at the conference -- we hope those of you unable to join us found this post useful. If you want to continue to talk shop with me, other Googlers, and your fellow webmasters, join the follow-up conversation in the Webmaster Help Group.
Update: Additional PubCon notes from Jonathan Simon are available in our discussion group.
Posted by Michael Wyszomierski
Permalink 12 comments Links to this post
Labels: events
Sitemaps FAQs
Tuesday, January 15, 2008 at 10:35 AM
Written by Susan Moskwa & Trevor Foucher, Webmaster Tools Team
Last month, Trevor spoke on the Sitemaps: Oversold, Misused or On The Money? panel at Search Engine Strategies in Chicago. After receiving a lot of great questions at the conference in addition to all the feedback we receive in our Help Group, we've pulled together a FAQ:
Q: I submitted a Sitemap, but my URLs haven't been [crawled/indexed] yet. Isn't that what a Sitemap is for?
A: Submitting a Sitemap helps you make sure Google knows about the URLs on your site. It can be especially helpful if your content is not easily discoverable by our crawler (such as pages accessible only through a form). It is not, however, a guarantee that those URLs will be crawled or indexed. We use information from Sitemaps to augment our usual crawl and discovery processes. Learn more.
Q: If it doesn't get me automatically crawled and indexed, what does a Sitemap do?
A: Sitemaps give information to Google to help us better understand your site. This can include making sure we know about all your URLs, how often and when they're updated, and what their relative importance is. Also, if you submit your Sitemap via Webmaster Tools, we'll show you stats such as how many of your Sitemap's URLs are indexed. Learn more.
Q: Will a Sitemap help me rank better?
A: A Sitemap does not affect the actual ranking of your pages. However, if it helps get more of your site crawled (by notifying us of URLs we didn't previously didn't know about, and/or by helping us prioritize the URLs on your site), that can lead to increased presence and visibility of your site in our index. Learn more.
Q: If I set all of my pages to have priority 1.0, will that make them rank higher (or get crawled faster) than someone else's pages that have priority 0.8?
A: No. As stated in our Help Center, "priority only indicates the importance of a particular URL relative to other URLs on your site, and doesn't impact the ranking of your pages in search results." Indicating that all of your pages have the same priority is the same as not providing any priority information at all.
Q: Is there any point in submitting a Sitemap if all the metadata (
, , etc.) is the same for each URL, or if I'm not sure it's accurate?
A: If the value of a particular tag is the same for 100% of the URLs in your Sitemap, you don't need to include that tag in your Sitemap. Including it won't hurt you, but it's essentially the same as not submitting any information, since it doesn't help distinguish between your URLs. If you're not sure whether your metadata is accurate (for example, you don't know when a particular URL was last modified), it's better to omit that tag for that particular URL than to just make up a value which may be inaccurate.
Q: I've heard about people who submitted a Sitemap and got penalized shortly afterward. Can a Sitemap hurt you?
A: Only if it falls on you from a great height. (Seriously, though: if it ever happened that someone was penalized after submitting a Sitemap, it would have been purely coincidental. Google does not penalize you for submitting a Sitemap.)
Q: Where can I put my Sitemap? Does it have to be at the root of my site?
A: We recently enabled Sitemap cross-submissions, which means that you can put your Sitemap just about anywhere as long as you have the following sites verified in your Webmaster Tools account:
the site on which the Sitemap is located
the site(s) whose URLs are referenced in the Sitemap
Note that cross-submissions may not work for search engines other than Google. Learn more about Sitemap cross-submissions.
Q: Can I just submit the site map that my webmaster made of my site? I don't get this whole XML thing.
A: There's a difference between a (usually HTML) site map built to help humans navigate around your site, and an XML Sitemap built for search engines. Both of them are useful, and it's great to have both. A site map on your domain can also help search engines find your content (since crawlers can follow the links on the page). However, if you submit an HTML site map in place of a Sitemap, Webmaster Tools will report an error because an HTML page isn't one of our recognized Sitemap formats. Also, if you create an XML Sitemap, you'll be able to give us more information than you can with an HTML site map (which is just a collection of links). Learn more about supported Sitemap formats.
Q: Which Sitemap format is the best?
A: We recommend the XML Sitemap protocol as defined by sitemaps.org. XML Sitemaps have the advantage of being upgradeable: you can start simple if you want (by just listing your URLs), but—unlike a text file Sitemap—you can easily upgrade an XML Sitemap later on to include more metadata. XML Sitemaps are also more comprehensive than an Atom or RSS feed submitted as a Sitemap, since feeds usually only list your most recent URLs (rather than all the URLs you want search engines to know about).
Q: If I have multiple URLs that point to the same content, can I use my Sitemap to indicate my preferred URL for that content?
A: Yes. While we can't guarantee that our algorithms will display that particular URL in search results, it's still helpful for you to indicate your preference by including that URL in your Sitemap. We take this into consideration, along with other signals, when deciding which URL to display in search results. Learn more about duplicate content.
Q: Does the placement of a URL within a Sitemap file matter? Will the URLs at the beginning of the file get better treatment than the URLs near the end?
A: No, and no.
Q: If my site has multiple sections (e.g. a blog, a forum, and a photo gallery), should I submit one Sitemap for the site, or multiple Sitemaps (one for each section)?
A: You may submit as few or as many Sitemaps as you like (up to these limits). Organize them in whatever way you find easiest to maintain. If you create multiple Sitemaps, you can use a Sitemap Index file to list them all. Learn more.
If your question isn't covered here, you can find even more questions and answers in our Sitemaps Help Group.
Posted by Susan Moskwa
Permalink 14 comments Links to this post
Labels: sitemaps
Remove your content from Google
Wednesday, January 09, 2008 at 11:39 AM
Posted by Ríona MacNamara, Webmaster Tools Team
Confused about the best uses of robots.txt, nofollow, URL removal tool? Wondering how to keep some of your pages off the web? Our webspam lead, Matt Cutts, talks about the best ways to stop Google from crawling your content, and how to remove content from the Google index once we've crawled it.
We love your feedback. Tell us what you think about this video in our Webmaster Help Group.
* Note from Matt: Yes, robots.txt has been around since at least 1996, not 2006. It's hard for me to talk for 12-13 minutes without any miscues. :)
Posted by Ríona MacNamara
Permalink 26 comments Links to this post
Labels: crawling and indexing
Growing the Webmaster Help Groups Team
Monday, January 07, 2008 at 10:39 AM
Written by Kas, Search Quality Team
One of the ways in which we continuously work to improve our service for users everywhere is to localize and adjust our tools for different regions. In November, we had a chance to introduce our international Webmaster Help Group Team and its guides from Dublin. Today I'm very happy to announce that we've launched three more Google monitored groups in Hebrew, Turkish and Hungarian.
Keeping with tradition, the new guides want to introduce themselves, so please meet our new Webmaster Help Group guides :-)
Turkish Webmaster Help Group
Merhaba! I am Bahar and I am very happy about our brand new Turkish Webmaster Help Group. I will be helping out on any topics related to this new group. Having an engineering background, I've worked in different parts of the IT field as a C and C++ developer and now I work for Google in Dublin. It was quite a big change for me to move here, leaving sunny Istanbul behind. But it was very easy to get used to this new life since I have met really friendly people here who are very enthusiastic about the job they have been doing. You're going to meet a couple of them as well when you keep reading further. Please do not hesitate to have a look at the group and drop your posts either to introduce yourselves or to discuss Webmaster related issues with me and hopefully soon many other helpful, open-minded members!
- Bahar
Hebrew Webmaster Help Group
I’m Alon and I’ll be looking after the Hebrew Webmaster Help Group. Originally coming from the wonderful Israeli city of Holon, now I spend my days and nights in the green island called Ireland. So let me tell you a bit about myself by ranking keywords and their importance in my life: family, social activism, work, web, friends, Tel-Aviv, coffee and rollerblading, to mention a few. Prior to joining Google I graduated in economics and worked as a freelancer in the IT field. I love the online sphere, and for a couple of years now I've been volunteering for an organization facilitating international webmaster communication. I’m looking forward to seeing many of you in our Hebrew Webmaster Help Group.
- Alon
Hungarian Webmaster Help Group
Sziasztok, my name is Tibor, and I am very excited about the launch of the Hungarian Webmaster Help Group. I grew up in the heart of Hungary, in the prosperous and pleasant city of Székesfehérvár, right between Lake Balaton and Budapest. I have an engineering and management background, seasoned with marketing and design experience, and have worked in various assignments, including dozens of web related projects (the first in '94). Today I am a happy owner of a small blog, but have some ambitious plans to start some bigger projects too, to get first hand experience of the constantly growing arsenal of Google's Webmaster Tools! I work primarily on search, dealing with Turkish and Lithuanian languages besides Hungarian. I also oversee trends in a dozen other languages scattered across Europe, the Middle East and Asia. Just like my colleague János, I also enjoy old-school times with StarCraft, and my "weaknesses" also include Guitar Hero and the Nintendo Wii. As Webmaster Tools is now available in Hungarian, I look forward to hearing from you in the Hungarian Webmaster Help Group and to engage myself in vibrant discussions :-)
- Tibor
Hungarian Webmaster Help Group
Sziasztok! My name is János and I'm in charge of the Hungarian Webmaster Help Group, too. I studied linguistics and worked in the IT industry before I ended up with Google in Ireland. Currently my primary work focus is search in several languages. In the past I've been lucky to support a wide range of different markets including Russian, Turkish, Polish and many more. Well, I like learning languages, as you might guess. I'm running my own website where I develop content, sometimes I need to troubleshoot it as well - no crawling/indexing issues yet though - I'm a happy Webmaster Tools user ;-) When I don't work, I'm usually at home surfing the Internet or leading my zealous Protoss forces against other races in the StarCraft universe. Please drop by the Hungarian Help Group and introduce yourself, too! My guide colleague Tibor and I are eagerly waiting to help you with webmaster issues.
- János
Posted by Maile Ohye
Permalink 4 comments Links to this post
Labels: feedback and communication
Looking back on 2007
Thursday, January 03, 2008 at 1:31 PM
Written by Liza Ma, Webmaster Tools Team
2007 has been a great year of change and growth for the Webmaster Central team. Without the webmaster community helping us drive our vision to create better tools we wouldn't have accomplished much. All of us with the Webmaster Central team would like to say a big thank you to everyone! Before we look ahead to 2008 and begin to incorporate your feedback from the blog posts and discussion groups, we would like to take a minute to reflect upon some of our bigger feats of the last twelve months.
Our blog has been very active since its launch in August 2006, and we hope you've found it helpful. For those who are new readers, you can find useful information about our Webmaster Tools updates here, as well as articles on how to make your sites more search engine-friendly and user-friendly. Outside of the US, we have readers from the UK, France, Canada, India and Germany, just to name a few—it just goes to show what a big and widespread community you are! In fact, we also have a dedicated group of readers for our Webmaster Central German and China blogs. For those keeping track, 47.28% of you are using Firefox to read our blog and 45.06% are using Internet Explorer, and our most popular post this past December answered your questions on meta tags and web search. Here's a snapshot of what our top search queries look like for December:
January
Made preparations for our big debut (see February)
February
Moved Webmaster Tools officially out of beta
Traveled the globe and met with webmasters like you at conferences in Australia and London
Launched the external and internal links feature in Webmaster Tools to make it easier for you to learn about your links
March
Extended anchor text information to show the entire phrase sites use to link to you (not just individual words)
Posted some helpful articles about robots.txt
Kicked off a new blog for German webmasters
April
Met with more of you at SES New York
Added URL removal to Webmaster Tools
May
Shared some useful tips on how webmasters can make use of Google's newly unveiled universal search
Met with webmasters like you at conferences in China, Italy, New York and San Jose
June
Revamped the Help Center to make it easier for everyone to find the content they need
Reached out to webmasters at conferences in Seattle, Santa Clara and Toronto
July
Unveiled the Message Center, a key component in helping us keep in touch with you!
Improved Sitemap feedback to help you identify important issues in your Sitemaps
Launched a Webmaster Central blog in China
August
Enabled non-English domain names in Webmaster Tools
Chatted with you at SES San Jose
Updated Webmaster Tool's feature to help you analyze your robots.txt files
September
Updated Webmaster Tools:
Introduced Subscriber stats
Revamped the UI
Added support for 2 new languages: Romanian & Turkish
October
Went all out for Halloween in true Google fashion
Launched new features in Webmaster Tools—and a lot of them:
Sitemap updates: cross-submissions and Code Search Sitemaps
Top search queries—for the last 6 months!
Sitelinks feature to help you identify the ones you have and help you get rid of the ones you don't like
Set your geographic target to help your users find you more easily in country-specific searches
Improved data freshness to keep you better informed about your site's performance
November
Highlighted terrific webmasters in the discussion group and introduced Googlers ready to answer your questions in 11 non-English languages.
Made it easier for Go Daddy users to access our tools
Produced some helpful videos featuring our very own Matt Cutts on the anatomy of a search result and how to use ALT attributes intelligently
December
Another Webmaster Tools update loaded with new features:
Improved Sitemap summary pages to help you identify issues in your Sitemaps
Content analysis
Video Sitemaps
Added support for 2 new languages: Czech and Hungarian
Attended PubCon in Las Vegas and SES Chicago
It was a busy year, but we had a lot of fun. I hope you'll stick with us as we embark on more great adventures in 2008. Remember, you can always reach out to us through our discussion group or at various search conferences throughout the year and around the globe!
Posted by Liza Ma
Permalink 10 comments Links to this post
Labels: feedback and communication, webmaster tools
A Festivus for our webmasterus
Friday, December 21, 2007 at 2:00 PM
Written by Maile Ohye
If it's good enough for the Costanzas, it's good enough for Webmaster Central: it's time for a Festivus for the rest of us (webmasterus)!
Our special celebration begins not with carols and eggnog, but by remembering some of the popular Webmaster Tools features -- make that Feats of Strength -- for 2007. This year, you gained the ability to chickity-check out your backlinks (<-- that's Festivus-inspired anchor text) and tell Google you want out with URL Removal. And let's not forget Message Center and IDNA support, perfect for those times when [a-zA-Z0-9\-] just doesn't cut it.
Feel the power! Festivus Feats of Strength!
Now comes our webmaster family's traditional Airing of Grievances. You can air your woes and "awww man!"s in the comments below. Just remember that bots may crawl this blog, but we humans review the comments, so please keep your grievances constructive. :) Let us know about features you'd like implemented in Webmaster Tools, articles you'd like written in our blog or Help Center, and stuff you'd like to see in the discussion group. Bonus points if you also explain how your suggestion helps the whole Internet—not just your site's individual rankings. (But of course, we understand that your site ranking number one for all queries in all regions is truly, objectively good for everyone.)
Last, there are so many Festivus Miracles to share! Such as the many helpful members of the discussion group from all around the world, the new friendships formed between Susan Moskwa, JohnMu, Wysz, Matt D, Bergy, Patrick, Nathanj and so many webmasters, and the fun of chatting with our video watchers, fellow conference attendees, and those in the blogosphere keepin' it real.
On behalf of the entire Webmaster Central team, here's to you, Festivus Miracle and Time Magazine's Person of the Year in 2006 -- happy holidays. See you in 2008. :)
Posted by Maile Ohye
Permalink 43 comments Links to this post
The Ultimate Fate of Supplemental Results
Tuesday, December 18, 2007 at 8:10 PM
Posted by Yonatan Zunger, Search Quantity Team
In 2003, Google introduced a "supplemental index" as a way of showing more documents to users. Most webmasters will probably snicker about that statement, since supplemental docs were famous for refreshing less often and showing up in search results less often. But the supplemental index served an important purpose: it stored unusual documents that we would search in more depth for harder or more esoteric queries. For a long time, the alternative was to simply not show those documents at all, but this was always unsatisfying—ideally, we would search all of the documents all of the time, to give users the experience they expect.
This led to a major effort to rethink the entire supplemental index. We improved the crawl frequency and decoupled it from which index a document was stored in, and once these "supplementalization effects" were gone, the "supplemental result" tag itself—which only served to suggest that otherwise good documents were somehow suspect—was eliminated a few months ago. Now we're coming to the next major milestone in the elimination of the artificial difference between indices: rather than searching some part of our index in more depth for obscure queries, we're now searching the whole index for every query.
From a user perspective, this means that you'll be seeing more relevant documents and a much deeper slice of the web, especially for non-English queries. For webmasters, this means that good-quality pages that were less visible in our index are more likely to come up for queries.
Hidden behind this are some truly amazing technical feats; serving this much larger of an index doesn't happen easily, and it took several fundamental innovations to make it possible. At this point it's safe to say that the Google search engine works like nothing else in the world. If you want to know how it actually works, you'll have to come join Google Engineering; as usual, it's all triple-hush-hush secrets.*
--------------------------------------------------------------------------------
* Originally, I was going to give the stock Google answer, "If I told you, I'd have to kill you." However, I've been informed by management that killing people violates our "Don't be evil" policy, so I'm forced to replace that with sounding mysterious and suggesting that good engineers come and join us. Which I'm dead serious about; if you've got the technical chops and want to work on some of the most complex and advanced large-scale software infrastructure in the world, we want you here.
