Short answer: Google recommends percent-encoding non-ASCII characters in the links on your site, so /gemüse should appear in your HTML as /gem%C3%BCse. Encoding itself does not hurt indexing. Problems start when the same page is reachable under several different strings, such as double-encoded paths or mixed hex case, and each one returns a normal 200 page. Pick one encoded form, redirect the others to it, put it in your rel="canonical" and list exactly that form in your XML sitemap.
What URL Encoding Does
Percent-encoding (RFC 3986) replaces a byte with % followed by two hex digits. Non-ASCII characters are first converted to UTF-8 bytes, so ü becomes two bytes and two escapes: %C3%BC. A space becomes %20. Letters, digits and - . _ ~ never need encoding. Reserved characters such as / ? # & = must be encoded only when they are part of the data and not part of the URL structure.
Google's URL structure guidelines say to use percent-encoding in href attributes as necessary, and that non-ASCII characters should be percent-encoded. The same page notes that Google treats URLs as case sensitive, so /APPLE and /apple are different URLs.
Encoded vs Decoded: Variants That Cause Duplicates
| Variant | Example | What happens |
|---|---|---|
| Raw Unicode in a link | /gemüse | Browsers encode it before sending, so it usually reaches the same page. Google recommends the encoded form in your HTML. |
| Correct encoding | /gem%C3%BCse | The form to use in links, canonical tags and sitemaps. |
| Lower-case hex | /gem%c3%bcse | RFC 3986 treats hex case as equivalent and recommends upper case. Some servers and CMSs still treat it as a different string, so link with one form only. |
| Double encoding | /gem%25C3%25BCse | The % itself was encoded again. This is a different URL that often returns a 404 or, worse, a duplicate 200 page. |
| Plus instead of space | /red+shoes vs /red%20shoes | In a path, + is a literal plus sign, not a space. Only form-style query strings treat + as a space. |
When two strings return the same content, Google groups them as duplicates and picks one canonical. In its guide on consolidating duplicate URLs, Google calls redirects and rel="canonical" strong signals and sitemaps a weak signal, and asks for absolute URLs in rel="canonical". If you send no signals, Google chooses the version itself, and it may not be the one you want.
How to Fix Encoding Duplicates, Step by Step
- Collect the variants. Export URLs from your server logs or the Search Console page indexing report. Paste them into the URL Decode operation of our tool to read them as plain text and spot double encoding.
- Choose one form. Use UTF-8 percent-encoding with upper-case hex. Our URL Encode operation uses PHP's
rawurlencode, which follows RFC 3986 and outputs upper-case hex. It encodes/as well, so encode each path segment separately, not the full URL. - Redirect the rest. Send double-encoded and other variants to the chosen URL with a 301. Avoid chains: one hop from the variant to the final URL. You can test each variant with the free redirect checker from SpeedyIndex, which shows whether a URL redirects, forms a chain or returns an error, without sign-up.
- Set the canonical. Put the absolute, encoded URL in
<link rel="canonical">on the page. - Fix internal links. Every menu, breadcrumb and in-text link should use the same encoded form, so you stop creating new variants.
- Update the sitemap and recheck. List only the canonical form, then confirm after recrawling that the right URL is in Google's index.
Sitemap Encoding Rules
- Google's guide to building a sitemap requires the file to be UTF-8 encoded.
- All tag values must be entity-escaped, as in any XML: write
&for&in query strings. - Use fully-qualified, absolute URLs, in the same encoded form as your canonical tag.
- One sitemap holds at most 50,000 URLs or 50MB uncompressed; split larger lists into several files.
Checking the Result in Google
After redirects and canonicals are in place, check which version Google keeps. For your own site, use URL Inspection in Search Console and compare the user-declared and Google-selected canonical. For long lists, the SpeedyIndex Google Index Checker checks many URLs at once with Google queries such as site: and the exact URL in quotes, so you can see whether the old variants are dropping out and the canonical form is indexed.
Frequently asked questions
Should URLs with non-English characters be percent-encoded for Google?
Yes. Google's URL structure guidelines say characters outside the ASCII range should be percent-encoded in links, and give https://example.com/gem%C3%BCse as the recommended form of https://example.com/gemüse.
Can an encoded and a decoded URL become duplicates in Google?
They can if your server returns the same page with a 200 status for two different strings, for example a double-encoded path or a path with lower-case and upper-case hex digits. Redirect the variants to one form and point rel="canonical" to that form.
How should URLs be written in an XML sitemap?
Use fully-qualified absolute URLs in exactly the form you use as canonical, save the file as UTF-8, and entity-escape XML characters such as the ampersand (&).
Split a URL into scheme, host, path, query and fragment, and decode every parameter.
Open the URL Parser