Upcoming LunaMetrics Seminars
Pittsburgh, Jan 12-16 Boston, Jan 12-16 New York City, Jan 26-30 Denver, Feb 9-13

Are You Missing Referral Traffic in Universal Analytics?

Don’t fall for that old Jedi mind trick and simply ignore what Universal Analytics tells you to ignore… they might be the referrals you are looking for.

fooling the troopers with an old mind trick

Did you know that Universal Analytics’ default setting is not to count referrals from your domain? That’s right, Universal Analytics is going to ignore self-referrals by default. This may not be a good thing if you need the information to fix coding errors, but that’s another story.

Today’s story is how to make sure that your idea of self-referrals matches what Universal Analytics is calling a self-referral. If it doesn’t, you may be ignoring some referrals that you didn’t want to ignore. Depending on your situation, you may need to change a setting or even add some custom code to see all the referrals you want.

Read more to understand what’s really going on with referrals in Universal Analytics so you can make an informed decision about what to ignore.

What’s in your referral exclusion list?

When you start a new property, Universal Analytics automatically adds your domain to that property’s referral exclusion list. This list controls which referrals are ignored. Right now you should take a look at what’s in there, if you don’t already know.

referral-exclusion-list

Here’s how Universal Analytics decides what to put in your referral exclusion list by default: It takes the domain you entered when you created the property, such as jobs.tatooine.com, and uses only the second-level and top-level parts, so that the default domain in the list would be tatooine.com. If you’re upgrading to Universal, this may not happen automatically, but you should still check out the list to confirm.

Sounds reasonable, right? In fact, this default setting works out fine as long as you want to ignore referrals from every hostname that contains tatooine.com.

If there are third-level or higher domains, the default setting ignores them all: www.tatooine.com, vacations.tatooine.com, as well as jobs.tatooine.com. The default setting also ignores referrals from second-level domains that include tatooine.com, such as visittatooine.com or cantinatatooine.com.

Whoa, whoa, whoa… you may be thinking, “I didn’t want to go that far. I’ll just change my referral exclusion list to specify exactly the domain I want to ignore.” And so you might enter jobs.tatooine.com – if that’s the property you’re collecting data for and you want to see referrals from everywhere else – and think your job is done.

Unfortunately, the result is that now you’re not excluding any referrals because the Referral Exclusion List accepts only the second-level and top-level parts of any domain. If you enter only third-level or higher domains, it’s like entering nothing at all.

Do you need to ignore any referrals?

In the simplest, ideal scenario, your tracking code would be implemented perfectly and your traffic reports would contain nothing or next to nothing in the way of self-referrals. It’s possible you wouldn’t need to enter anything in your referral exclusion list, i.e. you could remove the default entry. Or you might need only the domain of your third-party shopping cart in the list.

Here are a couple of scenarios with related domains where things could be more complicated:

Scenario 1: Ignore one subdomain, but not main domain or other subdomains
Your property collects data for jobs.tatooine.com. You want to ignore referrals only for that hostname, and you want to see referral traffic from everywhere else, including tatooine.com.

Scenario 2: Ignore main domain and one subdomain but not another
Your property collects data for tatooine.com and vacations.tatooine.com. You want to ignore referrals from these two hostnames, because your tracking counts a user who views pages from both as part of the same session. On the other hand, you want to see referral traffic from other hostnames such as jobs.tatooine.com.

What are your options in each case? As mentioned earlier, the default exclusion tatooine.com is going to ignore referrals from every hostname that contains tatooine.com, and entering anything more specific is like entering nothing at all.

Things look pretty hopeless. Maybe someday Universal Analytics will allow you to specify exact hostnames in the referral exclusion list, but until then…

Custom JavaScript to the rescue

Our solution is to come up with our own mind trick. With some custom JavaScript, we’ll make Universal Analytics think, “This is not the hostname you’re looking for.”

TrafficSources-Thumb

What makes this work is Universal Analytics’ Processing Flow for Campaigns and Traffic Sources.

When Universal Analytics is deciding what traffic information to record for a single hit, one of the things it may look at is the document referrer, i.e. the page the user was viewing when he or she clicked to arrive at the current page. Universal Analytics writes the document referrer into its own parameter named documentReferrer (clever, eh?) and then compares that parameter with your referral exclusion list.

Our trick is to set documentReferrer to a different name – one we still recognize, but that will not match the name(s) in the referral exclusion list. This can be anything that we make up, as long as WE know what it means. In this case, I’ll use tatooinetheplanet.com instead of tatooine.com.

Step 1: Keep the default domain in the referral exclusion list.
For either of my two scenarios, I want to have tatooine.com in the referral exclusion list. Remember, this is going to match everything with tatooine.com, so the next step is the tricky part.

Step 2: Conditionally rewrite the document referrer.
In Google Tag Manager, create a custom JavaScript macro named something like {{rewritten referrer}} with the function shown below. Or, if you’re not using Google Tag Manager, include the function in your page source. The function checks to see if the document referrer matches one of the actual domains we want to exclude. (Here’s where we cover specific domains that we cannot include in the referral exclusion list.)

If the referrer matches tatooine.com but does not match our specific domains, then we’ll trick Universal Analytics into thinking it also does not match the domain in the referral exclusion list by rewriting it as something like tatooinetheplanet.com.

Thanks to my colleague Alex Moore for writing the original code, which I’ve slightly modified here.

Step 3: Set the document referrer field to the value returned in Step 2.
Do this in Google Tag Manager by entering your macro as the value for Document Referrer, which you’ll find under Basic Configuration in your Universal Analytics tags. Or write it into your page source between the create and send methods, as described in the documentation for Document Referrer: ga(‘set’,’referrer’,VALUE); …replacing VALUE with the returned value of your function from Step 2.

document-referrer-GTM

Step 4: Use a Search & Replace filter to restore the referrer (optional).
Your rewritten document referrer will end up as one of the sources shown in the traffic reports. If you’d rather see the original hostname instead, you can use a Search & Replace filter to reverse engineer the Campaign Source field and put it back to the way it was. For example, search for tatooinetheplanet.com and replace it with tatooine.com.

Have you been ignoring referrals that you didn’t mean to ignore? If so, have you addressed the issue in other ways than the ones described here? Please share in the comments.

Dorcas Alexander

About Dorcas Alexander

Dorcas Alexander is a Digital Analyst working with Google Analytics. Her path to LunaMetrics included stints in ad agency creative, math, computer science, language technology research, and corporate training. She loves to learn and teach what she’s learned. One of the top-rated tournament Scrabble players in Pennsylvania, Dorcas has an insatiable drive to compete and win. “Impossible” is not in her vocabulary.

http://www.lunametrics.com/blog/2014/06/19/missing-referral-traffic-universal-analytics/

15 Responses to “Are You Missing Referral Traffic in Universal Analytics?”

Pavel says:

Hello Dorcas:
Thank you for interesting idea. Am I right thinking that you didn’t include the macro’s code into your text?
Best wishes,
Pavel

Dorcas Alexander Dorcas Alexander says:

Hi Pavel, The macro code is embedded directly from pastie.org and I can see it, but perhaps a setting on your browser is preventing it (and may be preventing others from seeing it as well). Here’s the link, hope it works for you! http://pastie.org/9304460

Pavel says:

Dorcas, thank you very much!

Boyislaw says:

Hello Dorcas,

Very usefull text!

I have one question, if I use cross domain autolink Universal Analytics code, do I need to set domain in Referral Exclusion list in order to have proper cross domain tracking?

Best regards

Dorcas Alexander Dorcas Alexander says:

Hi Boyislaw, Yes, you should always add to the referral exclusion list any domains that you want to be part of cross-domain tracking.

Dorothy says:

Hello Dorcas,
Do you have any simmilar solution for standard implementation of Universal? I mean not via GTM.

Regards,
Dorothy

Dorcas Alexander Dorcas Alexander says:

Hi Dorothy, If you’re not using Google Tag Manager, simply include the function in your page source, the same way you include the Universal Analytics code. Here’s the function again in case you can’t see the embedded script in the blog post. http://pastie.org/9304460 Then for step 3, add a line to your page source between the create and send methods to set the referrer, as described in the documentation for Document Referrer: ga(‘set’,’referrer’,VALUE); …replacing VALUE with the returned value of your function from Step 2.

Emily says:

Hey Dorcas, thanks very much for this very informative article! Just a question: I have a site that has been receiving a number of visits onto a landing page that is not accessible via any other channel except email. I see, however, that about 50% of visits are being attributed to “direct” instead of email, and I’m sure that my email URLs have been tagged. Could this be a result of adding my domain onto the referrals exclusion list? I’m not sure what this scenario would look like, but can only think that this is the root of the problem…?

Dorcas Alexander Dorcas Alexander says:

Hi Emily, No, adding your domain to the referral exclusion list would not cause GA to ignore campaign tags in the URLs of your landing pages, because a referral is the page they were on before they came to your landing page (i.e. the mail site, or no site if using a desktop mail client). The large amount of direct traffic could be a combination of factors, some of which you can’t do anything about. It’s possible that the URL of that page is being copied and shared, which would lead to direct visits. Or there might be a technical issue such as the campaign parameters being stripped in between the email and landing on your site. It could be difficult to track down. Think about the possible behavior of your users and follow the path yourself from several browsers on several devices, if you haven’t already. Good luck!

Vincent says:

Hi Dorcas,

Thanks for the informative post. If a site is on the referral exclusion list and your site receives a visit from it, does Analytics discard the entire visit and not record it?

Dorcas Alexander Dorcas Alexander says:

Hi Vincent, No, Google Analytics does not discard the visit if your site receives a visit from a site on the referral exclusion list. Instead, GA attributes that session to “direct traffic” and all data is collected normally otherwise.

Michael says:

Hi Dorcas,
Great article.

I want to make sure I understand the above statement as I’m fighting some cross domain battles currently.

You said that a visit from a site on the referral exclusion list will be attributed to “direct”. Is that correct? I thought domains on that list will ensure a new session is NOT started, and they will continue with their original source.

I want a visit from one of those sites to NOT have a new session, but rather continue with the original source attributed session so I can attribute a sale to the original source of the traffic. We are using: Main site + 3rd party Shopify cart site.

Note: I’m using GTM and configured my autolinked domains to include the same ones on the exclusion list.

Thanks!
Michael

Dorcas Alexander Dorcas Alexander says:

Hi Michael, That was the quick and simple answer, and is what Google’s processing flow chart indicates will happen. The more complicated reality is that only if there is no current session (session has timed out or it’s a new user), then an ignored referrer causes the session to be attributed to direct traffic. If there is a current session (session has not timed out), then referrals from third-party shopping carts (or any other site in the referral exclusion list) will allow the session to continue with the existing source. I just ran a quick test to be sure because, like you, I count on this happening, too.

Kim says:

Hi Dorcas,

thanks for this great article!

I’ve a question concerning third level domains:

We’ve added a bunch of third party payment websites to the referral exclusion list.

Most of these payment sites work with subdomains e.g.: paymentgate.paymentsite.com.

I’m wondering if this will work or if we should have added “paymentsite.com” instead, as your statement above implies: “If you enter only third-level or higher domains, it’s like entering nothing at all.”

Thanks,
Kim

Dorcas Alexander Dorcas Alexander says:

Hi Kim, Simply look at your Referrals report in GA. Do you see any referrals from third party payment sites? Make sure you look at a date range after you added sites to the referral exclusion list, and when you are sure that purchases were made using one or more of those sites. If there are referrals from third party payment sites in the Referrals report, then it’s not working, and you’ll need to adjust the referral exclusion list.

Leave a Reply