Statistical Significance Script for Google Analytics

By /

March 27, 2014

Graph Trend

When comparing two time periods in Google Analytics, we are given a percentage increase or decrease. In situations where there is a dramatic difference (as is often the case for year-over-year comparisons), we can safely assume that the result is statistically significant.

For example, in the below chart, every data point (day) is lower in the second period than in the first. We can reasonably conclude that there has been an increase in visits in our month-over-month comparison.

Clear Trend in Google Analytics

Clear Trend in Google Analytics

When we have a more-subtle increase (decrease) in a time comparison, however, the percentage increase (decrease) may not actually be statistically significant. This script will evaluate the graph’s data and determine whether (and at what level) the percentage change is statistically significant

Unclear Trend in Google Analytics

Unclear Trend in Google Analytics


Unclear Trend in Google Analytics E-Commerce

Unclear Trend in Google Analytics E-Commerce

This should be considered a Beta script. It has several limitations at this time, but they will be removed (hopefully) soon as I have time. Eventually I plan to release this as a Chrome Extension in the Google Marketplace.


  1. Set the date range and compare to date range such that they each have the same number of days, weeks, or months (6 to 40) and begin and end on the same weekday.
  2. On the graph, use the dropdown for the metric you want to test
  3. Copy and paste the Script into your developer console (F12 opens the developer console).
  4. The result of the test will be output to the developer console.

Script Limitations

  1. You must use comparable time periods in terms of Days of the week. If your date range starts on Monday and ends on a Friday and is 26 days, then the previous date range should also start on a Monday, end on a Friday, and be 26 days.
  2. Must use between 6 and 40 data points. If the graph is displaying days, between 6 and 40 days. If the graph is displaying months, between 6 and 40 months
  3. The script only determines if the percentage change is insignificant, or significant at 10%, 5%, or 1% level (p-values of 0.10, 0.05, or 0.01)
  4. Currently, the Script uses the Wilcoxon paired rank test. We lose power by not treating the data as a time series, and we make several other approximations. For greater than 40 data points, we can use a t-test to evaluate the significance of the percentage change displayed in the graph. This will be added in the next release.

I think it is important for us to incorporate statistical testing into our Google Analytics. Especially when there exists a subtle change in our data over time, we risk committing a type I error (false positive) and incorrectly appropriating our organization’s resources based on the faulty intelligence.

Noah is a former LunaMetrician and contributor to our blog.

  • Hristo Vassilev

    I would actually pay for this. :-)

  • Dan WIlkerson

    This is great, Noah; I’d love to see this built into the GA interface. I’d suggest turning this into a chrome extension.

  • Oriana

    Why do you apply statistics when analyzing the whole population of users in a given period? Statistics informs you how propable it is that the result observed in a sample would be also observed in the population. I doesn’t make sense to apply statistical test when you have the whole population!

  • Kamil van Buuren

    I like this very much!

    Would it also be possible to do this with differences in segments?
    So: not comparing over time, but comparing between two (or more) groups in the same period.

    Would be great :-)


  • Noah Haibach

    Thank you everyone!

    Dan, I am hoping to develop this as a Chrome extension.

    Oriana, this is a great question. In this situation, our population would be all the potential visitors to our web site. For each time period, each of those potential visitors is either a visitor or a non-visitor. There will always be variation between time periods, some due to factors in which we are interested (site design changes, SEO changes, changes in the Google algorithm, etc.), and some due to random and immeasurable factors. We are testing that the variation we observe is greater than would be permitted by random factors.

    Think about the statistical analysis of a drug study. Yes, we have access to the whole “population” that is being treated by the study. We use statistics to test for a real effect, one that is unlikely to have been observed due to chance.

    Hey Kamil, that is a really great idea (and one that might be even more useful than temporal comparisons). I will try to add it for the next iteration of this project.


  • Lauren

    Oriana – great question and Noah – great answer. I have struggled on whether or not I need to prove statistical significance with our visitor stats – particularly when setting goals and determining whether we hit the mark or not.

  • Noah Haibach

    Hey everyone,

    I’ve updated this script as a Chrome extension. I published a new blog post, and will be redirecting this old post there:

    Thank you, Lauren!

    And Kamil, I still do want to broaden the extension to test between segments. Haven’t added it yet, though.

Contact Us.


24 S. 18th Street, Suite 100,
Pittsburgh, PA 15203

Follow Us



We'll get back to you
in ONE business day.