Free Multivariate Web Performance Testing Arrives – WebsiteTest.com
Deep Insights for Tuning Site Performance
Yottaa is proud to introduce Websitetest.com, a free testing tool with multivariate testing capability. In a single test, see how your website performs across multiple locations, browsers, and connectivity types. Set tests to run in the future at time increments of your choice. Receive immediate results or check back later to see test results over time.
Get the most detailed and statistically significant snapshot of your website’s performance available. For free. The results include interactive visualization of the page loading sequence, and a
waterfall chart of the assets.
Why Testing, Why Now?
Web development depends heavily on testing. A developer must test his/her site’s appearance in a variety browsers, load test it for how it handles traffic, and make sure the UI elements function properly. These tests can mostly be performed on the developer’s computer in a testing environment.
Web performance testing, however, presents a different breed of challenge. To prevent performance issues or to tackle them as they arise, developers must find out how users will experience their site not only in different browsers, but also in different locations and with different types of connections. Does a user in Singapore running Internet Explorer on a DSL connection have a different page load experience than a user in Oregon running Chrome with FIOS? If so, which variable is the difference-maker?
To find out, developers previously had to perform repetitive manual tests. The process was time consuming and imprecise.
A Simple and Powerful Tool
Websitetest.com offers the ability to test across multiple variables (“multivariate testing”) packaged neatly into an easy-to-use interface. It’s never been easier for developers to isolate the causes of performance issues and gain instant visibility into website performance.
Even if you’re intimately aware of your site’s performance, frequent testing remains a best practice. Things change quickly on the web. This is especially true for third-party plugins and dependencies: these are now present on nearly every website, and in such cases overall site performance depends on elements that are outside of your direct control. This means that keeping close tabs on third party assets is important in ensuring consistent performance.
At Yottaa we’re working hard to make the web faster for everyone, and we think Websitetest.com is a big step in the right direction. In this post we hope to clarify the tool’s capabilities and summarize how developers and site owners can use the results to gain a broader understanding of a website’s performance.
If you have feedback about the tool, we’d love to hear it! Please comment, or Tweet @Yottaa.
Read on to learn how to take advantage of Websitetest.com’s features and options.
Setting up a Test
On the Websitetest.com homepage you’ll find options that refer to the “multi” side of the multivariate tests. For each of these options you’ll be presented with a particular set of choices: for a multi-location test you’ll chose one browser, one last-mile connectivity, and anywhere from one to 14 locations. Likewise for a multi-browser test you’ll chose one location and one connectivity, and up to three browsers. And so on.
After clicking the radio button of your desired test option, click the middle tab (highlighted above) to see the breakout options. (Content in the middle tab will change depending on your selection.) Once you make your choices in the middle tab, you may start the test: enter your URL in the field and hit the TEST button, and a single test will be run immediately.
You may also wait to begin your test, and instead click the rightmost tab on the homepage labeled Advanced Options. Here you can set up to 100 tests to occur over time, at increments ranging from every 5 minutes to every 12 hours.
The other important choice in Advanced Options is whether the test case will simulate a user with an empty cache, or simulate two users: one with an empty and one with a full cache. We recommend “empty and full” for a complete view of web performance – but if you’re sure that you will only need to test for empty-cache visitors, then making that selection will decrease the time it takes to complete the test.
Now you may enter the URL in the field and hit the TEST button.
Web Performance Results – Summary
The results may take a minute or two. Don’t worry – they’re on the way!
Once the “Summary” results page appears feel free to start exploring, but keep in mind that if you’ve selected multiple variables and/or scheduled multiple tests to run, the data on the Summary page may change in real-time as new data arrives.
The first stat you’ll notice is the Yottaa score. A detailed explanation of Yottaa Score can be found on our support site, but in short, the Yottaa Score is a number from 1-100 (higher = better) that rates overall web performance. It’s not just the time it takes to completely load the website. Rather it’s the result of an algorithm that factors in data from four distinct stages in the loading process. The equation assigns certain weight to each of these stages in order to approximate how an average visitor will perceive the speed of the page load process, as well as how quickly the page actually finishes loading. It’s the most complete view of a web page’s performance in 3 digits or less!
The rest of the front page contains summary statistics from the test. For example, “Time to interact” measures the elapsed time when the site is done loading from the perspective of the user. Third party trackers and scripts might still be loading in the background, but the user experiences full functionality. “Time to last byte” is the length of time elapsed when the entire site is finished, including the background scripts and trackers. “Content” breaks down all content on your site by category (# of images, # of JavaScript assets, etc.).
Now for the Good Stuff – Visuals
Under the Data Samples heading on the Summary page you’ll find information pertaining each instance of your test. (If you chose to run only one test with single variables, there will only be one line here). Clicking one of the hyperlinks in the Data Sample section will bring up a new tab within the page that looks much like the Summary page, but instead displays values that are specific to the test instance you clicked, as well as additional deep dive statistics.
See What Your Users See – Literally – With Screenshots
On each test instance’s results page you’ll find “page loading screenshots.” This feature allows you to see exactly what the visitor would see in the given test situation. It’s valuable for quickly finding out whether important elements, such as your site’s logo, show up early or late in the loading process.
Moreover, the screenshots offer a way to go beyond the raw load time numbers like “first paint” and “first byte” and make less-quantifiable judgments about the aesthetics of your page load process. Put yourself in the user’s shoes: does the page load experience engage the user or breed impatience?
Waterfall Chart: deep dive insights into how the page loads and renders
We introduced our new waterfall chart last month, when we integrated it with our monitoring service. We’ve now incorporated it into Websitetest.com as well! The waterfall chart displays by default below the screenshots on a results page.
Rather than portraying the elements of the web page in the order that they began loading, the Yottaa waterfall allows you to sort ascending and descending any of the columns, as well as to isolate by asset type. For instance, you may choose to view only the pages .PNG images, or look at the heaviest or slowest-loading elements in descending order. You can even isolate assets by HTTP response, such as 400-level failures. You may also click any asset to bring up “deep-dive” request and response header information, within the waterfall chart itself.
These features help web developers and operations teams to quickly and easily zero in on the key performance information, rather than requiring them to hunt through the list of elements for answers that may or may not be there. It’s the quickest way we know of to access specific performance information. (To get the most of Yottaa’s waterfall chart, check out our two tutorial blog posts:
Part I and
Part II.)
More Visuals – Content Complexity and HTTP
Toggling the pulldown menu above the waterfall, you’ll find options to view the HTTP timeline for the test instance and the content complexity graphs for the page.
The HTTP timeline is a blown-up version of the first line of your waterfall chart. This information is crucial because the entirety of your website is waiting for this HTTP request to go through. Viewing the HTTP timeline separately simply allows you to see the breakdown of the load process visually, since many times this will appear too compressed to see easily in the waterfall.
You can use this feature to figure out if you have a problem in your page rendering on the server. If your “Time to first byte” or “Time to last byte” is long, there may be a problem. If your “Time to connect” is long it may be the case that your server cannot handle the load.
Content complexity graphs are a simple way to see the types of assets on your site in relative terms. The graphs show your assets by quantity (number of requests) and weight (number of bytes). While the waterfall shows individual assets in great detail, these graphs provide an easily digestible overview of your whole site.
The graphs can help to make clear areas of potential improvement. In the example below, there are relatively few JavaScript assets, but they comprise a substantial amount of the total weight. This indicates that at least some of the JavaScript assets could be better optimized.
Also, simply seeing things visually can help to contextualize the website in a new way, even if the information is known. This site’s owner may be aware that there are 77 images on her site, but perhaps she would be surprised to see that images comprise such a large percentage of all requests. This might be cause enough for the site owner to consider cutting down the number of images through Data URIs or other methods.
In Conclusion…
Test to your heart’s content! Use single-instance tests to quickly get an idea of a site’s performance, or set multivariate tests over time to get the most detailed performance data available on the Web.
We’d love to hear your feedback on WebsiteTest.com, including suggestions for improvements to the tool. Leave comments below!