We are reader supported and sometimes earn a commission if you buy through a link on our site.

How We Test VPNs at Pixel Privacy

In this article, I explain the methodology we use to test and analyze VPN services. We test speed, protection, global server network and more.

We here at Pixel Privacy work to ensure that we write the most accurate and comprehensive Virtual Private Network (VPN) reviews available on the web today. We base all reviews on extensive testing as well as first-hand everyday usage. We don’t just parrot what the providers say about themselves.

All reviews are performed from the perspective of the average VPN user who:

  • Values their online security and privacy
  • Is looking for fast speeds and reliable service
  • Requires native device support for numerous devices
  • Wants to access geo-blocked content, like Netflix
  • Is looking to bypass censorship
  • Values comprehensive customer service
  • Wants a good return on their investment

Online Security

VPN online security includes a secure method of authentication, as well as a secure way of encrypting a user’s connection. These technologies ensure your connection is secure, keeping your online activities undercover, keeping your personal information safe as well as ensuring that your data is not modified in transit.

Preferences Auto-connect

The online security criteria for VPN reviews include the following:

  • VPN protocols: The VPN must use a secure VPN protocol, such as the reliable old standbys of OpenVPN, L2TP, IKEv2 and SSTP. Newer protocols are also available, including the WireGuard protocol that multiple providers now use, as well as the Lightway protocol that ExpressVPN uses and the NordLynx protocol that NordVPN uses. These newer protocols are designed to provide additional protection as well as improved performance.
  • Encryption: Must use the AES 128-bit algorithm or stronger. AES 256-bit encryption is preferred.
  • Authentication protocol: Must be SHA256 or higher.
  • Key exchange: RSA and DH keys must be 2,048-bit or better.
  • Perfect Forward Secrecy: Prevents session keys from being compromised, even if the server’s private key is compromised.
  • IP leak, DNS leak and WebRTC leak prevention: All three of these leak protections should be built into the provider’s apps.
  • Private DNS servers: The provider should operate its own DNS servers, not routing DNS requests through an Internet Service Provider’s (ISP’s) DNS servers, or through a third-party public provider, such as Google DNS.
  • Kill switch protection: Kill switch protection ensures that internet traffic is halted if the VPN connection is interrupted. While many providers offer a kill switch option in their desktop app offerings, they lack one in their mobile app offerings.
  • Physical vs. virtual servers: Physical servers stationed inside a country’s borders are preferable to virtual servers that serve a country but are physically located in another region of the world. This helps cut down on any issues related to which country’s data privacy laws apply to the server’s users.

Check out my dedicated article for more information about VPN security features.

How Pixel Privacy Tests for VPN Leaks

Data leaks occur when data is sent or received outside of the encrypted VPN tunnel, compromising a user’s privacy.

Types of VPN leaks include:

  • DNS leaks: This is when Domain Name System (DNS) traffic is sent to your ISP’s DNS servers instead of the VPN’s own DNS servers. Since DNS is how internet addresses (like PixelPrivacy.com) are translated to the numeric machine addresses that computers use, a DNS leak can reveal the websites you visit.
  • IP leaks: This occurs if your VPN fails to properly hide your IP address, or if data isn’t being properly sent through the VPN’s encrypted tunnel. This can reveal your actual IPv4 or IPv6 address.
  • WebRTC leaks: WebRTC is a communication protocol used by voice and video chat services like Discord and others. If this data is sent outside of the encrypted tunnel, user security and privacy can be compromised.

Pixel Privacy uses websites like ipleak.net and browserleaks.com to test for the above leaks. The tests reveal whether a VPN reveals our current real IP address, DNS services and WebRTC addresses.

Website leak test websites

The tests are run first without a VPN, and are then run again while using a VPN. If any of the tested addresses are the same across the two tests, we know there is a leak, risking a user’s online security and privacy.

Privacy

A VPN’s privacy protections should ensure that no third parties, whether it’s your ISP, the government, hackers, or even the VPN provider itself, can identify or monitor you or your online activities while connected to the VPN’s servers.

In addition to our reviews, you can also find out most of what you need to know about a VPN’s privacy protections by perusing its privacy policy.

Pixel Privacy examines a VPN’s privacy policy, which is usually posted on a VPN provider’s website. (If a VPN doesn’t have a privacy policy available on its site, avoid it.)

NordVPN Private Policy

A strong privacy policy should indicate the provider does NOT collect logs of the following:

  • Any online activities, such as websites and pages visited, apps and services used, purchases, downloads, search queries, and streaming services viewed, as well as any of your other online antics.
  • Your actual IP address.
  • The IP address of the server you connected to.
  • Connection time and date stamps.

The above information can be used to identify you and your online activities if the VPN logs any of the information.

It should be noted that while we prefer no server logs of any kind, some log information cannot be used to identify a user as long as it isn’t connected to a username. This can include the date of a connection (but not the time), the amount of data consumed, and which country a user connected to (but not the specific server that you connected to.)

While most VPNs that Pixel Privacy has tested claim to offer a no-logs policy, some still log some of the information listed above. We prefer a VPN service that collects no identifiable information. If it can be tied back to you, it shouldn’t be collected.

Also, make sure to check on where a VPN provider is located.

Many countries have mandatory data retention laws that force or allow ISPs and VPNs to log some types of user data and then hand it over when requested to, or to have a “backdoor” that allows government officials to access your user data. We prefer VPNs located in countries that have no data retention requirements.

A VPN should ideally offer anonymous ways to pay for their services. If you can use a form of cryptocurrency to pay, that’s a plus. By using cryptocurrency and a disposable email address to subscribe to a VPN, you score additional privacy protections.

Connection Speed

Pixel Privacy’s connection speed test results are posted as global averages.

Using a Windows machine connected to a 1 Gbps internet connection in the United States, we test each VPN for 3 days, 3 times per day, across 3 locations. We test servers based in North America, Europe and Asia. We then calculate an average for the tests, which is our final result.

All outliers – which are results that are more than three standard deviations away from the mean – are thrown out, and the VPN is retested.

We use the fastest secure connection protocol available from each provider. This can include WireGuard, OpenVPN and others. We measure all speeds using the popular Ookla Speedtest.net Windows app. We report only download speeds, as that is likely what most of our readers are interested in.

Ookla Speedtest.net Windows app

We do not measure VPN latency, also known as ping time or lag. This is because ping is indicative of a test machine’s distance from the VPN server, not an indication of the VPN server’s actual performance. (The closer you are to the VPN server, the lower your ping time will likely be.)

We also note a VPN’s performance during day-to-day use, including streaming, online gaming, file sharing and other popular online activities.

Please keep in mind that server download speeds can vary according to your location, the server’s location, the time of day you connect, how many users are connecting to a single server and much more. Our fastest server likely isn’t the same server that offers the best performance for your location.

Unblocking Geo-Blocked Content, Like Streaming Services

Unblocking geo-blocked content is arguably one of the most popular uses for a VPN.

The content most unblocked is streaming services, such as Netflix. Streaming services restrict access to their content on a country-by-country basis thanks to the content licensing contracts they sign with television and movie producers.

A VPN makes it appear as if you’re located in another country, opening up access to geo-blocked content in that country.

Disney+ website

Pixel Privacy checks to ensure that a VPN can access as many of the following streaming services as possible:

We test access to the streaming sites on both desktop and mobile devices. This is because some VPNs can offer access on one platform, but not others.

We have run thousands of tests for the various streaming services, and as you might imagine, Netflix is the most tested streaming service.

Censorship

If you reside in an overly restrictive country where online access is censored, such as China, your main reason for using a VPN is likely to be to regain access to blocked websites and services. Blocked content can include western social media, news sites, entertainment and even gaming.

When possible, we test a provider’s ability to unblock content from government blocks like China’s Great Firewall. However, keep in mind that blockade situations can change on a regular basis, so be sure to check with the VPN provider you’re considering to determine the current situation.

A VPN also comes in handy for accessing online content that may be filtered or blocked by your school or place of employment. (But make sure to keep up on your studies and your job tasks, okay?)

Global Server Networks

While the total number of VPN servers a provider has is a notable statistic, the number of places around the globe where the servers are located is actually a more important statistic.

The average VPN service has servers based in 30 to 60 countries around the globe. (Some providers, like ExpressVPN and CyberGhost, have servers stationed in more than 90 countries.)

Always check a provider’s list of countries to make sure that it has servers located in a country you wish to connect to.

Provider’s list of countries

Keep in mind that some VPNs use virtual server locations. Virtual server locations are servers that are physically in one country, but that provides IP addresses located in another country.

While virtual server locations can still provide access to geo-blocked content, you run a risk that the country where the server is actually located may have either questionable or no data privacy protection laws, which could endanger your online privacy.

On the other side of things, virtual servers offer IP addresses in countries where you don’t want a VPN to operate a server due to restrictive local laws, such as Iran or China.

Apps

Let’s face it, the best VPN provider in the world is worthless if it doesn’t offer support for your favorite devices. While most VPN providers offer support for the big 4 device platforms (Android, iOS, macOS and Windows), it’s a different tale when you look for support for other popular platforms.

Other popular device platforms include Amazon Fire TV, Android TV, Linux, ChromeOS and others. You’ll find that many VPN providers also offer browser extensions for use with the Chrome, Edge, Firefox and even Opera browser platforms.

VPN provider browser extensions

It should be noted that no VPNs offer VPN apps for game consoles, like the Xbox and PlayStation console lineup, or apps for streaming boxes and sticks, like Roku, Apple TV, many smart TVs, and other platforms.

However, if the provider offers router support, your game console or streaming box can still take advantage of the provider’s protection and enhancement features.

Features we look for in VPN apps include:

  • A kill switch: Drops your internet connection if you lose the connection with the VPN’s servers.
  • Ad and malware blocking: Many malicious websites will try to throw up malicious pop-ups and download malware to your devices. This feature helps protect against that.
  • Phishing protections: Protects you by recognizing and blocking sites that attempt to “phish” information from its visitors.
  • Split tunneling: Allows users to pick and choose which apps will use the VPN’s encrypted tunnel.
  • Automatic WiFi hotspot protection: Automatically enables VPN protection when a device connects to a public WiFi hotspot.
  • Obfuscation features: Hides your VPN connection by making it appear to be normal browser traffic.

We also take a close look at ease-of-use, reliability and system resources use.

VPN users who require protection for multiple devices, such as those who own a small business or are part of a large family, will want to take note of how many simultaneous (also known as concurrent) connections that a VPN provider allows. While some providers offer as few as 5 simultaneous connections, others offer unlimited connections at once.

If you’re simply looking for a way to access geo-blocked content without the online security protections of a VPN, many VPN providers also offer “smart DNS” services that you can use with streaming-capable devices when a VPN app is not available.

Customer Support

No matter how reliable your VPN of choice proves to be, there is a good chance that you’ll have questions about or need help with a VPN’s services. Even if it’s simply a billing question, you’ll still appreciate a quick and knowledgeable reply when you have a support issue.

NordVPN customer support page

So each full review of a VPN on the Pixel Privacy site includes information about the provider’s customer support offerings. We contact the provider and take note of when a question was asked and when a reply was received. We also note the accuracy of the response and how helpful the response was.

We look for live chat options (preferably available 24/7), ticket tracking systems, other available forms of contact, and whether or not a self-service support library is available.

Billing

Pixel Privacy gives bonus points to providers that accept cryptocurrency, and we also prefer providers that do not by default auto-renew your subscription each month, and frown upon providers that do not offer a quick and easy way to cancel your subscription. We also like to see no-risk money-back guarantee periods of at least 30 days.

In Closing

As you can see, there are numerous factors to consider when deciding on which VPN is right for your particular needs. This is why we try to cover every feature of any VPN we review, offering you the most comprehensive coverage of VPN features.

Leave a Comment