Block referral spam

The following example shows how to configure Fastly Edge Dictionary with a custom VCL snippet to block referral spam from your Magento Commerce Cloud site.

We recommend adding custom VCL configurations to a Staging environment where you can test them before running them against the Production environment.

Prerequisites:

Configure the environment for Fastly services. See Set up Fastly.
Get Admin credentials for your Magento Commerce Cloud environment.
Review your site logs for fake referral URLs and make a list of domains to block.

Create a referrer block list

Edge Dictionaries create key-value pairs accessible to VCL functions during VCL snippet processing. In this example, you create an edge dictionary that provides the list of referrer websites to block.

Log in to the Magento Admin UI.
Click Stores > Settings > Configuration > Advanced > System.
Expand Full Page Cache > Fastly Configuration > Edge dictionaries.
Create the Dictionary container:
- Click Add container.
- On the Container page, enter a Dictionary name—referrer_blocklist.
- Select Activate after the change to deploy your changes to the version of the Fastly service configuration that you are editing.
- Click Upload to attach the dictionary to your Fastly service configuration.
Add the list of domain names to block to the referrer_blocklist dictionary:
- Click the Settings icon for the referrer_blocklist dictionary.
- Add and save key-value pairs in the new dictionary. For this example, each Key is the domain name of a referrer URL to block and Value is true.
- Click Cancel to return to the system configuration page.
Click Save Config.
Refresh the cache according to the notification at the top of the page.

For more information about Edge Dictionaries, see Creating and using Edge Dictionaries and custom VCL snippets in the Fastly documentation.

Create a custom VCL snippet to block referrer spam

The following custom VCL snippet code (JSON format) checks incoming requests and blocks requests from any referrer site included in the referrer_blocklist edge dictionary.

{
  "name": "block_bad_referrer",
  "dynamic": "0",
  "type": "recv",
  "priority": "5",
  "content": "set req.http.Referer-Host = regsub(req.http.Referer, \"^https?://?([^:/\\s]+).*$\", \"\\1\"); if (table.lookup(referrer_blocklist, req.http.Referer-Host)) { error 403 \"Forbidden\"; }"
}

Review the example code and change values as needed:

name — Name for the VCL snippet. For this example, we used block_bad_referrer.
dynamic — Value 0 indicates a regular snippet to upload to the versioned VCL for the Fastly configuration.
priority — Determines when the VCL snippet runs. The priority is 5 to run this snippet code before any of the default Magento VCL snippets (magentomodule_*) assigned a priority of 50.
type — Specifies a location to insert the snippet in the VCL version. In this example, the VCL snippet is a recv snippet. When the snippet is inserted into the VCL version, it is added to the vcl_recv subroutine, below the default Fastly VCL code and above any objects.
content — The snippet of VCL code to run in one line, without line breaks.

In this example, the VCL code logic captures the host of a referrer website into a header, and then compares the host name to the list of URLs in the referrer_blocklist dictionary.

If the host name matches, the request is blocked with a 403 Forbidden error.

See the Fastly VCL reference for information about creating Fastly VCL code snippets.

Add the custom VCL snippet to your Fastly service configuration from the Magento Admin UI (requires Fastly module 1.2.58 or later). If you cannot access the Admin UI, save the JSON code example in a file and upload it using the Fastly API. See [Creating a VCL snippet using the Fastly API](/devdocs/2.2/cloud/cdn/cloud-vcl-custom-snippets.html(#manage-custom-vcl-snippets-using-the-api).

Add the custom VCL snippet

Log in to the Magento Admin UI.
Click Stores > Settings > Configuration > Advanced > System.
Expand Full Page Cache > Fastly Configuration > Custom VCL Snippets.
Click Create Custom Snippet.

Add the VCL snippet values:

Name — block_bad_referrer
Type — recv
Priority — 5

VCL snippet content —

set req.http.Referer-Host = regsub(req.http.Referer,
"^https?://?([^:/\s]+).*$", "1");
if (table.lookup(referrer_blocklist, req.http.Referer-Host)) {
  error 403 "Forbidden";
}

Click Create.
After the page reloads, click Upload VCL to Fastly in the Fastly Configuration section.
After the upload completes, refresh the cache according to the notification at the top of the page.

Fastly validates the updated VCL version during the upload process. If the validation fails, edit your custom VCL snippet to fix any issues. Then, upload the VCL again.

Modify the custom VCL snippet

Log in to the Magento Admin UI.
Click Stores > Settings > Configuration > Advanced > System.
Expand Full Page Cache > Fastly Configuration > Custom VCL Snippets.
In the Action column, click the settings icon next to the snippet to edit.
After the page reloads, click Upload VCL to Fastly in the Fastly Configuration section.
After the upload completes, refresh the cache according to the notification at the top of the page.

The Custom VCL snippets UI option shows only the snippets added through the Admin UI. You must use the Fastly API to manage custom snippets added through the API.

Delete the custom VCL snippet

You can delete custom VCL snippet code from your Fastly configuration by uploading an empty version of the snippet from the Magento Admin UI, or delete it completely using the Fastly API.

Upload an empty version of the snippet file to Fastly to remove the VCL logic from the active VCL version:
- Edit the snippet and delete the VCL snippet content.
- Save the configuration.
- Upload the VCL to Fastly to apply your changes.
Use the Fastly API Delete custom VCL snippet operation to delete the snippet completely, or submit a Magento support ticket to request deletion.

Instead of manually uploading custom VCL snippets, you can add snippets to the $MAGENTO_CLOUD_APP_DIR/var/vcl_snippets_custom directory in your environment. Snippets in this directory upload automatically any time you click upload VCL to Fastly in the Admin UI. See Automated custom VCL snippets deployment in the Fastly CDN module for Magento 2 documentation.

Dec 20th, 2019