The DataDome Developer Hub

Welcome to the DataDome developer hub. You'll find comprehensive guides and documentation to help you start working with DataDome as quickly as possible, as well as support if you get stuck. Let's jump right in!

Get Started    

Varnish Setup

DataDome Varnish module detect and protect against bot activity

Before the regular Varnish process, the module makes a call to the DataDome API using a keepalive connection.
Depending on the API response, the module will either block the query or let Varnish continue the regular caching process.
The module has been developed to protect user experience as if any error was to occur during the process or if the timeout was reached, the module would automatically disable its blocking process and allow those hits.

Compatibility

DataDome support module for Varnish 3.0, 4.0, 4.1, 5.0, 5.1.

Every new release of the module is strongly tested on the following distributions:

  • Debian 6/7/8.1
  • Ubuntu 12/14/15
  • Centos 6/7
  • SUSE 11

Prerequisites

You must have the following libs installed:

 apt-get install automake libtool python-docutils libvarnishapi-dev pkg-config
yum install automake libtool python-docutils varnish-libs-devel 

Installation

You just need to install DataDome Vmod as following:

rm -f DataDome-Varnish-5.1-latest.tgz
wget https://package.datadome.co/linux/DataDome-Varnish-5.1-latest.tgz
tar -zxvf DataDome-Varnish-5.1-latest.tgz
cd DataDome-VarnishDome-*
./autogen.sh
./configure
make
make install
# Copy `datadome.vcl` to your VCL folder 
cp datadome.vcl YOUR_VARNISH_VCL_FOLDER
rm -f DataDome-Varnish-5.0-latest.tgz
wget https://package.datadome.co/linux/DataDome-Varnish-5.0-latest.tgz
tar -zxvf DataDome-Varnish-5.0-latest.tgz
cd DataDome-VarnishDome-*
./autogen.sh
./configure
make
make install
rm -f DataDome-Varnish-4.1-latest.tgz
wget https://package.datadome.co/linux/DataDome-Varnish-4.1-latest.tgz
tar -zxvf DataDome-Varnish-4.1-latest.tgz
cd DataDome-VarnishDome-*
./autogen.sh
./configure
make
make install
rm -f DataDome-Varnish-4.0-latest.tgz
wget https://package.datadome.co/linux/DataDome-Varnish-4.0-latest.tgz
tar -zxvf DataDome-Varnish-4.0-latest.tgz
cd DataDome-VarnishDome-*
./autogen.sh
./configure
make
make install
## Varnish 3.0 requires the Varnish sources on the server. 
## In case Varnish was installed from binary, setup Varnish source first
## In case it was build from source, go to step 2.

# 1 - Setup Varnish source
wget https://repo.varnish-cache.org/source/varnish-3.0.7.tar.gz
tar -zxvf varnish-3.0.7.tar.gz
cd varnish-3.0.7
./autogen.sh
./configure
make
export VARNISHSRC=$PWD
cd ..

# 2 - Setup DataDome Module
rm -f DataDome-Varnish-3.0-latest.tgz
wget https://package.datadome.co/linux/DataDome-Varnish-3.0-latest.tgz
tar -zxvf DataDome-Varnish-3.0-latest.tgz
cd DataDome-VarnishDome-*
./autogen.sh
./configure
make
make install

VCL Integration

Set the the following parameters in the datadome.vcl:

Add to your VCL file a line bellow after backend configuration and before any subroutine like in the following example:

#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at https://www.varnish-cache.org/docs/
# and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.

# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "127.0.0.1";
    .port = "80";
}

include "datadome.vcl";

sub vcl_recv {
    # Happens before we check if we have this in cache already.
    #
    # Typically you clean up the request here, removing cookies you don't need,
    # rewriting the request, etc.
}

sub vcl_backend_response {
    # Happens after we have read the response headers from the backend.
    #
    # Here you clean the response headers, removing silly Set-Cookie headers
    # and other mistakes your backend does.
}

sub vcl_deliver {
    # Happens when we have all the pieces we need, and are about to send the
    # response to the client.
    #
    # You can do accounting or modifying the final object here.
}

VCL Logic

If the same subroutine is defined a few times, Varnish concats them by order of appearance. That's why you can include datadome.vcl file without changing your subroutines.
If you include a logic based on the number of restarts, you should increase the count because DataDome module adds a restart for each hit.

Settings

setting
description
required
default

data_dome_shield.init (data_dome_backend, "KEY");

Replace "KEY" with your DataDome License Key.

yes

host

hostname of the DataDome API Server. Please use Latency Script .

yes

apivarnish.datadome.co

connect_timeout

Timeout used at initial API connection when opening a new connection

optional

150ms

first_byte_timeout and between_bytes_timeout

Timeouts for regular API calls

optional

50ms

data_dome_shield.uri_regex

processes only matching URIs

optional

data_dome_shield.uri_regex_exclusion

ignores all matching URIs

optional

exclude static asset

FAQ

How does the module work?

The module should be injected in varnish.vlc before any other logic.

The module checks if the request should be processed by DataDome API on vcl_recv stage by calling data_dome_shield.is_suitable. The function returns true when requested URL has passed REGEX and the request hasn't got restarted (which means it was handled by DataDome). In other cases the function returns false and lets Varnish handle the regular process.

The function data_dome_shield.prepare_request replaces original request to the API request and updates backend to DataDome backend.

The function data_dome_shield.restore_request reverts to original request.

The module doesn't impact the Varnish caching logic because when the request is allowed, the original Varnish process is restored.

Can I activate DataDome on a specific Domain / IP?

If you would like process only specified virtual host or specified path you can do it by change vcl_recv.

For example you can find below the code that enable DataDome module only for requests to path /public on domain test.com or on IP A.B.C.D

sub vcl_recv {
    # check that host is test.com and URL starts with /public
    if ((req.http.host == "test.com" || client.ip == "A.B.C.D") && req.url ~ "^/public") {
      # usual DataDome configuration
      if (data_dome_shield.is_suitable()) {
          data_dome_shield.prepare_request();
          return (pass);
      }
    }
}

Can I get Bot Name, Bot Type and Bot/Human flag in my application?

DataDome module can inject headers in the HTTP Request that can be read by your application.
To active this premium feature, please contact DataDome Support.

X-DataDome-isbot

Is it a bot ?

0 -> Human
1 -> Bot
NA -> detection not activated on this segment

X-DataDome-botname

The Bot name

String

X-DataDome-botfamily

The bot family

good_bot / bad_bot / commercial_bot

How Can I add Bot information in logs?

For example to create a access-log file that contains the request URI and 'is it a bot', the API server response status and response time, you may use varnishncsa likes

sudo varnishncsa -a -w /var/log/varnish/datadome.log -D -P /var/run/varnishncsa_datadome.pid -F '%h %l %u %t "%r" "%{X-DataDome-isbot}i %{VCL_Log:DataDome_status}x %{VCL_Log:DataDome_spent_time}x"'

DataDome_status can have the following values:

DataDome_status

200

Hit was allow

401/403

Hit was blocked

700

Hit doesn't match with the regex and was not analyzed by DataDome

704

API server response hasn't got expected X-DataDomeResponse header

Can I use a customer installation folder?

The source tree is based on autotools to configure the building, and does also have the necessary bits in place to do functional unit tests using the varnishtest tool.

Building requires the Varnish header files and uses pkg-config to find the necessary paths.

Usage:

 ./autogen.sh
 ./configure

If you have installed Varnish to a non-standard directory, call autogen.sh and configure with PKG_CONFIG_PATH pointing to the appropriate path. For example, when varnishd configure was called with - -prefix=$PREFIX, use:

 PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig
 export PKG_CONFIG_PATH

Make targets:

make           # builds the vmod.
make install   # installs your vmod.
make check     # runs the unit tests in ``src/tests/*.vtc``
make distcheck # run check and prepare a tarball of the vmod.

Can I change the setup folder?

By default, the vmod configure script installs the built vmod in the same directory as Varnish, determined via pkg-config(1). The vmod installation directory can be overridden by passing the VMOD_DIR variable to configure.

Other files like man-pages and documentation are installed in the locations determined by configure, which inherits its default - -prefix setting from Varnish.

Can I build a DataDome module package (DEB/RPM)?

This module can be build as rpm or dev package. For example you can do it by command below:

rpmbuild -ba vmod-data_dome_shield.spec
dpkg-buildpackage -us -uc
rpmbuild -ba --define 'VARNISHSRC /home/user/varnish-3.0.7' vmod-data_dome_shield.spec
env DEBIAN_VARNISH_SRC=/home/user/varnish-3.0.5 dpkg-buildpackage -us -uc

Can I activate a debug mode?

  • configure: error: Need varnish.m4 -- see README.rst

    Check if PKG_CONFIG_PATH has been set correctly before calling autogen.sh and configure

  • How to enable debug?

    Rebuild module with - -enable-debug at configure option.

Varnish Setup

DataDome Varnish module detect and protect against bot activity