Blogs

Background

Kafka Streams is a client library for building mission-critical real-time applications and microservices, where the input and/or output data is stored in Kafka clusters. Kafka Streams combines the simplicity of writing and deploying standard Java and Scala applications on the client side with the benefits of Kafka's server-side cluster technology to make these applications highly scalable, elastic, fault-tolerant, distributed, and much more.

Kafka has four core APIs:

  • Behat configuration on staging
  • What is behat.yml
  • How to write a feature
  • How to write a context
  • How to run Behat tests (Manually & Jenkins)
  • Test step execution results
  • Behat Selenium Driver

Behat configuration on staging

Preparation

First of all, check both php and java versions by the following commands.

Overview

All INSERTs, UPDATEs and DELETEs should always be executed against the Master database. These queries affect the data in the database. When these data changes are made, they are automatically replicated to the Slave databases. So the Slave databases contain an near-realtime copy of all the data on the Master. This usually happens within a few seconds but we've seen the Slaves fall behind by several minutes at times. By running SELECTs against the slave, we free up available connections to the Master database meaning that more INSERTs, UPDATEs and DELETEs can be executed.

Fastly is a CDN provider that uses Varnish. By changing the CNAME of a sub domain, it is possible to serve a response from Fastly. If a Varnish cache record exists, this is served to the user, if not, Fastly makes a request to the web server (Rackspace) and saves the result for future requests. This allows us to serve all current Varnish requests through a CDN. The advantage of this is that the pages will be delivered much quickly in different locations around the global.

The Vagrant environment has xdebug installed. In order to get it working, you need to do a few things on your IDE.

PHPStorm

Set up the server:

I'm copying our live Varnish config from w4 to our Varnish config in Vagrant (which until now has been relatively untouched). Our w4 Varnish config has had quite a bit of work done on it, so is very different from the out-of-the-box config.

As part of this work, I'll look to refactor, clean up and add comments to the live Varnish config.

Server setup

Created two new Ubuntu 14 servers at https://mycloud.rackspace.co.uk

Changed root passwords to phoenix10

Rackspace name: api-test-nginx
ip address: 162.13.86.171
root: phoenix10

Rackspace name: apt-test-varnish
ip address: 162.13.86.181
root: phoenix10

Domain setup

Created three new A records on the atdtravel.com domain, all pointing at the Nginx server:

Overview

Varnish is an HTTP accelerator and caching reverse proxy. It sits in front of Apache and used to serve cached content to anonymous users. This provides a significantly faster page response takes the load of the back-end servers.

Here's a way I came up with for logging drush commands. This might be useful for troubleshooting server slowness.

Add the following block of code to the start of the drush command:

# Rangi: Log all drush commands
LOG_STR=`date --rfc-3339=seconds`
LOG_STR="$LOG_STR drush"
PARAMS=""
for a in ${BASH_ARGV[*]} ; do
  PARAMS="$a $PARAMS"
done
LOG_STR="$LOG_STR $PARAMS (`whoami`)"
echo $LOG_STR >>/var/log/drush

This will add lines like this to /var/log/drush:

Types of affiliate links

Affiliate links can take two forms, for example:

  1. http://www.orlando-ticket-deals.co.uk/?a=123456
  2. http://www.orlando-ticket-deals.co.uk/affiliate/123456

The first of these uses a query string (?a=123456) and as we usually strip out query string in Varnish, we need to treat this as a special case (see below).

Introduction

I've never used PHPStorm before so am no expert, but here are some notes that may help those of you who use it get it working with Vagrant and Xdebug. This approach should let you use the Nginx hostnames (eg. atd.dev) rather than the Apache hostnames (eg. atd-d6phx.dev, etc).

Xdebug settings

Xdebug has a config file (typically /etc/php.d/xdebug.ini) on each php environment (eg. each Apache VM).

Important settings are:

Intro

Here's a Selenium script to quickly place five 3DS test orders on ATD staging.

edit the conf

 

sudo vim /etc/php.d/xdebug.ini

 

Conf file

; Enable xdebug extension module

zend_extension=/usr/lib64/php/modules/xdebug.so

xdebug.remote_enable = 1

xdebug.remote_connect_back=1

xdebug.remote_log=/var/log/xdebug.log