Causal inference from dynamic systems upon events of low probability

The Old man sat on a log,
sipping from the jug, his daily quota of grog.

Down by the log,
sat the old man's dog.

The old man's dog followed a frog and pushed him into the old man's grog.

The old man's grog now had a a frog and a sly smile stuck to the old man's dog.

JMeter

took a while to figure out how to get around to using another one of the immensely simple yet powerful arrows from Apache's amply filled quiver.

The biggest problem with working with Jmeter was that I could not get a head-start. For the uninitiated, it is a load-testing platform developed by Apache.

To start with, ensure you download the latest source code. Fooling around with the one installed with apt-get ensures hours of frustration with no results. The results from internet searches are deceptively informative. Next, well we would prefer having our very own sampler. Why?
 //need to answer this one bro!!

Here we go. First, a little peek into the application I was trying to clobber with requests. A platform to book cabs called pyngcabs was the project. The cab booking was to be done using different clients, to start with, android.
 The Server side code was handled by a team of four writing in Spring and using the akka framework for asynchronously handling calls. This would facilitate a behaviour wherein the cab driver would be given the liberty to choose between cab requests he is given. The flow is as follows :

1. The customer requests a cab.
2. Filtered cabs using google maps API are shortlisted with assistance from the customer's GPS.
3. The cabs are sent the requests.
4. The cab drivers approve or reject requests. 
5. The customers are notified of the cab that wishes to accept.

When performing a single test case, the main difference between android and desktop client was the ability to use the awesomely-cool server-to-client notification mechanism using GCM, which although not documented to be guaranteed to receive proved to be nearly as fast and as reliable also without the need for a long living TCP connection with heartbeat messages or other similar mechanism adopted by most notification softwares which can be plugged in with android.

Back to jmeter, so what I wanted was a mechanism to perform requests from a customer, accept/reject one of the requests and notify the same to the customer. A polling based approach which was considered a second-line of defence approach if the GCM message does not arrive till a certain time period,was the path to go with jmeter. So, first we have a HTTP POST where the customer puts in a request. This request was being done using a JSON Object. Hang on!! Load testing would mean multiple customers, multiple users, how does that work? The first thing to notice/ use is a thread group. A thread group allows running a set of threads which would perform similar operations. So, I make two thread groups, one for the customers and one for the cabs. It can be found in add--> threads --> thread groups.

The next thing to do was to add the payload type of the HTTP request we would be sending. In my case, it was a json object and I added the same.A picture, to reduce my words:

Next, to reduce redundancy of our configurations, we add common info like server address, port number and other common variables in the test plan:

//Saving the post
 Next comes the customer thread group. First, I send a HTTP request to the server that I need a cab. For each customer thread that I run, I need to send a unique customer id in the object. Here I made a little script to generate sql INSERT files with serially ordered customer and cab ids and an extension to the same script to create a comma-separated-values file, popularly called csv.

To build a HTTP request, I need to make a JSON object with a value that I fetch from the csv file. For this, we use two things, a CSV data config to read from the csv files and a beanshell pre-processor to generate the JSON object which will be the payload. First, the CSV data config:

The variable names is where we write the name which we will later use in the beanshell pre-processor. In my case, I chose the name custid which is only one var per line. If there are more than one, we need to write them all separated by commas(obviously,it is a csv).Next, we set the delimited as',' without the quotes and the sharing mode as all threads. What is really smart with jmeter is, we never told jmeter about iteration, the thread group takes care of it!! Viola! this simple GUI handles loading different user IDs. Here is the pre-processor for the payload generation:

Do note above that the CSV data config and the beanshell pre-processor are children of the http request. In the beanshell pre-processor, we use the command vars.get("custid") to get the variable custid which we set in the csv config file. We use this and other variables which I have hardcoded for convenience. Once the JSON object has been generated, we need to put it back into the variables. We do it with the command : vars.put("custRequestJSON",toSend.toString()); where the variable name is the part in the double quotes. This variable will be sent with the HTTP request. //saving the post 2 Here is a screenshot of my HTTP request:

I use the var names we set in the test plan and we send a payload as a variable with no name and the value as ${custRequestJSON} which was set in the pre-processor. Now, we are done with sending the request. We follow this with parsing the reply. The reply, in my case has a tripId which I use to query the server to ask whether the trip is scheduled. First the post-processor, to get the tripId:

Next comes polling the server with the trip id which has been put into the variable tripId. For polling, I have used a while controller which keeps running the request till it gets an object where the value for state of the trip is scheduled. I also use a constant timer to space requests so that I do not just keep polling non-stop. I would prefer a near-real scenario wherein I poll somewhere between ~3-5 seconds. Below is the while controller, where I check the value of a variable 'confirmed' and run if not 1( which I set as 0 in the post processor).

Inside this while controller, is another HTTP request, this time a HTTP GET to check the state of the trip and a post-processor to process the state of the trip. Below is the beanshell post-processor part to check the state of the resulting payload.

So, that is half the story. A similar thread group for the cabs was made, where each thread uses a HTTP GET to check for new requests, use a post-processor to randomly accept one of the results and then sends a HTTP PUT with that request id. This is accomplished by an if controller which checks the post-processor. Do let me know if I can help with any queries.

Dudhsagar

Packed my bags again within a month, this time off towards Dudhsagar falls, I will fill in the specifics by the weekend,

Borrowing a bag from Shivangi/Piyush's place as I did not want to carry two, I re-packed my stuff into the huge backpack and was off to the Kachiguda station to ensure my arrival before the train leaves unlike before. A quick dinner later, I was waiting on the right platform and soon found co-trip people and was delighted to know that all the tickets were booked together. Boarding on the train, I found Mr.Munni who delighted me with this interesting job profile before we called it a day. The next morning was a looong one with the train barely chugging away but the fresh and green view outside flanked by foggy hills played an ideal part in keeping the moods alive. A little burst of droplets through the window cheered us up as everyone(21?) finally pushed into a single coupe for a round of introductions before we got down finally. Some tea and group photographs later, off we went on our way, right behind the train from castlerock to stop not till we saw the falls which could be loosely translated as an ocean of milk. Wary of approaching trains from both directions on the single track, specially after out organizers Srini and Jamila warned us of hugging the tunnel walls to prevent accidents when trains pass, we kept our pace on slowly enjoying the scenery but wary of gravel and the 'unclean' areas. The tunnels kept coming, one 400 meters long, with a blind turn ensured complete darkness in the day if not for our torch lights. Light drizzles kept the temperature in control with the rising levels of humidity. It seemed like forever although we had alighted at 1:15 pm and it would have been about 3:30 when we decided to freshen up at a small stream of water followed up by Muni garu's extra lunch which fueled all three of us(Srini was the third) for the rest of the trek along the rails. As we reached the last leg, the scenery finally began to open up with trees widening enough to clear the view. Walking between a rail track and a good long drop, with no thick vegetation, occasional monkeys and amazing greenery the walk slowly started getting more and more fun. Little bits of gossip gathered by us about more than 300 people trying to stay back at Dudhsagar made us keep our fingers crossed when we finally reached the station we were to come to. Around 6pm, we crossed the Dudhsagar station arriving to a loud noise which felt like a very heavy train moving and taking forever to go ahead. Walking ahead and there was the magical display of milky water flowing in streams from above with generous force and going on with a display certain to leave the viewer spellbound. Finding one of the sheds empty, we made ourselves at home outside leaving the dirty interiors and deciding to camp in the open for the night. A while later the rest of the crew arrived flanked by a visibly angry Jamila who appeared to have a hard time controlling the ecstatic photographers from stopping to click everywhere. All set up and a great packed lunch experience thanks to everyone who got something, we got ready for a long night of little or no sleep for many. I managed an hour or two after which I finally gave up on sleep and walked out of the tarp to stare in the darkness to the unending deafening sound which resembled the sound from a television when you lose cable connection. An amazing bath in the icy cold water later we trekked up another kilometer for an amazing top to toe view of the Dudhsagar beauty after which we were back near the dudhsagar station waiting to get down at Loonda for the next train to take us back to our non-linearly noisy workplaces. I, for one, carried back enough white noise in my ears to last for a long time and amazing company which would hopefully cross ways another time soon.