# Average, variance & standard deviation in Python

Here is a quick python script which calculates average, variance and standard deviation. This is also part of codeacademy work. I must say last time I worked with variance and standard deviation it was more than 10 years ago in statistics course. Hence a bit of reminder here for me too: (Some are from wikipedia and mathsisfun.com)

Average: In mathematics and statistics, it is called arithmetic mean according to wikipedia but simply it is the sum of a list of numbers divided by the number of numbers in the list.

Variance: The average of the squared differences from the mean. Here is the formula which we will use in our python code. $\displaystyle \sigma^2 = \frac{\displaystyle\sum_{i=1}^{n}(x_i - \mu)^2} {n}$

Standard deviation: Square root of the variance is the standard deviation which just means how far we are from the normal(mean) $\displaystyle \sigma = \sqrt{\frac{\displaystyle\sum_{i=1}^{n}(x_i - \mu)^2} {n}}$

Now here is the code which calculates given the number of scores of students we calculate the average,variance and standard deviation.

# a simple battleship python script

I have written a simple python game script to sink a battleship. It is very simple as it was almost a year ago last time when I wrote a long python script. Thanks to codeacademy that it helps to brush up my python skills. I think it is also good to put the script here for my documentation. Simply, you guess the coordinates of the point and by default you have maximum 4 turns to do it right. Each time a guess is run, coordinate is marked by “X”. Below is a sample execution:

Here is the code itself:

# Fragmented IP packet forwarding

I couldn’t really find a suitable topic for this post actually but I will try to find answers for the following questions:

• How can we fragment an IP packet manually in scapy
• How does a fragmented packet look like and how the transport layer (TCP/UDP) header is located
• How do we forward fragmented packets, do we reassemle them?
• If we don’t reassemble, can we force reassembly?

First of all a bit of a theory: if an incoming IP packet is to be forwarded to another next hop and the MTU of this new path is smaller than the packet to be transmitted, we must find a way to forward the packet. If the packet has DF (Don’t Fragment) bit on i.e we are instructed not to fragment the packet most probably by the source, then normally we are expected to send an ICMP packet with type “Fragmentation needed” and pray that on the way back to the source no devices block all ICMP type of traffic. Second scenario is that what if the source lets us fragment the packet. Then we need to fragment it and story from now on is about this part of the scenario and the topology we will use is something like below.

# Port Scanner in Python

Python is a great tool to do some socket operations. I have written a piece of code by which I can scan a port range.
It is very basic and missing bunch of checks as aim is the simplicity here.

You can run the script in the following way by which you scan ports between 1 and 1024:

# How to sort a python dictionary by value

If you want to search a dictionary by value in python, here is a handy one line using also lambda:

If you run this script output will be ;

# searching dictionary by value in python

The following list comprehension is a life saver for me to search a dictionary by value. Searching dictionary by value may not seem to be sensible as you may get multiple values but in case needed, it is a wonderful one line of code.

Here we take the ‘value’, iterate through the mydict dictionary and if any dictionary member is found having this value, then we grab the key value and assign them to the list ‘list_values’ . It may return for example 10 keys all of which can be accessed via the list assigned. It is a spectacular structure and I wanted to take a note here to reference later:)

# How to turn a list into string

Here is a quote from python.org describing how we can use the join method.

str.join(iterable)

As it can be seen it isn’t only for strings but for something called “iterable”. Again when we look at python.org what this iterable means;

iterable

So it means we can use list, tuples all sequence types. This means I can use it to turn a list into a string which is very handy when I read files via python. For example;

Bingo, we have a string out of a list!

# Python list comprehensions

I don’t know if you have used list comprehension in Python. It is a great feature. You can go through a list and do some small changes with very little effort. Here is an example;

For example you have a list of user names and you would like to shorten these user names to 8 characters.

As you know you can iterate a python list with a construct like;

If you want to do some operations on this returned value (user), you can just put an expression in front of this for loop and remove the colon like;

The output from the code above will be ;

It looks fine. What about the user names laving less than 8 characters. Assume you want to fetch those which have at least 8 characters or more and ignore the others! It is also possible with a one liner. If you put an if condition right next to our for loop, magic happens;

The new output will be like;

To be honest, although list comprehension is quite nice, to me it looks a bit perlish way. Because when you look at the statement it looks like a complex one liner but if you can break it down to pieces like “operation : for expression : if statement” it is really easy!

# How to preserve quoted strings in python split

If you want to analyse for example apache log files and split the lines by space by using the usual “split” method, you will see that split doesn’t respect quoted strings. For example if you have a line like below;

192.168.2.1 – – [06/Mar/2012:10:02:22 +0100] “GET /2011/10/19/jncip-sec-exam/ HTTP/1.1” 200 3331 “-” “mm”

You can’t get the HTTP_REQUEST easily with split. There is a very nice module named
shlex which allows you to split strings by space and treats quoted strings as single columns. Below is an example of my code which shows how you can fetch HTTP REQUEST from an apache log.

# Finding the frequency of words in a file with python

If you want to find how many times a single word is repeated in a file, I have quoted my code below. The method that I didn’t know before was the “get” method. It allows you to get the value of the key but if it isn’t set before, set the value specified in our example 0. This allows us to count the occurrence indeed.