Socket Programming

Overview

Today we will begin to talk about how to write applications which pass data across the Internet. Applications send data through the networking layers which we have begun to discuss.

From an application's point of view, the network is hidden behind something called a socket. Sockets connect 1 computer to another computer. If we want to send data, we send it through the socket. If we want to receive data, we do so through the socket. The actual details of how data gets transmitted will be discussed as we go through the layers.

Essentially sockets are the interface and the network stack is the implementation of that interface.

Today we will talk about how to use the socket interface from within Python programs.

The Client/Server Model

It takes two parties to communicate. In network programming, this means that there are always two sockets involved in communication. The model for sockets is based around the idea of having a client and a server.

The exact responsibilities of a client and a server differ by application. In general though, the server creates a socket first and waits for a connection. The client then connects a socket to the server, and communication goes from there. When finished, the client disconnects, and the server waits for another connection.

Socket communication is bi-directional. The client and server can both send data to each other. However if both parties are waiting for data from the other, nothing will get accomplished - this is the importance of having a protocol and sticking to it!

There are a few caveats to the above:

The same computer can be both the client and the server. Some applications use sockets to pass data between two different processes, and it's common to run both a server and client on the same computer during testing.
There are peer-to-peer network systems, such as BitTorrent that don't follow a client-server model. However, one peer still makes a socket first for the other to connect to, and they each follow the general steps given here.

The client/server socket programming model was first developed in C with Berkeley Unix (which became BSD) in the 80s. The same interface has been ported to a number of other languages and systems. Learning sockets in Python will easily translate to these other systems.

Creating a Client Socket

In order to begin, we will need to import the socket module:


import socket

We then create a socket object with the socket.socket function. This function takes two required arguments:

family - this specifies how addresses are formed. For our purposes, we will pass socket.AF_INET for this parameter which specifies that we will use IPv4 addresses.
type - this specifies the way that the packets will be transported. We will normally pass socket.SOCK_STREAM for this parameter which specifies TCP. We could also pass socket.SOCK_DGRAM which specifies UDP.

So to make a socket called sock, we could do the following:


sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

Now we have a socket. The next step is to connect it to a server. To do this, we need to know the address and the port. The address can be either a domain address, or an IP address. The domain or IP address identifies which machine we want to talk to.

The port identifies which application on that machine we want to talk to. One machine might have an SSH server, web server, email server, and more. The port allows these separate applications to identify their messages and keep them straight.

For instance, let's say we want to connect to the Google mail server, we might use the following connect call:


sock.connect(("smtp.gmail.com", 587))

Notice that there are two pairs of parenthesis in the parameter list. That's because connect takes a Python tuple as its argument, instead of two separate values.

Now the client has connected and can send or receive data! Next we will talk about how the server socket is set up.

What the Server Does

Setting up a socket from the server's point of view is just a bit more complicated. It starts out the same with the creation of a socket. We will again create an IPv4, TCP socket most of the time:


sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

What comes next is different, the server doesn't connect to a client. Instead it sets itself up on a specific address and port. This is done with a call to bind. Like connect, this function takes a tuple containing an address and a port. The operating system will try to reserve the address/port combination for our server application.

One common issue is that the OS often will not recycle which ports are considered used. So if we stop and start our server a lot, it may not be able to reserve the same port again. To fix this, we can tell the socket to reuse its address if needed. This is done with the following code:


sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

We can then call bind itself, passing the IP address of the server and the port we want to use. When developing a server and client on the same machine, you can use "localhost" or "127.0.0.1" for the address. When using a Google cloud VM, you should use the internal IP instead of the external one. That's because bind is a request to the OS dealing with its own network. The internal IP address identifies which network interface the OS should reserve.

For instance, to try to reserve port 2586 on a Google cloud VM, we could use the following:


sock.bind(("10.142.0.3", 2586))

Port numbers can range from 0 to 65,536. Ports from 0 to 1023, are system ports and are reserved for system use. You will not be able to bind those ports unless you run your application as root. Ports after that can be bound to user applications. Wikipedia maintains a list of common ports along with what applications use them.

Generally if you are writing a server, you can pick a port number over 1024 that's not already being used and you will be OK.

After binding a port, the server must then call listen so that it will receive connections on that port:


sock.listen()

Next, we need to call accept. This function will block until a client connects to us. This function returns a tuple with two things: a connection object and the address of the client:


conn, addr = sock.accept()

Now we can use conn to send data to, or receive data from, this particular client. Normally, a server is capable of dealing with multiple clients at once, but we will deal with that later.

Data Formats

Before sending data across our sockets, we need to talk about what sort of data to send. Networks are independent of any particular programming language or platform.

This is nice in that we can write clients and servers in whatever language we please. For instance, an email server can be written in C, but clients that connect to it can be Python, Java etc.

One downside of this is that data is sent across a network in "raw" bytes which doesn't necessarily correspond to the built-in objects of any language.

In Python what this means is that we can't just send a string, or int or any other sort of Python object across the network! Instead we have to convert it to raw bytes.

Say we have a string variable in Python called username and wish to send it to a server. We have to first convert it to bytes by calling the encode method on it:


raw_username = username.encode("utf-8")

We pass to it the way that we want the string to be encoded as bytes. There are many possible encodings, but "utf-8" is a safe option since it is widely supported and allows Unicode characters to be encoded. It is actually the default parameter, so we can leave the parameter off and get UTF-8:


raw_username = username.encode()

The point of this is to get straight bytes out of a string which may have multi-byte characters in it. For instance, if we happen to have a taco emoji in our string, it will encode it into a 4 byte sequence:


>>> "I am hungry for 🌮".encode()
b'I am hungry for \xf0\x9f\x8c\xae'

We will then send the encoded bytes across the network instead of the Python string object. Of course this will only make sense if the receiver is expecting data encoded this way! When receiving data, we can get it back into a string by calling decode. Like encode, this takes an encoding scheme, or will use UTF-8 by default:


>>> b'I am hungry for \xf0\x9f\x8c\xae'.decode()
'I am hungry for 🌮'

Note that the result of the encode function is a string beginning with a lowercase b (which stands for bytes). If we stick to straight ASCII values (meaning English letters, numbers and common punctuation), then we can get raw byte strings just by putting a b in front:


raw_username = b"ifinlay"

To send other data like numbers, it's probably simplest to simply convert to strings first. For instance, we can convert an integer to a string, and the encode it as:


value = 42
raw_value = str(value).encode()

There are other, somewhat more efficient, ways to encode other data, but this way is easy and will work well for our purposes!

Sending Data

Once we have encoded byte data to send, we can do so using the sendall method. For the client, this is called on the socket which we connected. For the server, we call it on the connection object which accept returns:


# client
sock.sendall(b"to the server")


# server
conn.sendall(b"to the client")

There is also a send function which we can use instead of sendall. The difference is that send will take our data and start send some of the bytes in one packet. It will return to us the number of bytes it sent. We would then have to keep calling it until it's all sent. sendall will keep sending packets until all of the data is sent.

Receiving Data

We can receive data from a socket using the recv method of a socket. This function takes one mandatory argument which is the maximum number of bytes to read from the socket. recv will never return more bytes than this limit. It is important to plan a protocol with this in mind.

Like sendall this can be called on a client socket, or a server's connection object:


# client
message = sock.recv(1024)


# server
message = conn.recv(1024)

Remember these messages will be encoded bytes. If we want to get a Python string back out of them, we will need to call decode.

Closing a Socket

Just like files, it's good practice to close a socket when you are done with it. This is done with the close method:


# client
sock.close()


# server
conn.close()

Below is a diagram of the way that the function calls flow on the client and the server. The call to setsockopt is optional. The actual sequence of sendall and recv calls will depend on the protocol of the particular application.

The flow of function calls on the server & client

Example: Capitalization

As a complete, but very simple example, consider the following "hello world" for client/server programs. The server accepts one client. It reads some bytes from the client, converts it to a string, capitalizes it, and then sends it back. The client sends one string and prints the server's response.

Here is the server code:


#!/usr/bin/python3

import socket

# host (internal) IP address and port
HOST = "10.142.0.3"
PORT = 5220

# create our socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# allow us to reuse an address for restarts
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)

# set the socket host and port number up
sock.bind((HOST, PORT))

# listen for any clients connecting
sock.listen()

# wait for a client to connect to us
# accept a connection which has come through
conn, addr = sock.accept()
print("Connection from:", addr)

# read some bytes from the client
data = conn.recv(1024)

# decode it into a string
string = data.decode()

# convert it to uppercase
string = string.upper()

# now encode the data for sending back
data = string.encode()

# send it back
conn.sendall(data)

# and done
conn.close()

# done with listening on our socket to
sock.close()

And here is the client code:


#!/usr/bin/python3

import socket

# the host we are connecting to and the port
HOST = "34.73.15.22"
PORT = 5220

# create our socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# connect the socket to the server
sock.connect((HOST, PORT))

# get a string from the user
mesg = input("Enter a string: ")

# convert it to raw bytes
data = mesg.encode()

# send it to the server
sock.sendall(data)

# read the response
data = sock.recv(1024)

# convert it to a string
mesg = data.decode()

# print it out
print(mesg)

# and close the socket
sock.close()

Note that to run these programs yourself, you would need to adjust the HOST variable to reflect the actual addresses of the machine(s) you are running on. Also, by default firewalls will normally disallow traffic on user ports like 5220. We will talk about how to open ports on Google cloud later.

Conclusion

Sockets provide a software interface to the network. They allow user programs to send data back and forth between different machines.

The interface is largely the same from one programming language to the next. Here we saw how this interface works in the case of Python.

As we go, we will use the socket interface to build protocols and create applications.