Today we will begin to talk about how to write applications which pass data across the Internet. Applications send data through the networking layers which we have begun to discuss.
From an application's point of view, the network is hidden behind something called a socket. Sockets connect 1 computer to another computer. If we want to send data, we send it through the socket. If we want to receive data, we do so through the socket. The actual details of how data gets transmitted will be discussed as we go through the layers.
Essentially sockets are the interface and the network stack is the implementation of that interface.
Today we will talk about how to use the socket interface from within Python programs.
It takes two parties to communicate. In network programming, this means that there are always two sockets involved in communication. The model for sockets is based around the idea of having a client and a server.
The exact responsibilities of a client and a server differ by application. In general though, the server creates a socket first and waits for a connection. The client then connects a socket to the server, and communication goes from there. When finished, the client disconnects, and the server waits for another connection.
Socket communication is bi-directional. The client and server can both send data to each other. However if both parties are waiting for data from the other, nothing will get accomplished - this is the importance of having a protocol and sticking to it!
There are a few caveats to the above:
The client/server socket programming model was first developed in C with Berkeley Unix (which became BSD) in the 80s. The same interface has been ported to a number of other languages and systems. Learning sockets in Python will easily translate to these other systems.
In order to begin, we will need to import the
We then create a socket object with the
This function takes two required arguments:
socket.AF_INETfor this parameter which specifies that we will use IPv4 addresses.
socket.SOCK_STREAMfor this parameter which specifies TCP. We could also pass
socket.SOCK_DGRAMwhich specifies UDP.
So to make a socket called sock, we could do the following:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Now we have a socket. The next step is to connect it to a server. To do this, we need to know the address and the port. The address can be either a domain address, or an IP address. The domain or IP address identifies which machine we want to talk to.
The port identifies which application on that machine we want to talk to. One machine might have an SSH server, web server, email server, and more. The port allows these separate applications to identify their messages and keep them straight.
For instance, let's say we want to connect to the Google mail server, we might use the following connect call:
Notice that there are two pairs of parenthesis in the parameter list. That's because connect takes a Python tuple as its argument, instead of two separate values.
Now the client has connected and can send or receive data! Next we will talk about how the server socket is set up.
Setting up a socket from the server's point of view is just a bit more complicated. It starts out the same with the creation of a socket. We will again create an IPv4, TCP socket most of the time:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
What comes next is different, the server doesn't connect to a client.
Instead it sets itself up on a specific address and port. This is done with a
bind. Like connect, this function takes a tuple
containing an address and a port. The operating system will try to reserve the
address/port combination for our server application.
One common issue is that the OS often will not recycle which ports are considered used. So if we stop and start our server a lot, it may not be able to reserve the same port again. To fix this, we can tell the socket to reuse its address if needed. This is done with the following code:
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
We can then call
bind itself, passing the IP address of the
server and the port we want to use. When developing a server and client on
the same machine, you can use "localhost" or "127.0.0.1" for the address.
When using a Google cloud VM, you should use the internal IP instead
of the external one. That's because bind is a request to the
OS dealing with its own network. The internal IP address identifies which
network interface the OS should reserve.
For instance, to try to reserve port 2586 on a Google cloud VM, we could use the following:
Port numbers can range from 0 to 65,536. Ports from 0 to 1023, are system ports and are reserved for system use. You will not be able to bind those ports unless you run your application as root. Ports after that can be bound to user applications. Wikipedia maintains a list of common ports along with what applications use them.
Generally if you are writing a server, you can pick a port number over 1024 that's not already being used and you will be OK.
After binding a port, the server must then call
that it will receive connections on that port:
Next, we need to call
accept. This function will block
until a client connects to us. This function returns a tuple with two
things: a connection object and the address of the client:
conn, addr = sock.accept()
Now we can use
conn to send data to, or receive data from,
this particular client. Normally, a server is capable of dealing with
multiple clients at once, but we will deal with that later.
Before sending data across our sockets, we need to talk about what sort of data to send. Networks are independent of any particular programming language or platform.
This is nice in that we can write clients and servers in whatever language we please. For instance, an email server can be written in C, but clients that connect to it can be Python, Java etc.
One downside of this is that data is sent across a network in "raw" bytes which doesn't necessarily correspond to the built-in objects of any language.
In Python what this means is that we can't just send a string, or int or any other sort of Python object across the network! Instead we have to convert it to raw bytes.
Say we have a string variable in Python called
wish to send it to a server. We have to first convert it to bytes by calling
encode method on it:
raw_username = username.encode("utf-8")
We pass to it the way that we want the string to be encoded as bytes. There are many possible encodings, but "utf-8" is a safe option since it is widely supported and allows Unicode characters to be encoded. It is actually the default parameter, so we can leave the parameter off and get UTF-8:
raw_username = username.encode()
The point of this is to get straight bytes out of a string which may have multi-byte characters in it. For instance, if we happen to have a taco emoji in our string, it will encode it into a 4 byte sequence:
>>> "I am hungry for 🌮".encode() b'I am hungry for \xf0\x9f\x8c\xae'
We will then send the encoded bytes across the network instead of the Python
string object. Of course this will only make sense if the receiver is
expecting data encoded this way! When receiving data, we can get it back into
a string by calling
encode, this takes
an encoding scheme, or will use UTF-8 by default:
>>> b'I am hungry for \xf0\x9f\x8c\xae'.decode() 'I am hungry for 🌮'
Note that the result of the
encode function is a string beginning
with a lowercase b (which stands for bytes). If we stick to straight ASCII
values (meaning English letters, numbers and common punctuation), then we
can get raw byte strings just by putting a b in front:
raw_username = b"ifinlay"
To send other data like numbers, it's probably simplest to simply convert to strings first. For instance, we can convert an integer to a string, and the encode it as:
value = 42 raw_value = str(value).encode()
There are other, somewhat more efficient, ways to encode other data, but this way is easy and will work well for our purposes!
Once we have encoded byte data to send, we can do so using the
method. For the client, this is called on the socket which we connected. For the
server, we call it on the connection object which
# client sock.sendall(b"to the server")
# server conn.sendall(b"to the client")
There is also a
send function which we can use instead of
sendall. The difference is that send will take our data and start
send some of the bytes in one packet. It will return to us the number of bytes
it sent. We would then have to keep calling it until it's all sent.
sendall will keep sending packets until all of the data is
We can receive data from a socket using the
recv method of a
socket. This function takes one mandatory argument which is the maximum number
of bytes to read from the socket.
recv will never return more
bytes than this limit. It is important to plan a protocol with this in
sendall this can be called on a client socket, or a server's
# client message = sock.recv(1024)
# server message = conn.recv(1024)
Remember these messages will be encoded bytes. If we want to get a Python
string back out of them, we will need to call
Just like files, it's good practice to close a socket when you are done with it.
This is done with the
# client sock.close()
# server conn.close()
Below is a diagram of the way that the function calls flow
on the client and the server. The call to
is optional. The actual sequence of
recv calls will depend on the protocol of the particular
The flow of function calls on the server & client
As a complete, but very simple example, consider the following "hello world" for client/server programs. The server accepts one client. It reads some bytes from the client, converts it to a string, capitalizes it, and then sends it back. The client sends one string and prints the server's response.
Here is the server code:
#!/usr/bin/python3 import socket # host (internal) IP address and port HOST = "10.142.0.3" PORT = 5220 # create our socket sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # allow us to reuse an address for restarts sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) # set the socket host and port number up sock.bind((HOST, PORT)) # listen for any clients connecting sock.listen() # wait for a client to connect to us # accept a connection which has come through conn, addr = sock.accept() print("Connection from:", addr) # read some bytes from the client data = conn.recv(1024) # decode it into a string string = data.decode() # convert it to uppercase string = string.upper() # now encode the data for sending back data = string.encode() # send it back conn.sendall(data) # and done conn.close() # done with listening on our socket to sock.close()
And here is the client code:
#!/usr/bin/python3 import socket # the host we are connecting to and the port HOST = "18.104.22.168" PORT = 5220 # create our socket sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # connect the socket to the server sock.connect((HOST, PORT)) # get a string from the user mesg = input("Enter a string: ") # convert it to raw bytes data = mesg.encode() # send it to the server sock.sendall(data) # read the response data = sock.recv(1024) # convert it to a string mesg = data.decode() # print it out print(mesg) # and close the socket sock.close()
Note that to run these programs yourself, you would need to adjust the
HOST variable to reflect the actual addresses of the machine(s)
you are running on. Also, by default firewalls will normally disallow traffic
on user ports like 5220. We will talk about how to open ports on Google cloud
Sockets provide a software interface to the network. They allow user programs to send data back and forth between different machines.
The interface is largely the same from one programming language to the next. Here we saw how this interface works in the case of Python.
As we go, we will use the socket interface to build protocols and create applications.
Copyright © 2018 Ian Finlayson | Licensed under a Creative Commons Attribution 4.0 International License.