Protocol Design Document Due: Monday, Nov 3rd, 11:59 PM

Due: Monday, Nov 17th, 11:59 PM

1. Overview

For this lab, you will be designing your very own music-streaming application! Since the protocol is your own, this time, you will be developing the functionality of both the client and the server.The protocol you design will support a few basic operations that the user can request from a client, including retrieving a list of songs, getting information about a song, playing a song, and stopping playback.

1.1. Lab 4 Goals

  • Design and implement your own application-layer client-server Jukebox protocol.

  • Develop both the client and server protocol communication formats.

  • Develop a persistent server to connect with multiple interactive clients using event-based concurrency.

  • Write a client that uses producer-consumer threading model to receive song data, and play song data.

1.2. Weekly Deliverables

Week 1

By the end of the first week, you should have written down:

  • A specification for your protocol, including header formats, field sizes, and anything else someone might need to know to implement your protocol.

  • A list of the states (or actions) you intend to perform as the server and what each state represents.

These documents don’t have to be super long or formatted like RFCs, but they should provide enough detail for you to refer to as you work on your implementation. They also aren’t set in stone — you may have to change the design slightly as you encounter implementation challenges, but my expectation is that that they won’t change too drastically.

I would like to have a brief (15-minute) protocol specification and progress meeting with each group before labs start before or on November 4. Please sign up for a time. For this meeting, you should come prepared to discuss the details of your protocol and the per-client state you intend to keep at the server.

Week 2

By the end of the second week, you should:

  • Be able to accept clients at the server.

  • Be able to exchange text (list and info) requests and replies between multiple concurrent clients and the server.

  • Be able to exchange music with one client, initially, and then multiple clients.

Week 3

  • Be able to interleave music and text inputs.

1.3. Handy references

2. Requirements

Jukebox Protocol
Figure 1. The figure shows a client-server architecture, with multiple clients connected to a server. Both clients and server "speak" your jukebox protocol. Clients can make the following requests of the server: list, info <num>, play <num>, stop and exit. The server is single-threaded and maintains persistent connections with each of the incoming clients (until a client calls exit). The server keeps a list of song files, song info and client state associated with connected clients. In the figure, clients (1-4) are requesting "info 3", "play 1", "stop" and "list" respectively. The server is shown to keep this client state list as a table. Row 1 has client 1; info 3, Row 2 has client 2; play song 1, and so on.
#run your server on a different machine from the client
$ ssh lab_machine_1
$ ./server 5000 /home/rware/music/
$ #Song Listing#
# Display on the server side:
New connection (4)!
Client 4 request file list.
Client 4 disconnected:
  recv: Success
New connection (5)!
Client 5 now playing Cowboy Junkies - Sweet Jane.mp3
Client 5 requested stop
Ctrl+C # kill the server

# separately ssh into a client. PLEASE BE AT THIS LAB COMPUTER IF YOU PLAN TO   # PLAY SONG DATA
$ ssh lab_machine_2
$ python client.py lab_machine_1.cs.swarthmore.edu 5000
$ >> list  #enter command here
$ >> exit
$ python client.py lab_machine_1.cs.swarthmore.edu 5000
$ >>
$ >> play 1
$ >> stop

2.1. Protocol Design & Server State

Thus far, we’ve seen a few different types of application-layer protocols that follow the client-server architecture. Having discussed various protocol design trade-offs, consider at least the following when designing your protocol:

  • Header information: what information must be in the header of each message to ensure correctness?

  • Field format and size: for each piece of information that you exchange in a message, how large will it be and how will the receiver know the size? How will it be formatted (e.g., text vs. binary, byte order, etc.)

  • Message delimiters: how will each side of the communication recognize where one message ends and the next one begins?

  • Per-client state: what does the server need to maintain to keep track of the state of a client? What variables does it need to store and what do they mean?

Common Gotchas

Students often neglect to consider these cases:

  • What happens when a user enters stop at a client? If it immediately stops playing any music data it has buffered, what should happen if new data continues to arrive afterward?

  • Does your protocol allow for interleaving of messages? For example, if the user asks to play a large song file and then enters list while the file is still transferring, how does your protocol reply with the list response without the user needing to wait for the entire file transfer to complete?

  • When keeping track of the state of a client at the server, account for the case when you call send and fewer than len bytes were sent (that is, the return value of send indicated that some, but not all of the bytes got sent). The next time you send to the same client, you need to resume sending the old message where you left off rather than generating a new one. What sort of variables do you need to keep track of this case?

2.2. Server Requirements: Event-Driven Concurrency

$ ./server [port] /home/rware/music/
Found 15 songs.
Thread-based vs. Event-based Concurrency

There is a continued debate about thread vs. event-based programming in the systems community. To process a single client request, a server is mostly doing I/O rather than using CPU cycles. To keep the server busy (i.e., improve performance), we want to handle as many simultaneous client requests as possible. Fundamentally, the debate is about performance and scalability - with ease of programming (i.e., synchronization with threads harder to reason than single-threaded event-based programming).

  • You must use C to implement your server, and it must use select() (rather than threading) to support multiple concurrent clients.

  • To simplify the file I/O, it’s fine for the server to keep its data, including the audio files, in memory, but the client should not store data that it isn’t actively using.

  • Your server will receive two command line arguments: a port number on which to listen for incoming connections and the name of a directory with music files. For example:

  • See reference material: Event-based programming and select()

2.2.1. Workflow

  1. Start up and read .mp3 and .info file data into in-memory storage (e.g., a global array). You may also want to build a string to handle list messages, since that won’t change.

  2. Begin listening for incoming client connections.

  3. Infinitely execute the main event loop, which uses the select function to decide:

    1. Is there a new client that is connected for us to accept? If so, accept the new socket and initialize the state associated with that new client. (This is true when select says that the server socket is ready for reading.)

    2. Have any clients sent data to the server? If so, for each client, receive the data and update the client’s associated state to reflect what they’ve just requested.

    3. Do any clients that the server would like to send a message to (i.e., the state of the client indicates that it has messages pending) have space available in the outgoing socket buffer? If so, for each client, look at the state of the client and send the appropriate data to it.

When sending song data, it’s helpful to send relatively large chunks at a time (e.g., 16 or 32 kilobytes). In my solution, I initially sent 4-kilobyte chunks, and song playback stuttered at the start. Switching to 16-kilobyte chunks seems to work much better.

2.2.2. Expectations

  • After it starts up and begins serving clients, your server should never block on any call other than select. Blocking on a send, recv, or accept is a bug!

  • If a client disconnects from your server, for any reason, your server should not crash. Instead, it should reset the disconnected client’s such that it will ignore it until some other client comes along that happens to use the same socket descriptor number. You might detect a client leaving in two ways:

    1. When a client disconnects, if you’re blocked on a call to select, the select call will return and the disconnected client’s socket will show up as readable. Calling recv will return 0 to indicate "end-of-file". If you see a return value of 0 from a client socket, that client is gone.

    2. When you call send on a client socket, if the client has disconnected, your call to send will fail and generate a SIGPIPE signal, which by default will terminate your entire server process. If you pass the MSG_NOSIGNAL flag to send, it will not deliver the signal and instead return -1. If you see a return value of -1 from send, the client is gone.

  • When using a language without garbage collection (e.g., your server in C), you should generate no valgrind warnings. Because the server is still active when you hit CTRL-C, it’s fine for memory to be still reachable, but none should be lost.

2.3. Client

$ ./client.py [hostname].cs.swarthmore.edu [port]
>>

Your client should be interactive, with users typing any of the following commands to request jukebox behavior at any time:

  • list: retrieve a list of the songs available at the server.

  • info <song number>: retrieve text information about the specified song number.

  • play <song number>: retrieve the bytes of the specified song file (in MP3 format) for the client to play the song.

  • stop: end the song data file transfer (if data is being transferred) and stop playing the current song (if one is playing).

  • exit: disconnect the client.

The client should not cache data. In other words, every time a user enters a command, the client needs to send a request to the server for the corresponding data.

2.3.1. Workflow

When you start the client, it will create two additional threads (for a total of three threads):

  • The main thread, which reads user input commands and, when the user enters a command, sends the corresponding request to the server. Its sequence is:

    1. Create the other two threads.

    2. Enter an infinite loop that waits for user input. When the user inputs a command, build a request and send it to the server.

  • The receiving thread, which always tries to receive and process data from the server. Its sequence is (in an infinite loop):

    1. Receive a message header.

    2. Receive a message body.

    3. If the message contains a text response (e.g., in response to list or info), print it.

    4. If the message contains song data, add the new song data to any other song data you’ve already buffered and inform the playing thread that new data is available. Because the song data is shared with the playing thread, you need to protect access to it so that both threads don’t attempt to access it at the same time.

    5. If your protocol specifies other message types, handle those as needed. (You may find it helpful to get stop messages too.)

  • The playing thread, which waits for song data and plays it when available. Its sequence is (in an infinite loop):

    1. Wait for song data to become available. When data arrives, process it and play it.

The receiving thread and playing thread have a producer/consumer relationship. As the receiving thread receives song data, it produces data by appending it to the outstanding data that needs to be played. The playing thread consumes the data by working through from start to end. This relationship means you need to guard against two potential problems using thread synchronization:

  1. Only one of the two threads should access the song data at a time, so you need mutual exclusion.

  2. The player thread needs to worry about data underflow. That is, it shouldn’t attempt to play song data when no song data is available. Thus, you want it to wait on a condition variable that will indicate that new data has arrived. The receiving thread will need to signal the condition variable to wake up the sleeping playing thread every time it adds new data.

In Python, a condition variable implicitly contains a mutex lock. You should create one condition variable and share it with both threads. Threads can call .acquire() and .release() on it to lock or unlock the mutex, respectively.

While holding the lock, a thread can call .wait() on the condition variable to wait for some future event to occur (e.g., data arriving). Calling .wait() implicitly gives up the lock and automatically reacquires the lock upon returning.

While holding the lock, a thread can call .notify() on the condition variable to unblock a thread that is waiting on it (e.g., to let it know that data has arrived). If there is a waiting thread, it will wake up from its call to .wait() as soon as the lock becomes free. If no threads are waiting, then nothing happens.

2.3.2. Expectations

  • The client should not cache data. In other words, every time a client would like to either play/list/info, the item needs to be requested from the server. Each retrieved item should not be stored on the client side and repeated on subsequent requests!

  • A user should be able to request that a song start playing and then immediately (while the song data is still transferring) be able to request text via list or info without having to wait for the file transfer to complete.

  • The protocol you design (and your implementation of it) must allow list and info messages to be interleaved with song data. That is, while the server is sending data in response to a play command, the user should still be able to request the list of songs or information about songs without needing to wait for the entire song file to transfer first.

2.3.3. Sending requests to the server

You can use the Python readline module to get user-friendly command line behavior for reading commands from the user. (Done for you in provided client example.)

2.3.4. Python ao and mad audio libraries

If you want your client to begin playing a new file, you need to create a new MadFile object. This tells mad (our audio library) to interpret the next bytes as the beginning of a new file (which includes some metadata) rather than the middle of the previously playing file.

2.4. Assumptions / Simplifications

  • To simplify the file I/O, it’s fine for the server to keep its data, including the audio files, in memory, but the client should not store data that it isn’t actively using.

  • You may assume that the client has infinite buffering capacity for song data. That is, unlike "real" streaming protocols, you don’t have to keep track of how much data the client has processed before sending them more data. Once you start sending song data, just keep sending more until you reach the end (unless the client requests something else in the meantime).

  • One of the parameters to your server is a path to a directory that contains audio files and their corresponding information. Within this directory, you may assume that any file ending in .mp3 is an mp3 audio file. For each mp3 file, there will be a corresponding information file that is identical to the file name with .info tacked on to the end. For example, if there were a file in the directory named song1.mp3, there will also be a file named song1.mp3.info containing human-readable plain text information about that song. This info file is what should be supplied when the client issues an info command. I’ve made my music directory publicly available at /home/rware/music/, and I’ve provided code to read the names of these files. Feel free to use that or your own mp3 files for testing, but please do not submit audio files to GitHub!

2.5. Sending and Receiving Data

  • Use send() and recv() in as few places as possible. Never call either one on a socket unless select has told you that it’s safe to do so, otherwise, you’ll block (and potentially deadlock!).

  • Set your sockets to non-blocking mode for debugging: You can do this by calling set_non_blocking(int sock) in your starter code. This is not required (you should never block regardless because you should never try to send, recv, or accept unless select tells you it’s ok), but it will prevent deadlocking during your testing. Check their return values and errno for EAGAIN/EWOULDBLOCK, which indicate that you made a syscall that you shouldn’t have. That’s send/recv/accept 's way of telling you "I would have blocked on this call had this socket not been set for non-blocking mode."

  • Client closes a connection just before your server sends data: In this case, by default, two things will happen:

    1. your process will receive the SIGPIPE signal, which by default, kills your process (you don’t want this to occur, since you still want to service other clients)

    2. send will return an error and set errno to EPIPE to indicate the connection (pipe) was broken.

  • Luckily, we can easily prevent the SIGPIPE signal by using the extra flags parameter in send that we’ve been ignoring thus far. By setting the flags to MSG_NOSIGNAL, the kernel will only do (2), which is a much more convenient way for you to detect and handle a client disconnection.

3. Tips / FAQ

  • START EARLY! The earlier you start, the sooner you can ask questions if you get stuck. Test your code in small increments. It’s much easier to localize a bug when you’ve only changed a few lines.

  • Wireshark will not help you for this lab, since you’re designing the protocol this time. Wireshark knows nothing about how to decode your protocol!

3.1. Testing: Slowing Down the Server

To test message interleaving (handling text messages for a client that is also receiving song data), it helps to have a server that’s sending slower than our typical gigabit network. I’ve set up the machine staryu.cs.swarthmore.edu to artificially slow done its sending rate when sending from ports in the range 5001 - 5099.

Execute your server on staryu and bind to your favorite number within that port range. Try playing a song and then immediately making a list or info request. How long do you have to wait? If you’re properly interleaving messages, the delay should be short!

Sending from those ports, you’ll get a maximum throughput of approximately 256 kilobytes per second, so it will take about 20 seconds to fully transfer a 5-megabyte file. The files in my public music directory range from 2.5 MB to 13 MB, so that should give you plenty of time to test a play command followed by list or info.

4. Other Reference Materials

4.1. Event-based programming and select()

Event-based concurrency uses:

  • a single thread for all client requests.

  • we will use select() for event-based concurrency. Using select() the program is only notified when there is space in the socket buffers to send/receive across the list of available clients.

Look up the man page on select() and take a look at the input parameters:

int select(int nfds, fd_set *readfds, fd_set *writefds, fd_set *exceptfds, struct timeval *timeout);

  1. nfds: The first argument to the select function is (the largest numerical socket descriptor value in any of the subsequent FD_SET parameters) plus one. For example, if you’re populating your FD_SETs with socket descriptors 7, 9, and 10, your first argument to select should be 11 (10 + 1).

  2. fd_set: select() takes as input variables that by convention are named, rfds and wfds of type fd_set. These are file or socket descriptors (fd stands for file descriptor). As far as the OS is concerned file and socket descriptors are equivalent.

  3. FD_ZERO(input_var): clears out the set (input_var is usually either rfds - read/recv file descriptors or wfds - write/send file descriptors).

  4. FD_SET: allows you to put a socket in the file descriptor set. For e.g., FD_SET(serv_sock, &rfds) puts the server socket (the one on which we will call bind, listen and accept on) into the read set. We will always put the server socket in this set.

  5. When select() returns, make sure that as you check the FD_SET s , you differentiate between your server socket (that you accept connections on) and client sockets. If select tells you that your server socket is ready for reading, it means you can safely call accept.

  6. When select() leaves a socket descriptor in your read/write FD_SET after it returns, it means that you can safely (without blocking) recv/send on that socket once, but not more than once.

  7. If a client closes a connection while your server is blocked on select(), select will return that client’s socket in the set of file descriptors that are available for reading (usually named rfds). Then, when your server goes to recv() on that socket, it will get a return value of 0, which as we saw in lab 1, is recv() 's way of telling us that the connection is closed and no more data is coming (we’ve reached the EOF).

    • Thus, as I’ve been harping on all semester, you should always check the return value of your system calls to check for these types of conditions so that your server can detect and account for disconnecting clients.

4.2. Packing/Unpacking in C

To pack your data, you can create a buffer like you normally would: char data[] or char * data = malloc(#num bytes).

  • To pack or unpack a single byte you can just index into the array using data[idx].

  • To pack or unpack more than one byte the byte-ordering functions are as follows. Look up their man pages for more info.

    htons() - convert a short (2-byte int) from host byte order to network byte order

ntohs() - convert a short (2-byte int) from network byte order to host byte order

htonl() - convert a long (4-byte int) from host byte order to network byte order

ntohl() - convert a long (4-byte int) from network byte order to host byte order

  • For example, if you know that you have a short contained in positions 3 and 4 of a character array, you can call ntohs() on it as:

    short result = ntohs(*((short *) &buf[3]));

    Which says: take the address of the third offset in buf and cast the next two bytes to be a short. Then, dereference the pointer to the short data, to get a short value unpacked - i.e., 2 byte value starting at position 3.

5. Grading Rubric

This assignment is worth 10 points.

  • 1 point for completing the protocol design and meeting.

  • 1 point for designing a reasonable protocol to solve the problem as described in the protocol design document.

  • 1 point for accepting clients

  • 1 point for successfully replying to list or info requests.

  • 2 points for successfully streaming an audio file over the network.

  • 1.5 points for the server handling multiple concurrent connections, without blocking, via select(). Your server should NOT use threading. Using threads in your client is fine.

  • 1.5 points for interleaving different messages (play, list, info, stop) between the client and server. That is, a client should be able to request and receive info while it is currently playing a song.

  • 1 point for server resiliency - the server should be able to cope with clients joining and departing at any time. Be sure to check that your server does not crash when trying to both receive from and send to a disconnected client.

6. Submitting

Please remove any debugging output prior to submitting.

Please do not submit audio files to GitHub!

To submit your code, simply commit your changes locally using git add and git commit. Then run git push while in your lab directory.