Disclaimer, I’m not a low-level network coding expert, so I’m sure there are easier/better architectures for how to handle this sort of thing. I’ll study them someday
To keep my mind off the world going to shit recently, I redid parts of my game’s networking code last night to add “sync” support to my packet transmissions (make sure a packet was received). Mini case study below.
Basically since UDP does not guarantee delivery of packets, packets can arrive late, never, out of order, or duplicated, you’re royally fucked if a certain important packet such as (the unit was split in two) arrives twice. So for packets that aren’t just player position, you have to handle the sync properly. Each packet starts with the same header
|
1 2 3 4 5 6 |
struct Header
{
uint16_t type; //state type
SeqType seq; //packet number (sequence)
SeqType ack;
}; |
The first step is to add seq and ack to all outgoing packets where seq is the packet number of this packet and ack is the highest number of any packet we’ve received back. For example, if the client is sending the 100th packet, seq will be 100. If the server receives this packet, upon it’s next update to this client, it’ll send a packet with the ack containing 100, so the client can know it was received. This is a super basic well known mechanism for reliability (TCP does something like this but considerably more complex)
So, the issue last night was, how the fuck do I handle a missed ack. Architecturally it got a bit messy. Basically my game works on a heartbeat type mechanism where the app is set into a certain state, (like a battle match), and the same packet type is sent over and over again from the client with updated data (the player positions). If they get lost, no big deal, because another one is coming right away. The receiver only considers the highest ordered seq packet, and disregards anything older than that.
For packets that must be received once and only once,
I had to start marking these critical packets as “sync” packets, meaning they are important and nothing else will be sent by the client until the server reports back that it got the sync packet, and that it successfully sync’ed it to every client connected to it.
The devil was completely in the details on doing this right. For example, for a sync packet I had to ensure the seq number was the same during all retransmissions, so they would be processed only once, etc etc.
Key parts of this in the client are below (the server was considerably more complex, since it has to sync to multiple clients where the seqs/acks are different for each one)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 |
class Client
{
public:
Client(const std::string &serverAddressStr, const std::string &playerName) { ... }
~Client() { ... }
void poll()
{
if(sendOnPoll.load())
{
sendState();
sendOnPoll = false;
}
while(connection->isDataAvailable())
{
size_t bytesRead = 0;
if(connection->receive(packetData, MaxPacketSize, &bytesRead))
{
auto sender = connection->getLastReceivedFromAddress();
auto state = ServerState::stateFromPacketData(packetData, bytesRead, sender);
uint32_t ack = state->getPacketAck();
if(state->isStale())
{
//ignore out-of-order older (stale) packets
continue;
}
if(syncState)
{
if(syncSeq && ack >= syncSeq)
{
currentState->setSync(false);
currentState->syncComplete();
currentState->setAdvanceSeqBeforeSend(true);
syncState = false;
}
}
switch(state->getType()) { ... }
}
}
//...
}
///Send our state to the server
void sendState()
{
currentState->sendToConnection(connection.get(), serverAddress.get());
if(syncState)
{
//make note of the per-client seq for this packet sent to the server
syncSeq = currentState->getPacketSeq();
}
currentState->setAdvanceSeqBeforeSend(false);
sentPacketSeqs.push_front({ currentState->getPacketSeq(), vgl::System::system().globalTime() });
}
void setSyncState(bool b) { .... }
private:
//.....
vnet::UDPConnection::Pointer connection;
vnet::SocketAddress::Pointer serverAddress;
std::atomic_bool sendOnPoll;
bool syncState = false;
uint32_t syncSeq = 0;
}; |
..and the actual packet transmission code is below..
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
void ClientState::sendToConnection(vnet::UDPConnection *conn, vnet::SocketAddress *destAddress)
{
if(dataLen > 1024)
{
throw std::runtime_error("ClientState::sendToConnection() packet too large!");
}
assert(data != nullptr);
assert((StateType)type != StateType::None);
Header *hPtr = (Header *)data;
auto addr = conn->getFromAddress()->getAddr();
uint64_t ipPort = vnet::SocketAddress::ipPort(ntohl(addr->sin_addr.s_addr), ntohs(addr->sin_port));
SeqType &sendSeq = packetSeqSend[ipPort];
if(sync && advance)
{
sendSeq++;
}
if(!sync)
{
hPtr->seq = sendSeq++;
}
else
{
hPtr->seq = sendSeq;
}
auto destAddr = destAddress->getAddr();
uint64_t destIPPort = vnet::SocketAddress::ipPort(ntohl(destAddr->sin_addr.s_addr), ntohs(destAddr->sin_port));
hPtr->ack = packetSeqRecv[destIPPort];
conn->send(destAddress, data, dataLen);
} |
Currently, any ClientState (my packet base class) can set itself as a sync packet, once the Client encounters one of those, it’s in syncState mode (and retransmits the same packet over and over) until it’s resolved with an ack as shown above. In the higher level game code, if the client is in a sync state, I treat the game as if it’s paused (can’t move).
..and lastly, the pertinent sections of the actual UDPConnection class follows below. This is where my game actually touches the OS sockets API. Generally, this API is the lowest level of networking programming available to most applications.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
bool UDPConnection::send(SocketAddress *destAddress, void *data, size_t length)
{
if(sendto(socket, (const char *)data, length, 0, (sockaddr *)&destAddress->addr, sizeof(destAddress->addr)) < 0)
{
return false;
}
return true;
}
bool UDPConnection::receive(void *data, size_t dataLen, size_t *bytesRead)
{
sockaddr_in remoteAddr;
socklen_t addrlen = sizeof(remoteAddr);
memset(&remoteAddr, 0, sizeof(remoteAddr));
int64_t ret = recvfrom(socket, (char *)data, dataLen, 0, (sockaddr *)&remoteAddr, &addrlen);
if(ret >= 0)
{
if(!lastReceivedFromAddress)
{
lastReceivedFromAddress = make_shared<SocketAddress>(&remoteAddr);
}
else
{
lastReceivedFromAddress->setAddr(&remoteAddr);
}
}
*bytesRead = ret;
return (ret >= 0);
} |