C++ - Working with Bytes - Raw Sockets

[mikeo01]

Hey all,

Not getting much luck asking around so I thought I will ask here too.

Right, my current project is working with decoding SNMP messages (a.k.a data payload field within UDP datagrams). Everything is working dandy, as in I can break up the header, split everything up so I am left with the parts I need.

I can mix and match the data types and values from the SNMP message (working on TLV structure, type, length value). I can output the hexadecimal values, ASCII values to a text file which is nice.


However, I need to be able to deal with the byte's value in order to determine the next sequence (a.k.a its field length); so I need to be able to see whether a bit signal is marked or not. If a byte is greater than 127 it means the most significant bit is marked (to signal that there is more data coming); so I need to be able to identify this.


At the moment I have received datagrams into a char array and I have been working through this array (bitwise operations to find hex values etc).


However, what I am unsure of is why I cannot output a byte whose value is greater than 127. I am simply TEXT_FILE << (unsigned int) ARRAY[X].


To give some indication of what I mean see a bit of my output below (snippet):

Hexadecimal and ASCII values of array[?]

30 82 00 A3 02 01 00 04 0C 4D 4F 4E 49 54 4F 52 	0........MONITOR

Decimal of array[?] (8-bit)

48 4294967170 0 4294967203 2 1 0 4 12 77 79 78 73 84 79 82

Actual file output; so you know what my program does at the moment

_________________________________UDP Packet_________________________________

_IP HEADER_
IP Version: 4
IP Header Length: 5 DWORDS or 20 Bytes
Type Of Service: 0
IP Total Length: 195 Bytes(Size of Packet)
Identification: 5932
Reserved ZERO Field: 0
Dont Fragment Field: 0
More Fragment Field: 0
TTL: 128
Protocol: 17
Checksum: 51930
Source IP: 172.16.0.2
Destination IP: 172.16.0.1

_UDP HEADER_
Source Port: 51249
Destination Port: 161
UDP Length: 175
UDP Checksum: 16360

_IP HEADER_
45 00 00 C3 17 2C 00 00 80 11 CA DA AC 10 00 02 	E....,.......... 
AC 10 00 01 	                                    .... 

_UDP HEADER_
C8 31 00 A1 00 AF 3F E8 	                        .1....?. 

_DATA PAYLOAD_
30 82 00 A3 02 01 00 04 0C 4D 4F 4E 49 54 4F 52 	0........MONITOR 
5F 53 4E 4D 50 A0 82 00 8E 02 02 6B 64 02 01 00 	_SNMP......kd... 
02 01 00 30 82 00 80 30 82 00 0C 06 08 2B 06 01 	...0...0.....+.. 
02 01 01 01 00 05 00 30 82 00 0C 06 08 2B 06 01 	.......0.....+.. 
02 01 01 02 00 05 00 30 82 00 0C 06 08 2B 06 01 	.......0.....+.. 
02 01 01 03 00 05 00 30 82 00 0C 06 08 2B 06 01 	.......0.....+.. 
02 01 01 04 00 05 00 30 82 00 0C 06 08 2B 06 01 	.......0.....+.. 
02 01 01 05 00 05 00 30 82 00 0C 06 08 2B 06 01 	.......0.....+.. 
02 01 01 06 00 05 00 30 82 00 0C 06 08 2B 06 01 	.......0.....+.. 
02 01 01 07 00 05 00 30 82 00 0C 06 08 2B 06 01 	.......0.....+.. 
02 01 02 01 00 05 00 	                           .......

Data payload in decimal becomes:

48 4294967170 0 4294967203 2 1 0 4 12 77 79 78 73 84 79 82 95 83 78 77 80 4294967200 4294967170 0 4294967182 2 2 107 100 2 1 0 2 1 0 48 4294967170 0 4294967168 48 4294967170 0 12 6 8 43 6 1 2 1 1 1 0 5 0 48 4294967170 0 12 6 8 43 6 1 2 1 1 2 0 5 0 48 4294967170 0 12 6 8 43 6 1 2 1 1 3 0 5 0 48 4294967170 0 12 6 8 43 6 1 2 1 1 4 0 5 0 48 4294967170 0 12 6 8 43 6 1 2 1 1 5 0 5 0 48 4294967170 0 12 6 8 43 6 1 2 1 1 6 0 5 0 48 4294967170 0 12 6 8 43 6 1 2 1 1 7 0 5 0 48 4294967170 0 12 6 8 43 6 1 2 1 2 1 0 5 0 48 4294967169 4294967222 2 1 0 4 12 77 79 78 73 84 79 82 95 83 78 77 80 4294967202 4294967169 4294967202 2 2 107 100 2 1 0 2 1 0 48 4294967169 4294967189 48 29 6 8 43 6 1 2 1 1 1 0 4 17 51 67 111 109 32 83 117 112 101 114 83 116 97 99 107 32 51 48 24 6 8 43 6 1 2 1 1 2 0 6 12 43 6 1 4 1 43 10 27 4 1 2 4 48 14 6 8 43 6 1 2 1 1 3 0 67 2 127 64 48 12 6 8 43 6 1 2 1 1 4 0 4 0 48 16 6 8 43 6 1 2 1 1 5 0 4 4 83 78 77 80 48 12 6 8 43 6 1 2 1 1 6 0 4 0 48 13 6 8 43 6 1 2 1 1 7 0 2 1 2 48 13 6 8 43 6 1 2 1 2 1 0 2 1 57

Everything is dandy, except when the value is greater than 127, so 82 and A3 for example (Hex:82 decimal:130 and Hex:A3 decimal: 163).

So I know it is working through each byte correctly, but I don't know why it is having an issue with greater than 127.

Bit pattern 0111 1111 is fine, 127.
Bit pattern 1000 0000 no likey.

If I use signed int the output becomes the largest negative number (-127). So not sure?

RECVFROM() takes a char array


If you'd like to see what I am actually doing see below:

SEQUENCE: Field Length (Bytes): 197
Integer: 00 00 
Octet String: MONITOR_SNMP
GetResponse-PDU: Field Length (Bytes): 380
Integer: 611 611 64 64 
Integer: 00 00 
Integer: 00 00 
SEQUENCE: Field Length (Bytes): 556
SEQUENCE: Field Length (Bytes): 669
Object Identifier: 211 06 01 02 01 01 01 00 
Octet String: 3Com SuperS
SEQUENCE: Field Length (Bytes): 30
SEQUENCE: Field Length (Bytes): 18
Object Identifier: 211 06 01 02 01 01 02 00 
Object Identifier: 211 06 01 04 01 211 010 111 04 01 02 04 
SEQUENCE: Field Length (Bytes): 14
UNKNOWN: 08 08 211 211 06 06 01 01 02 02 01 01 
UNKNOWN: 00 00 43 43 02 02 
IpAddress: 012 06 08 211 06 01 02 01 01 04 00 04 00 30 10 06 08 211 06 01 02 01 01 05 00 04 04 53 414 413 
SEQUENCE: Field Length (Bytes): 12
UNKNOWN: 08 08 211 211 06 06 01 01 02 02 01 01 
UNKNOWN: 00 00 04 04 00 00 30 30 013 013 06 06 
UNKNOWN:

As you may see my octet string 3Com SuperS is the name of my managed switch, however, the hex of the "length" is 11, which in decimal is actually 17. So it hasn't looped over the values correctly. Problem is, is if this length identifier is bigger than 127 my program will try and loop 4294967169 or something similar.

(Output, I am decoding the data payload field in regards to SNMP messages, so as you can see getting the length right is essential to work out the next data types and values.

Any suggestions? Or a little lesson on what I am doing wrong?


I am a novice programmer so this is my first ever network specific program.

I appreciate this is a pretty big chunk and question to ask, but it seems something so simple is tripping me up.


Thanks,

Regards,
Mike

[BobF64]

Try

TEXT_FILE << (unsigned int) (ARRAY[X] & 0xFF)

Usually works for me.

[Goobley]

I had a similar issue once, (never worked network stuff, so I'm just talking about base conversions), mine was due to the way the data was originally pulled to the file and hidden characters messing the whole thing up. I couldn't see said hidden characters when the code pulled them but they were there.
My initial solution was to pull the characters individually, keep only the ints and perform the base conversion separately (I set up a class for this). Yes, that's an extremely yucky bit of coding and hideously inoptimised.
I remember that the second method involved not using stdin (cannot for the life of me remember what I used), and doing some string conversion operations prior to base conversion (as I knew exactly what my string should look like). I'll have to see if I can drag my code out from somewhere.

Good Luck,
Chris

Edit: What OS are you running? I remember my initial code that had the issues only had its issues on windows but worked fine on Linux.

[mikeo01]

I had a similar issue once, (never worked network stuff, so I'm just talking about base conversions), mine was due to the way the data was originally pulled to the file and hidden characters messing the whole thing up. I couldn't see said hidden characters when the code pulled them but they were there.
My initial solution was to pull the characters individually, keep only the ints and perform the base conversion separately (I set up a class for this). Yes, that's an extremely yucky bit of coding and hideously inoptimised.
I remember that the second method involved not using stdin (cannot for the life of me remember what I used), and doing some string conversion operations prior to base conversion (as I knew exactly what my string should look like). I'll have to see if I can drag my code out from somewhere.

Good Luck,
Chris

Edit: What OS are you running? I remember my initial code that had the issues only had its issues on windows but worked fine on Linux.

Funnily enough I have had hidden characters not showing; well I say characters, integers in particular. Although my code still worked. On Windows 8

So it looks like the way I have been converting it has been completely wrong.

Try

TEXT_FILE << (unsigned int) (ARRAY[X] & 0xFF)

Usually works for me.

This has worked, that's odd. Another user from a different forum has mentioned that char is 2 bytes, which I did not know (ooops).

Very confusing. Thank you! so basically what your code is telling is me to completely ignore the second byte of the char array?

So, is the second byte of my char array... empty? I am scratching my head here, if my program works and everything is fine (I have checked with Wireshark on this) then half of my char array is simply 0?

Thanks both!
I am still slightly confused about it; clearly I don't fully understand the concepts very well

EDIT: Ah this explains it all, my program does two things (>>4&0x0f) and (&0x0f); which means I am always going to get both nibbles of the first byte. Which begs, what on earth is in the second byte?

[Goobley]

No problem, sorry my assistance isn't all that great; like yourself I'm only learning

IIRC char length comes down to encoding, so there are different ways of handling different lengths. Aha found something on the topic: http://msdn.microsoft.com/en-us/library/cc953fe1.aspx "Type char is an integral type that usually contains members of the execution character set — in Microsoft C++, this is ASCII.". As most linux systems use 32-bit unicode I'd think that this means that char is going to be 4 bytes there while it is 1 byte on Microsquish C++.

[mikeo01]

No problem, sorry my assistance isn't all that great; like yourself I'm only learning

IIRC char length comes down to encoding, so there are different ways of handling different lengths. Aha found something on the topic: http://msdn.microsoft.com/en-us/library/cc953fe1.aspx "Type char is an integral type that usually contains members of the execution character set — in Microsoft C++, this is ASCII.". As most linux systems use 32-bit unicode I'd think that this means that char is going to be 4 bytes there while it is 1 byte on Microsquish C++.

Your assistance is really helpful anything to question my approach is always a great thing. Thank you!

So in C++ it is 1 byte, which begs the question, does that differ to C? I mean I may be using C style to handle incoming data then switching over to C++ style to actually process it.

Either way it seems clear that my char is in fact 2 bytes (as the code provided by BobF64) worked wonderfully. So, somewhere along the lines I've stored data into 2 bytes of memory (I think?).

[DanceswithUnix]

This has worked, that's odd. Another user from a different forum has mentioned that char is 2 bytes, which I did not know (ooops).

No, a char is 8 bits.

What is biting you here would seem to be that a char is 8 bits of storage in whatever the compiler thinks is the fastest/most natural form and the compiler is choosing signed. Taking one of the examples in your output, the decimal value:

4294967170

in hex is:

FFFFFF82

So the char value was hex 0x82. If you access that in a signed char, the top bit will extend to fill the register to preserve the idea that the top bit is set so it is a negative number.

I guess somewhere you have an array that should be declared as "unsigned char", and then it will see 0x82 as a positive integer.

Edit to add: Some internationalization systems expect a 16 bit character set, making text require 2 bytes per character. That is a text encoding, not a "char" which is a storage type.

[Goobley]

Whilst in c++ a char may refer to 1 byte (on ms), I would look at what your input file is encoded in (ASCII, UTF-8 or UTF-16), clearly you need the length of a character in the file to match the length that your code expects. These are the sorts of things that always trip me up.

Just saw DanceswithUnix's post: that makes sense, maybe one day I'll actually be able to avoid similar errors... a guy can dream.

[mikeo01]

No, a char is 8 bits.

What is biting you here would seem to be that a char is 8 bits of storage in whatever the compiler thinks is the fastest/most natural form and the compiler is choosing signed. Taking one of the examples in your output, the decimal value:

4294967170

in hex is:

FFFFFF82

So the char value was hex 0x82. If you access that in a signed char, the top bit will extend to fill the register to preserve the idea that the top bit is set so it is a negative number.

I guess somewhere you have an array that should be declared as "unsigned char", and then it will see 0x82 as a positive integer.

Edit to add: Some internationalization systems expect a 16 bit character set, making text require 2 bytes per character. That is a text encoding, not a "char" which is a storage type.

Ahhh I think I get what you mean.

Well, my array is declared as char; problem is declaring it as an unsigned char makes RECVFROM() complain. RECVFROM()

	char *BUFF = (char *)malloc(65536); // Allocate large memory block
	int SRC, STOP=0;

	if (BUFF==NULL) // Ensure BUFF isn't empty (means memory allocation failure
	{
	std::cout << "Memory Block Allocation - Failed.\n";
	return;
	}

	do // Loop around capturing packets IF
	{
		SRC=recvfrom(SNMP_ANALYSE,BUFF,65536,0,0,0); // Recieves all datagrams on this socket using large buffer (as to not buffer overload)

		if(SRC>0) // If datagram has been captured the buffer should be above 0 bytes
		{
		PROCESS(BUFF, SRC); // Send over what the socket has captured (raw data)
		}
		else // Otherwise do nothing with it
		{
			ERROR_LOG << WSAGetLastError() << "\n";
		}
	}
	while (STOP==0); // Free buffer if the socket has received datagrams as to not eat up large amounts of memory - dangerous

free(BUFF);

[TheAnimus]

OK, there appears to be a lack of understanding of what a char is or isn't.

A char is a datatype in C++, it's just a few bits of memory, that is it.

It isn't a representation of a textural character, don't think it is, and you won't be burned.

I'd highly recommended reading Joel's bit here:
http://www.joelonsoftware.com/articles/Unicode.html

What happens is that with unicode and certain formats, 7 bits can be a letter, 8 bits can be a letter 32bits can be a letter etc.

It sounds like you are trying to handle your strings in correctly. In the world of C++ this is often how we get nasty security issues! So it's really worth learning about strings and encodings properly.

[BobF64]

In compiler terms, because thats what ultimately chooses how data is represented or stored, all values are the size of the primary architecture registers.

So on x86 its 32bits, or a DWORD or long, and on x64 it will be 64bits, or a QWORD, long long, DWORD64 or whatever the language calls it.

Just because you cast it to something, doesnt necessarily truncate the value underneath.

[mikeo01]

I mostly understand what was said in that article; thanks great information.

I am going to link to what I am working off so you all understand (Should of done this in the beginning ): HERE

I am working with the raw data, which is encoded in ASN.1 (BER).


Now reading through everything it is said that the most significant bit is marked to signal the next byte of data is in fact apart of the current byte. (So anything above 127 means fragmentation).

OCTET Strings are sent in ASCII, so these are easily pulled out.


I know I must be interpreting the data wrong or storing it incorrectly because SNMP works off a simple Type, Length and Value approach.

Type defines the data type (integer, octet string, bit string and object identifiers).
Length defines the field length (Correct me if I am wrong but does this mean the length will be there for me, clear as day? Simply taking the value of each byte?
Value?


So with value, shouldn't this be a similar concept? If the decimal value is above 127 then the data is fragmented, so whatever is in there, OID, string, integer it'll be fragmented. Unless I am sticking them back together incorrectly?

Essentially what I am trying to say is: I am like a lost sheep

My head!!

[DanceswithUnix]

Ahhh I think I get what you mean.

Well, my array is declared as char; problem is declaring it as an unsigned char makes RECVFROM() complain. RECVFROM()

Well that is broken. I would expect recvfrom() to take a void*, so it shouldn't matter what you pass to it as a pointer. Are you using a Microsoft compiler? Not much good experience with those. You could cast your call something like:

recvfrom(SNMP_ANALYSE,(char *)BUFF,65536,0,0,0);

Some style advice also, all upper case is assumed to be a constant so you want to lower case your variables, and the magic 65536 should be replaced in both places with a constant as if one of them ever gets changed without the other changing then you have a bug.

If you were using Linux I would suggest you run your code with the DDD debugger which is excellent and will let you see what is happening not just line by line but watching the data flow around your code.

The comment from TheAnimus is key, a char in C is a "storage type". The fact that 4 decades ago it was considered a good way to store text leaves us with an unfortunate name.