Most computers use a 1 or 2 byte binary sequence to encode the various characters required. A character is anything you’d like to be able to display on a screen that can be typed on a keyboard.
This is the standard American format. It uses binary values 0..127 to encode 128 character values in a standard way. However, note that 8 bits give you 256 values. So what do you do with the rest? They are either ignored, or used to extend the ASCII set in a non standard way.
The set is divided into four groups of 32 characters. The first group of 32 characters holds values 0..31(in hex, 0x0..0x1F) and forms a special set of control characters. They perform various printer and display operations rather than displaying actual symbols.
One stupid thing is that control characters do different things on different output devices so you have to read the manuals to write drivers etc.
The second group of character codes consists of various punctuation symbols, special characters, and numeric digits. space is 0x20, numbers 0-9 are 0x30..0x39
The third group containes uppercase alphabetic. A..Z are in 0x41..0x5a. Obviously there are only 26 different letters, so the extra values store different special chars.
The fourt group consists of lowercase alphabetics and some special symbols as well as the delete control char. Converting between upper and lower case is easy you just flip bit 5.
This structure is kind of nice. Bits 5 and 6 determine the character group, and you can convert things very quickly by flipping those bits. For example, the digits 0..9 in character form look like 0011_0000, 0011_0001, .... To conver from the character to actual number representation, just set the top nibble to 0.
Using a single byte runs into problems because you can only represent 256 values. In many languages like Chinese, you need more than that. Thus, double byte character sets exist.
These sets don’t use 16 bits for everything, they use 8 bits for most things, and the other 8 only when needed(for special characters). They normally build on ascii by using the first 128 values, and then using the rest of the 128 for extending + having some special value that says that another byte is coming.
Unicode is the standard for this. It comes from old engineers at Apple and Xerox. It’s backwards compatible with ASCII.
uses 16 bit words. If the HO 9 bits are 0, then LO 7 bits are standard ASCII. If the HO 9 bits contain a non 0 value, then the 16 bits for an extended character code(extended from ASCII). In fact, Unicode even provides a set of codes you can use to create your own user defined character set.
Unicode has good support, but is obviously slower. Every string becomes twice as long in memory, and you have to do some fuckery to convert between the two or check characters etc. ASCII is simple, and is thus fast, as normal.
Most common type used in modern programs.
A sequence of characters that posses a length and character data. Like any other data structure, you can have regular zero terminated strings which are fast, or add a bunch of other data to that structure like length, reference count, etc. (Think smart pointers for example).
These are super common, default in c and cpp.
Advantages:
Disadvantages:
Are a slight upgrade to null terminated. They store the length of the string as the first byte. Thus, they have the same overhead as the null terminated strings, and make string operations more efficient.
They can obvioulsy only handle strings of length 255 or below(since we have only one byte for the length)
Instead of holding the descriptive data next to the string, you create a data structure that defines characteristics(max length, current length, etc) and then maintains a pointer to the string itself.
This is obviously more overhead, but can make operations much faster. This is common in higher level languages, go does something like this I believe, although the strings are immutable slices of bytes.
Static are those whose maximum size a programmer chooses when writing the program. Think arrays of characters in cpp, go too. There is no way to increase the size, the data is stored as static.
Pseudo-Dynamic are those whose length the system sets are run time by calling malloc or some other memory management function. Once the system allocates the memory, the length is fixed. Increasing means copying.
Dynamic will automatically allocate sufficient storage for a string when you create a new string or do something else. These usually work with descriptors(the struct which references the value through a pointer).
Slices work kind of like this I think??? Look into that!
Keeping substrings is easy, you just create a new data structure that holds a new pointer to different indexes in the string value.
How do we get around to not knowing when we can claim back some space used for a string? I mean when we use descriptors, multiple data structures can point back to the same string data. We need to make sure we don’t delete the contents until they are not needed. To do that, one approach is reference counting.
An approach similar to this is used in rust through RC and ARC types.
You can always define your own character sets, there’s nothing that keeps you to using ASCII. That’s how game tilemaps work too. But they will be appliction specific by definition.