Showing posts with label ASCII. Show all posts
Showing posts with label ASCII. Show all posts

Monday, February 26, 2024

Encodings

ASCII

ASCII stands for American Standard Code for Information Interchange. It is a character encoding standard that assigns numerical values (codes) to represent characters, including letters, numbers, punctuation marks, and control characters. 

The American Standard Code for Information Interchange (ASCII) was developed by a committee of the American Standards Association (ASA), called the X3 committee, by its X3.2.4 working group in early 60's. The ASA later became the United States of America Standards Institute (USASI) and ultimately became the American National Standards Institute (ANSI).

ASCII is a 7-bit character set containing 128 characters. 27=128. It contains the numbers from 0-9, the upper and lower case English letters from A to Z, and some special characters.

ASCII has been widely used in computers and communication equipment for encoding text data. However, with the need to represent characters beyond the original 128, extended versions of ASCII have been developed, such as ISO 8859 and UTF-8, which support a wider range of characters including those from non-English languages and special symbols.

ASCII Printable Characters

Char

Number

Description

 

0 - 31

Control characters (see below)

 

32

space

!

33

exclamation mark

"

34

quotation mark

#

35

number sign

$

36

dollar sign

%

37

percent sign

&

38

ampersand

'

39

apostrophe

(

40

left parenthesis

)

41

right parenthesis

*

42

asterisk

+

43

plus sign

,

44

comma

-

45

hyphen

.

46

period

/

47

slash

0

48

digit 0

1

49

digit 1

2

50

digit 2

3

51

digit 3

4

52

digit 4

5

53

digit 5

6

54

digit 6

7

55

digit 7

8

56

digit 8

9

57

digit 9

:

58

colon

;

59

semicolon

< 

60

less-than

=

61

equals-to

> 

62

greater-than

?

63

question mark

@

64

at sign

A

65

uppercase A

B

66

uppercase B

C

67

uppercase C

D

68

uppercase D

E

69

uppercase E

F

70

uppercase F

G

71

uppercase G

H

72

uppercase H

I

73

uppercase I

J

74

uppercase J

K

75

uppercase K

L

76

uppercase L

M

77

uppercase M

N

78

uppercase N

O

79

uppercase O

P

80

uppercase P

Q

81

uppercase Q

R

82

uppercase R

S

83

uppercase S

T

84

uppercase T

U

85

uppercase U

V

86

uppercase V

W

87

uppercase W

X

88

uppercase X

Y

89

uppercase Y

Z

90

uppercase Z

[

91

left square bracket

\

92

backslash

]

93

right square bracket

^

94

caret

_

95

underscore

`

96

grave accent

a

97

lowercase a

b

98

lowercase b

c

99

lowercase c

d

100

lowercase d

e

101

lowercase e

f

102

lowercase f

g

103

lowercase g

h

104

lowercase h

i

105

lowercase i

j

106

lowercase j

k

107

lowercase k

l

108

lowercase l

m

109

lowercase m

n

110

lowercase n

o

111

lowercase o

p

112

lowercase p

q

113

lowercase q

r

114

lowercase r

s

115

lowercase s

t

116

lowercase t

u

117

lowercase u

v

118

lowercase v

w

119

lowercase w

x

120

lowercase x

y

121

lowercase y

z

122

lowercase z

{

123

left curly brace

|

124

vertical bar

}

125

right curly brace

~

126

tilde



ASCII Control Characters  

 Char

Number

Description

NUL

00

null character

SOH

01

start of header

STX

02

start of text

ETX

03

end of text

EOT

04

end of transmission

ENQ

05

enquiry

ACK

06

acknowledge

BEL

07

bell (ring)

BS

08

backspace

HT

09

horizontal tab

LF

10

line feed

VT

11

vertical tab

FF

12

form feed

CR

13

carriage return

SO

14

shift out

SI

15

shift in

DLE

16

data link escape

DC1

17

device control 1

DC2

18

device control 2

DC3

19

device control 3

DC4

20

device control 4

NAK

21

negative acknowledge

SYN

22

synchronize

ETB

23

end transmission block

CAN

24

cancel

EM

25

end of medium

SUB

26

substitute

ESC

27

escape

FS

28

file separator

GS

29

group separator

RS

30

record separator

US

31

unit separator

 

 

 

DEL

127

delete (rubout)

 

URL Encodings

URL encoding, also known as percent-encoding, is a mechanism used to convert certain characters in a URL (Uniform Resource Locator) into a format that can be safely transmitted over the internet.

In URLs, certain characters have special meanings or functions. For example, the characters ?, &, /, and = are used to separate different parts of the URL or to denote query parameters. If you want to include these characters in the URL as part of data rather than as delimiters or special characters, they need to be encoded.

URL encoding works by replacing non-alphanumeric characters with a '%' followed by their ASCII hexadecimal value. For instance, a space character ' ' is represented as '%20', since its ASCII hexadecimal value is 20.


ANSI

The American National Standards Institute (ANSI) is a private, non-profit organization that administers and coordinates the U.S. voluntary standards and conformity assessment system. The ANSI character set was the standard set of characters used in Windows operating systems through Windows 95 and Windows NT, after which Unicode was adopted.


ISO-8859

ISO 8859, also known as Latin alphabet No. 1, is a series of character encoding standards developed by the International Organization for Standardization (ISO) in 1987. These standards are designed to extend the ASCII character set to include characters from various languages that use Latin scripts, such as English, French, German, Spanish, and many others. Each ISO 8859 standard defines an 8-bit character encoding, allowing for a total of 256 characters. There are several versions of the ISO 8859 standard, each tailored to support specific languages or language groups. The most commonly used versions include ISO 8859-1, ISO 8859-2, ISO 8859-3, and so on, up to ISO 8859-16. Each version provides support for additional characters beyond the original ASCII character set while maintaining compatibility with ASCII for the first 128 characters.

In ISO-8859-1, the characters from 128 to 159 are not defined. The next part of ISO-8859-1 (codes from 160-191) contains commonly used special characters. If you use the less than (<) or greater than (>) signs in your HTML text, the browser might mix them with tags. Entity names or entity numbers can be used to display reserved HTML characters. Entity names are represented as &entity_name; Entity numbers are represented as &#entity_number;


ISO-8859-1 Symbols (160-191)

Character

Entity Number

Enity Name

Description

 

&#160;

&nbsp;

non-breaking space

¡

&#161;

&iexcl;

inverted exclamation mark

¢

&#162;

&cent;

cent

£

&#163;

&pound;

pound

¤

&#164;

&curren;

currency

¥

&#165;

&yen;

yen

¦

&#166;

&brvbar;

broken vertical bar

§

&#167;

&sect;

section

¨

&#168;

&uml;

spacing diaeresis

©

&#169;

&copy;

copyright

ª

&#170;

&ordf;

feminine ordinal indicator

«

&#171;

&laquo;

angle quotation mark (left)

¬

&#172;

&not;

negation

­

&#173;

&shy;

soft hyphen

®

&#174;

&reg;

registered trademark

¯

&#175;

&macr;

spacing macron

°

&#176;

&deg;

degree

±

&#177;

&plusmn;

plus-or-minus

²

&#178;

&sup2;

superscript 2

³

&#179;

&sup3;

superscript 3

´

&#180;

&acute;

spacing acute

µ

&#181;

&micro;

micro

&#182;

&para;

paragraph

·

&#183;

&middot;

middle dot

¸

&#184;

&cedil;

spacing cedilla

¹

&#185;

&sup1;

superscript 1

º

&#186;

&ordm;

masculine ordinal indicator

»

&#187;

&raquo;

angle quotation mark (right)

¼

&#188;

&frac14;

fraction 1/4

½

&#189;

&frac12;

fraction 1/2

¾

&#190;

&frac34;

fraction 3/4

¿

&#191;

&iquest;

inverted question mark



ISO-8859-1 Characters (192-255)

Character

Entity Number

Entity Name

Description

À

&#192;

&Agrave;

capital a, grave accent

Á

&#193;

&Aacute;

capital a, acute accent

Â

&#194;

&Acirc;

capital a, circumflex accent

Ã

&#195;

&Atilde;

capital a, tilde

Ä

&#196;

&Auml;

capital a, umlaut mark

Å

&#197;

&Aring;

capital a, ring

Æ

&#198;

&AElig;

capital ae

Ç

&#199;

&Ccedil;

capital c, cedilla

È

&#200;

&Egrave;

capital e, grave accent

É

&#201;

&Eacute;

capital e, acute accent

Ê

&#202;

&Ecirc;

capital e, circumflex accent

Ë

&#203;

&Euml;

capital e, umlaut mark

Ì

&#204;

&Igrave;

capital i, grave accent

Í

&#205;

&Iacute;

capital i, acute accent

Î

&#206;

&Icirc;

capital i, circumflex accent

Ï

&#207;

&Iuml;

capital i, umlaut mark

Ð

&#208;

&ETH;

capital eth, Icelandic

Ñ

&#209;

&Ntilde;

capital n, tilde

Ò

&#210;

&Ograve;

capital o, grave accent

Ó

&#211;

&Oacute;

capital o, acute accent

Ô

&#212;

&Ocirc;

capital o, circumflex accent

Õ

&#213;

&Otilde;

capital o, tilde

Ö

&#214;

&Ouml;

capital o, umlaut mark

×

&#215;

&times;

multiplication

Ø

&#216;

&Oslash;

capital o, slash

Ù

&#217;

&Ugrave;

capital u, grave accent

Ú

&#218;

&Uacute;

capital u, acute accent

Û

&#219;

&Ucirc;

capital u, circumflex accent

Ü

&#220;

&Uuml;

capital u, umlaut mark

Ý

&#221;

&Yacute;

capital y, acute accent

Þ

&#222;

&THORN;

capital THORN, Icelandic

ß

&#223;

&szlig;

small sharp s, German

à

&#224;

&agrave;

small a, grave accent

á

&#225;

&aacute;

small a, acute accent

â

&#226;

&acirc;

small a, circumflex accent

ã

&#227;

&atilde;

small a, tilde

ä

&#228;

&auml;

small a, umlaut mark

å

&#229;

&aring;

small a, ring

æ

&#230;

&aelig;

small ae

ç

&#231;

&ccedil;

small c, cedilla

è

&#232;

&egrave;

small e, grave accent

é

&#233;

&eacute;

small e, acute accent

ê

&#234;

&ecirc;

small e, circumflex accent

ë

&#235;

&euml;

small e, umlaut mark

ì

&#236;

&igrave;

small i, grave accent

í

&#237;

&iacute;

small i, acute accent

î

&#238;

&icirc;

small i, circumflex accent

ï

&#239;

&iuml;

small i, umlaut mark

ð

&#240;

&eth;

small eth, Icelandic

ñ

&#241;

&ntilde;

small n, tilde

ò

&#242;

&ograve;

small o, grave accent

ó

&#243;

&oacute;

small o, acute accent

ô

&#244;

&ocirc;

small o, circumflex accent

õ

&#245;

&otilde;

small o, tilde

ö

&#246;

&ouml;

small o, umlaut mark

÷

&#247;

&divide;

division

ø

&#248;

&oslash;

small o, slash

ù

&#249;

&ugrave;

small u, grave accent

ú

&#250;

&uacute;

small u, acute accent

û

&#251;

&ucirc;

small u, circumflex accent

ü

&#252;

&uuml;

small u, umlaut mark

ý

&#253;

&yacute;

small y, acute accent

þ

&#254;

&thorn;

small thorn, Icelandic

ÿ

&#255;

&yuml;

small y, umlaut mark


Variants of ISO-8859

Number

Description

Covers

8859-1

Latin 1

North America, Western Europe, Latin America, the Caribbean, Canada, Africa.

8859-2

Latin 2

Eastern Europe.

8859-3

Latin 3

SE Europe, Esperanto, miscellaneous others.

8859-4

Latin 4

Scandinavia/Baltics (and others not in ISO-8859-1).

8859-5

Latin/Cyrillic

The Cyrillic alphabet. Bulgarian, Belarusian, Russian and Macedonian.

8859-6

Latin/Arabic

The Arabic alphabet.

8859-7

Latin/Greek

The modern Greek alphabet and mathematical symbols derived from the Greek.

8859-8

Latin/Hebrew

The Hebrew alphabet.

8859-9

Latin/Turkish

The Turkish alphabet. Same as ISO-8859-1 except Turkish characters replace Icelandic.

8859-10

Latin/Nordic

Nordic alphabets. Lappish, Nordic, Eskimo.

8859-15

Latin 9 (Latin 0)

Similar to ISO-8859-1 but replaces some less common symbols with the euro sign and some other missing characters.

2022-JP

Latin/Japanese 1

The Japanese alphabet part 1.

2022-JP-2

Latin/Japanese 2

The Japanese alphabet part 2.

2022-KR

Latin/Korean 1

The Korean alphabet.

However, ISO 8859 has limitations, particularly in supporting characters from languages outside the Western European region, which led to the development of more comprehensive encoding standards like Unicode.


ANSI Code Page & Windows-1252

ANSI code pages officially called "Windows code pages after Microsoft accepted the former term being a misnomer are used for native non-Unicode (byte oriented) applications using a graphical user interface on Windows systems. The term "ANSI" is a misnomer because these Windows code pages didn't comply with any ANSI standard. Code page 1252 was based on an early ANSI draft that later became the international standard ISO 8859-1. Windows-1252 was the first default character set in Microsoft Windows. Undeclared charsets in HTML are also assumed to be Windows-1252. Windows-1252 is identical to ISO-8859-1 except for the code points 128-159 (0x80-0x9F). In ISO-8859-1, the characters from 128 to 159 are not defined. Windows-1252 has several characters, punctuation, arithmetic and business symbols assigned to these code points.

Character

Number

Entity Name

Description

128

&euro;

euro sign

 

129

 

NOT USED

130

&sbquo;

single low-9 quotation mark

ƒ

131

&fnof;

Latin small letter f with hook

132

&bdquo;

double low-9 quotation mark

133

&hellip;

horizontal ellipsis

134

&dagger;

dagger

135

&Dagger;

double dagger

ˆ

136

&circ;

modifier letter circumflex accent

137

&permil;

per mille sign

Š

138

&Scaron;

Latin capital letter S with caron

139

&lsaquo;

single left-pointing angle quotation mark

Œ

140

&OElig;

Latin capital ligature OE

 

141

 

NOT USED

Ž

142

&Zcaron;

Latin capital letter Z with caron

 

143

 

NOT USED

 

144

 

NOT USED

145

&lsquo;

left single quotation mark

146

&rsquo;

right single quotation mark

147

&ldquo;

left double quotation mark

148

&rdquo;

right double quotation mark

149

&bull;

bullet

150

&ndash;

en dash

151

&mdash;

em dash

˜

152

&tilde;

small tilde

153

&trade;

trade mark sign

š

154

&scaron;

Latin small letter s with caron

155

&rsaquo;

single right-pointing angle quotation mark

œ

156

&oelig;

Latin small ligature oe

 

157

 

NOT USED

ž

158

&zcaron;

Latin small letter z with caron

Ÿ

159

&Yuml;

Latin capital letter Y with diaeresis


HEXADECIMAL

Hexadecimal, often abbreviated as "hex," is a base-16 numeral system used in mathematics and computer science. In hexadecimal, numbers are represented using 16 symbols: the digits 0-9 and the letters A-F (where A represents 10, B represents 11, and so on up to F representing 15). Hexadecimal is commonly used in computing because it provides a more concise way to represent binary data, as each hexadecimal digit corresponds to four binary digits (bits).

Decimal

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Hexadecimal

0

1

2

3

4

5

6

7

8

9

A

B

C

D

E

F

After reaching 9 in decimal, hexadecimal uses letters A-F to represent values from 10 to 15. Lets convert decimal value 269 to hexadecimal..






,



The hexadecimal number is the reverse of the remainder we get in each step.

Base64

Base64 is a binary-to-text encoding scheme that represents binary data in an ASCII string format by translating it into a radix-64 representation. It's used to encode binary data, such as images, audio files, or any other binary content, into a format that can be transmitted over text-based channels, such as email or URLs, without corruption due to special characters or encoding issues. The binary data is divided into groups of 6 bits. Each group of 6 bits is then represented by a character from a predefined set of 64 printable ASCII characters. These characters typically include uppercase and lowercase letters (A-Z, a-z), digits (0-9), and two additional symbols (usually '+' and '/'). Padding characters ('=') are added if the number of bits in the original binary data is not divisible by 6. For example, consider the sentence Hi\n, where the \n represents a newline. The first step in the encoding process is to obtain the binary representation of each ASCII character.



UTF-8

UTF-8, which stands for Unicode Transformation Format 8-bit, is a variable-width character encoding capable of encoding all possible Unicode code points. It's the most commonly used encoding on the internet and in computing systems worldwide because it efficiently represents a wide range of characters while maintaining backward compatibility with ASCII.

UTF-8 converts a code point (which represents a single character in Unicode) into a set of one to four bytes. UTF-8 is compact and efficient, especially for languages that use mostly ASCII characters. For example, an English text encoded in UTF-8 will use the same space as of ASCII text. A code point is a number assigned to represent an abstract character in unicode. The code point for a character is typically represented in hexadecimal notation. For example, the code point for the letter "A" is U+0041, where "U+" indicates that the following digits represent a Unicode code point, and "0041" is the hexadecimal representation of the code point.

In Unicode, a "plane" refers to a continuous group of 65,536 (2^16) code points. Unicode is organized into a multilevel hierarchical structure, and planes are one of the key components of this structure. The Unicode Standard assigns code points to different planes to accommodate a vast number of characters from various scripts and symbol sets. Unicode divides its code space into 17 planes, labeled from 0 to 16 (0x0 to 0x10 in hexadecimal). Each plane contains 65,536 code points, providing a total of 1,114,112 (17 * 65,536) code points. The first plane, Plane 0 (U+0000 to U+FFFF), known as the Basic Multilingual Plane (BMP), contains most commonly used characters for modern text processing, covering scripts such as Latin, Cyrillic, Greek, Hebrew, Arabic, Chinese, Japanese, and Korean, as well as many symbols, punctuation marks, and control characters. Planes 1 through 16 are referred to as "supplementary planes." They contain additional characters and symbols, including historical scripts, rare characters, emoji, mathematical symbols, musical symbols, and more. Plane 1 in Unicode, also known as the Supplementary Multilingual Plane (SMP), consists of code points ranging from U+10000 to U+1FFFF. Likewise Plane 2 consists of code points ranging from U+20000 to U+2FFFF.

In UTF-8 encoding, code points in Plane 0 are represented using sequences of one to three bytes, depending on the code point's value

  1. For code points in the range U+0000 to U+007F (0 to 127), UTF-8 encodes them as follows: Code points in this range are represented using a single byte. The byte's value directly corresponds to the code point's value.
  2. For code points in the range U+0080 to U+07FF, UTF-8 encodes them as follows: Code points in this range are represented using two bytes. The high-order 5 bits of the code point are stored in the first byte, and the low-order 6 bits are stored in the second byte.
  3. For code points in the range U+0800 to U+FFFF, UTF-8 encodes them as follows: Code points in this range are represented using three bytes. The high-order 4 bits of the code point are stored in the first byte, the next 6 bits in the second byte, and the low-order 6 bits in the third byte.

Here's a general pattern for representing code points in Plane 0 using UTF-8:

  • One-byte sequence: 0xxxxxxx
  • Two-byte sequence: 110xxxxx 10xxxxxx
  • Three-byte sequence: 1110xxxx 10xxxxxx 10xxxxxx

Where 'x' represents bits from the code point.
For example, let's say we have the code point U+0041. Its binary representation is: U+0041 = 0000 0000 0100 0001 To represent this code point in UTF-8: Since it falls in the range U+0000 to U+007F, it is represented using a single byte. Therefore, the UTF-8 representation of U+0041 would be: 01000001

Lets take another example U+0081

Its binary representation is: 0000 0000 1000 0001

The high-order 5 bits = 00010 & low-order 6 bits = 000001

U+0081=11000010 10000001

Lets take another example U+FFFF

Its binary representation is: 1111 1111 1111 1111

high order 4 bits=1111

first low order 6 bits=111111

second low order 6 bits=111111
U+FFFF= 11101111 10111111 10111111

Lets take another example UTF-8 representation of U+10348 which is a plane 1 codepoint Binary representation = 0001 0000 0011 0100 1000 This needs 20 bits to represented and the unicode variable size format can support upto 21 bits. Lets break it down into nine high order bit and 12 low order bits. 000100000 001101001000 As you can see the last group is only 11 bits hence we will prefix 0. Final UTF-8 encoding will be 11110000 10100000 10001101 10001000

source - w3Wikipedia