哈工大軟件學院編譯原理實驗1——詞法分析

時間 2019-11-11

標籤軟件學院編譯原理實驗詞法分析简体版

原文原文鏈接

此次實驗被「過來人」們定位很是easy，實驗內容例如如下：python

-----------------------------------------------------------------------------------git

對例如如下工做進行展開描寫敘述數組

(1) 給出語言的詞法規則描寫敘述網絡

· 標識符、keyword、整常數、字符常數、浮點常數數據結構

· 單界符：＋，－，×，;，…app

· 雙界符：/*，:=，>=，<=，!=，…函數

· 凝視工具

(2) 針對這樣的單詞的狀態轉換圖和程序框圖編碼

(3) 核心數據結構的設計spa

如符號表、keyword等

(4) 錯誤處理

錯誤的位置及類型等

-----------------------------------------------------------------------------------

此次實驗我是用python寫了一個簡單的C語言的詞法分析器，詞法分析器的編寫有很是多種方式，比方可以用正則表達式編寫，也可以用LEX工具本身主動生成，固然，也可以用比較樸素的方式——基於有窮本身主動機（Finite Automata，FA），也即基於有窮本身主動機的狀態轉移來編敲代碼。

提及這個有窮本身主動機（Finite Automata，FA），真心感受是個好東西，分析問題簡單清晰，而且很是直觀。記得對有窮本身主動機有感性認識是在上學期考試分數並不高的《計算機網絡》課上，全龍哥講那個RDT協議的不一樣版本號的時候，用這個本身主動機來代表遇到不一樣狀況時發送端和接收端要採取的行動。

本身主動機的形式化定義：

M=(Q，Σ，δ，q0，F)
„ Q→有窮狀態集
„ Σ→有窮輸入字母表
„ δ→從Q×Σ→2Q的映射函數（2Q是Q的冪集）
„ q0∈Q，是惟一的初態
„ F →終態集合，是Q的子集

這裏說說我我的的理解。對有窮本身主動機的形式化定義的理解比較重要的是上述第三條說明的理解：δ→從Q×Σ→2Q的映射函數。也即這個δ定義了某個詳細FA的狀態間轉移關係，或者說定義了某個FA的狀態間轉移的規則。所謂狀態的冪集就是狀態集Q的所有子集構成的集族。則這句話的字面意思是：狀態集和字母表的笛卡爾乘積到狀態集的冪集的映射函數。

比方：M1 = (Q，Σ，δ，q0，F)，當中Q = {q,q0,q1,q2...,qn}，又((q，a) , {q1,q2,q3})∈δ，也即δ((q,a)) = {q1,q2,q3}。則說明本身主動機M1有一個狀態q，q在遇到字母a的時候，本身主動機狀態可能跳轉到q1,q2,q3三個狀態。本身主動機又分爲有窮本身主動機和無窮本身主動機兩種，這裏再也不贅述。

有窮本身主動機可以用狀態圖直觀表示，樣例見下文中圖。

至於詞法分析的一些基本知識，簡單敘述一下：

-----------------------------------------------------------------------------------

定義：
詞法分析器的功能輸入源程序，依照構詞規則分解成一系列單詞符號。單詞是語言中具備獨立意義的最小單位，包含keyword、標識符、運算符、界符和常量等
(1) keyword 是由程序語言定義的具備固定意義的標識符。好比，Pascal 中的begin，end，if，while都是保留字。這些字一般不用做通常標識符。
(2) 標識符用來表示各類名字，如變量名，數組名，過程名等等。
(3) 常數常數的類型通常有整型、實型、布爾型、文字型等。
(4) 運算符如+、-、*、/等等。
(5) 界符如逗號、分號、括號、等等。
輸出：
詞法分析器所輸出單詞符號常常表示成例如如下的二元式：
(單詞種別，單詞符號的屬性值)
單詞種別通常用整數編碼。標識符通常統歸爲一種。常數則宜按類型（整、實、布爾等）分種。keyword可將其全體視爲一種。運算符可採用一符一種的方法。界符通常用一符一種的方法。對於每個單詞符號，除了給出了種別編碼以外，還應給出有關單詞符號的屬性信息。單詞符號的屬性是指單詞符號的特性或特徵。
演示樣例：
比方例如如下的代碼段：

while(i>=j) i--;

經詞法分析器處理後，它將被轉爲例如如下的單詞符號序列：

   <while, _>
   <(, _>
   <id, 指向i的符號表項的指針>
   <>=, _>
   <id, 指向j的符號表項的指針>
   <), _>
   <id, 指向i的符號表項的指針>
   <--, _>
   <;, _>

詞法分析分析器做爲一個獨立子程序：
詞法分析是編譯過程當中的一個階段，在語法分析前進行。詞法分析做爲一遍，可以簡化設計，改進編譯效率，添加�編譯系統的可移植性。也可以和語法分析結合在一塊兒做爲一遍，由語法分析程序調用詞法分析程序來得到當前單詞供語法分析使用。

-----------------------------------------------------------------------------------

我寫的這個詞法分析器，不是很是健全，尤爲是錯誤處理機制，像在字符串識別中，'ab'是C語言中不合法的char變量，但是個人詞法分析器不能推斷出錯誤，會死循環；此外，僅僅能識別出有限的keyword、有限形式的字符串（相信讀者看懂個人狀態機就知道哪裏有限了），由於時間不夠了，我不想再改了，如下貼出代碼，供你們參考。

對了，貼代碼以前，先說說個人詞法分析器的狀態機的設計。

我對「數字」的詞法分析用了一個狀態機，包含浮點數、整形數，狀態機例如如下：

對「字符（串）」的識別用了一個狀態機，包含keyword、char、以及char *，例如如下：

固然，對C語言的凝視的識別也用了一個狀態機，必須先把源代碼中的凝視cut掉才幹進行分析，例如如下：

我對運算符的識別（包含雙目和單目）沒有採用明顯的狀態機，都是直接分析推斷的，實際從某種意義上來說對它們的分析也是採用了狀態機的原理，僅僅是狀態機結構比較簡單，就沒再顯式用state表示，它們的狀態機實際上例如如下：

如下上代碼：

Scanner.py，做爲主模塊來運行：

'''
Created on 2012-10-18

@author: liushuai
'''
import string
import Category
import FileAccess

_currentIndex = 0
_Tokens = []
_prog = ""
_categoryNo = -1

_stateNumber = 0
_stateString = 0
_potentialNumber = ""
_potentialString = ""

def readComments(prog):
    '''Read the comments of a program'''
    state = 0
    currentIndex, beginIndex, endIndex = (0, 0, 0)
    commentsIndexs = []
    for c in prog:
        if state == 0:
            if c == '/':
                beginIndex = currentIndex
                state = 1
            else:
                pass
        elif state == 1:
            if c == '*':
                state = 2
            else :
                state = 0
        elif state == 2:
            if c == '*':
                state = 3
            else:
                pass
        elif state == 3:
            if c == '*':
                pass
            elif c == '/':
                endIndex = currentIndex
                commentsIndexs.append([beginIndex, endIndex])
                state = 0 #set 0 state
            else:
                state = 2
        currentIndex += 1
    return commentsIndexs
        
def cutComments(prog, commentsIndexs):
    '''cut the comments of the program prog'''
    num = len(commentsIndexs)
    if num == 0:
        return prog
    else :
        comments = []
        for i in xrange(num):
            comments.append(prog[commentsIndexs[i][0]:commentsIndexs[i][1] + 1])
        for item in comments:
            prog = prog.replace(item, "")
        return prog
    
def scan(helper):
    '''scan the program, and analysis it'''
    global _stateNumber, _stateString, _currentIndex, _Tokens, _prog, _categoryNo, _potentialNumber, _potentialString
    currentChar = _prog[_currentIndex]
    ######################################CHAR STRING######################################
    if currentChar == '\'' or currentChar == '\"' or currentChar in string.letters + "_$\\%\@"  or (currentChar in string.digits and _stateString != 0):
        if _stateString == 0:
            if currentChar == '\'':
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 1
                _currentIndex += 1
            elif currentChar == "\"":
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 2
                _currentIndex += 1
            elif currentChar in string.letters + "$_":
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 7
                _currentIndex += 1
            else:
                _currentIndex += 1
                _stateNumber = 10
        elif _stateString == 1:
            if currentChar in string.letters + "#$@%":
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 3
                _currentIndex += 1
            elif currentChar == '\\':
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 9
                _currentIndex += 1
            else:
                _currentIndex += 1
                _stateNumber = 10
        elif _stateString == 2:
            if currentChar in string.letters + "\\% ":
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 4
                _currentIndex += 1
            else:
                _currentIndex += 1
                _stateNumber = 10
        elif _stateString == 3:
            if currentChar == '\'':
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 5
                _currentIndex += 1
            else:
                _currentIndex += 1
                _stateNumber = 10
        elif _stateString == 4:
            if currentChar == '\"':
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 6
                _currentIndex += 1
            elif currentChar in string.letters + "\\% ":
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 4
                _currentIndex += 1
            else:
                _currentIndex += 1
                _stateNumber = 10
        elif _stateString == 7:
            if currentChar in string.digits + string.letters + "$_":
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 8
                _currentIndex += 1
            else:
                _currentIndex += 1
                _stateNumber = 10
        elif _stateString == 8:
            if currentChar in string.digits + string.letters + "$_":
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 8
                _currentIndex += 1
            else:
                _currentIndex += 1
                _stateNumber = 10
        elif _stateString == 9:
            if currentChar in ['b', 'n', 't', '\\', '\'', '\"']:
                _potentialString = "%s%s" % (_potentialString, currentChar)
                _stateString = 3
                _currentIndex += 1
            else:
                _currentIndex += 1
                _stateNumber = 10
        else:
            _currentIndex += 1
    ######################################  NUMBERS  ######################################
    elif currentChar in string.digits + ".":
        if _stateNumber == 0:
            if currentChar in "123456789":
                _potentialNumber = "%s%s" % (_potentialNumber, currentChar)
                _stateNumber = 6
                _currentIndex += 1
            elif currentChar == '0':
                _potentialNumber = "%s%s" % (_potentialNumber, currentChar)
                _stateNumber = 4
                _currentIndex += 1
            else:
                _stateNumber = 8
                _currentIndex += 1
        elif _stateNumber == 4:
            if currentChar == '.':
                _potentialNumber = "%s%s" % (_potentialNumber, currentChar)
                _stateNumber = 5
                _currentIndex += 1
            else:
                _stateNumber = 8
                _currentIndex += 1
        elif _stateNumber == 5:
            if currentChar in string.digits:
                _potentialNumber = "%s%s" % (_potentialNumber, currentChar)
                _stateNumber = 7
                _currentIndex += 1
            else:
                _stateNumber = 8
                _currentIndex += 1
        elif _stateNumber == 6: 
            if currentChar in string.digits:
                _potentialNumber = "%s%s" % (_potentialNumber, currentChar)
                _stateNumber = 6
                _currentIndex += 1
            elif currentChar == '.':
                _potentialNumber = "%s%s" % (_potentialNumber, currentChar)
                _stateNumber = 5
                _currentIndex += 1
            else:
                _stateNumber = 8
                _currentIndex += 1
        elif _stateNumber == 7:
            if currentChar in string.digits:
                _potentialNumber = "%s%s" % (_potentialNumber, currentChar)
                _stateNumber = 7
                _currentIndex += 1
            else:
                _stateNumber = 8
                _currentIndex += 1
        else:
            _currentIndex += 1
    ######################################OTEAR OPERATERS######################################
    else:
        if _stateNumber == 6 or _stateNumber == 4:
            helper.outPutToken(_potentialNumber, "INT", Category.IdentifierTable["INT"])
        elif _stateNumber == 7:
            helper.outPutToken(_potentialNumber, "FLOAT", Category.IdentifierTable["FLOAT"])
        elif _stateNumber != 0:
            helper.outPutToken("ERROR NUMBER", "None", "None")
        _stateNumber = 0
        _potentialNumber = ""
        
        if _stateString == 7 or _stateString == 8:
            if _potentialString in Category.KeyWordsTable:
                helper.outPutToken(_potentialString, _potentialString.upper(), Category.IdentifierTable[_potentialString.upper()])
            else:
                helper.outPutToken(_potentialString, "IDN" , Category.IdentifierTable["IDN"])
                helper.setSymbolTable(_potentialString, "IDN" , Category.IdentifierTable["IDN"])
        elif _stateString == 5:
            helper.outPutToken(_potentialString, "CHAR", Category.IdentifierTable["CHAR"])
        elif _stateString == 6:
            helper.outPutToken(_potentialString, "CHAR *", Category.IdentifierTable["CHAR *"])
        elif _stateString != 0:
            helper.outPutToken("ERROR STRING", "None", "None")
        _stateString = 0
        _potentialString = ""
        
        if currentChar == " ":
            _currentIndex += 1
        elif currentChar == '>':
            _currentIndex += 1
            currentChar = _prog[_currentIndex]
            if currentChar == "=":
                helper.outPutToken(">=", ">=", Category.IdentifierTable[">="])
                _currentIndex += 1
            else :
                helper.outPutToken(">", ">", Category.IdentifierTable[">"])
        elif currentChar == '<':
            _currentIndex += 1
            currentChar = _prog[_currentIndex]
            if currentChar == "=":
                helper.outPutToken("<=", "<=", Category.IdentifierTable["<="])
                _currentIndex += 1
            else :
                helper.outPutToken("<", "<", Category.IdentifierTable["<"])
        elif currentChar == '+':
            _currentIndex += 1
            currentChar = _prog[_currentIndex]
            if currentChar == '+':
                helper.outPutToken("++", "++", Category.IdentifierTable["++"])
                _currentIndex += 1
            else :
                helper.outPutToken("+", "+", Category.IdentifierTable["+"])
        elif currentChar == '-':
            _currentIndex += 1
            currentChar = _prog[_currentIndex]
            if currentChar == '-':
                helper.outPutToken("--", "--", Category.IdentifierTable["--"])
            else:
                helper.outPutToken("-", "-", Category.IdentifierTable["-"])
        elif currentChar == '=':
            _currentIndex += 1
            currentChar = _prog[_currentIndex]
            if currentChar == '=':
                helper.outPutToken("==", "==", Category.IdentifierTable["=="])
                _currentIndex += 1
            else :
                helper.outPutToken("=", "=", Category.IdentifierTable["="])
        elif currentChar == '!':
            _currentIndex += 1
            currentChar = _prog[_currentIndex]
            if currentChar == '=':
                helper.outPutToken("!=", "!=", Category.IdentifierTable["!="])
                _currentIndex += 1
            else :
                helper.outPutToken("!", "!", Category.IdentifierTable["!"])  
        elif currentChar == '&':
            _currentIndex += 1
            currentChar = _prog[_currentIndex]
            if currentChar == '&':
                helper.outPutToken("&&", "&&", Category.IdentifierTable["&&"])
                _currentIndex += 1
            else :
                helper.outPutToken("&", "&", Category.IdentifierTable["&"])
        elif currentChar == '|':
            _currentIndex += 1
            currentChar = _prog[_currentIndex]
            if currentChar == '|':
                helper.outPutToken("||", "||", Category.IdentifierTable["||"])
                _currentIndex += 1
            else :
                helper.outPutToken("|", "|", Category.IdentifierTable["||"])    
        elif currentChar == '*':
            helper.outPutToken("*", "*", Category.IdentifierTable["*"])
            _currentIndex += 1
        elif currentChar == '/':
            helper.outPutToken("/", "/", Category.IdentifierTable["/"])
            _currentIndex += 1
        elif currentChar == ';':
            helper.outPutToken(";", ";", Category.IdentifierTable[";"])
            _currentIndex += 1
        elif currentChar == ",":
            helper.outPutToken(",", ",", Category.IdentifierTable[","])
            _currentIndex += 1
        elif currentChar == '{':
            helper.outPutToken("{", "{", Category.IdentifierTable["{"])
            _currentIndex += 1
        elif currentChar == '}':
            helper.outPutToken("}", "}", Category.IdentifierTable["}"])
            _currentIndex += 1
        elif currentChar == '[':
            helper.outPutToken("[", "[", Category.IdentifierTable["["])
            _currentIndex += 1
        elif currentChar == ']':
            helper.outPutToken("]", "]", Category.IdentifierTable["]"])
            _currentIndex += 1
        elif currentChar == '(':
            helper.outPutToken("(", "(", Category.IdentifierTable["("])
            _currentIndex += 1
        elif currentChar == ')':
            helper.outPutToken(")", ")", Category.IdentifierTable[")"])
            _currentIndex += 1
            

if __name__ == '__main__':
    helper = FileAccess.FileHelper("H://test.c", "H://token.txt", "H://symbol_table.txt")
    prog = helper.readProg()
    print prog
    comments = readComments(prog)
    _prog = cutComments(prog, comments)
    print _prog
    while _currentIndex < len(_prog):
        scan(helper)
    helper.closeFiles()

Category.py，這個模塊裏面定義了一些C語言中的keyword、運算符等等，是種別碼錶：

'''
Created on 2012-10-18

@author: liushuai
'''
IdentifierTable = {"INT":1,"FLOAT":2,"CHAR":3,"IDN":4,"WHILE":5,"FOR":6,"DO":7,"BREAK":31,"CONTINUE":32,"CHAR *":33,"IF":37,
                   "*":8,"/":9,"+":10,"-":11,">":12,"<":13,"=":14,
                   "++":15,"--":16,"==":17,"!=":18,">=":19,"<=":20,
                   "&&":28,"||":29,"!":30,"&":35,"|":36,
                   ";":21,",":34,
                   "{":22,"}":23,"[":24,"]":25,"(":26,")":27}
KeyWordsTable = ("int", "float", "char", "while", "for", "do","break","continue","char *","if")

FileAccess.py，裏面定義了一個類用來進行對文件的操做，好比輸出Token序列以及符號表到文件：

'''
Created on 2012-10-23

@author: liushuai
'''

import Category

class FileHelper(object):
    def __init__(self,progPath,tokenPath,symbolTablePath):
        self.progPath = progPath
        self.tokenPath = tokenPath
        self.symbolTablePath = symbolTablePath
        
        self.tokenFp = open(self.tokenPath,"w")
        self.symbolTableFp = open(self.symbolTablePath,"w")
        
        self.symbolTable = {}.fromkeys(Category.KeyWordsTable) #initialize symbol table
        print self.symbolTable

    def readProg(self):
        '''read the program into the RAM'''
        fp = open(self.progPath, "r+")
        prog = ""
        for eachLine in fp.readlines():
            #print eachLine 
            prog = "%s%s" % (prog, eachLine.strip())
        fp.close()
        return prog

    def outPutToken(self,tokenSelf,tokenInner,tokenNo):
        '''output token into a file'''
        self.tokenFp.write("(" + tokenInner + "," + tokenSelf + ")" + "\n")
        print "(" + tokenInner + "," + tokenSelf + ")"
        
    def setSymbolTable(self,tokenSelf,tokenInner,tokenNo):
        '''output symbol into symbol table'''
        if not self.symbolTable.has_key(tokenSelf):
            self.symbolTable[tokenSelf] = None
    
    def writeSymbolToFile(self):
        for k in self.symbolTable:
            self.symbolTableFp.write(k + "\n")
            
    def closeFiles(self):
        '''close token Files'''
        self.writeSymbolToFile()
        if self.tokenFp != None:
            self.tokenFp.close()
        if self.symbolTableFp != None:
            self.symbolTableFp.close()

如下是個人測試C語言程序：

int main ()
{
    char  str[10000];
    int num[30]={0};
    char std[28]={"abcdefghijklmnopqrstuvwxyz"};
    int i,j,temp;
    float test=-0.34;
    float test1=23.45;
    
    char tom;
    for (i=0;i<1000;i++)             /*字符的讀入*/
    {
        str[i]=getchar();
        if (str[i]=='\n')
        break;
    }
    for (i=0;i<10000;i++)             /*字符的統計數量*/
    {
        if (str[i]=='\n')
        break;
        j=str[i] - 97; /*-97 or sth - 97?*/
        num[j]++;
    }
    for (i=0;i<27;i++)              /*字符的按出現頻率排序*/
    {
        for (j=i+1;j<26;j++)
        {
            if (num[j]>num[i])
            {
                temp=num[j];
                num[j]=num[i];
                num[i]=temp;

                tom=std[j];
                std[j]=std[i];
                std[i]=tom;
            }
        }
    }
    for (i=0;i<27;i++)                     /*字符的按字母表順序排序*/
    {
        for (j=i+1;j<26;j++)
        {
            if (num[i]==num[j])
            {
                if (std[j]<std[i])
                {
                    tom=std[j];
                    std[j]=std[i];
                    std[i]=tom;
                }
            }
        }
    }
    for (i=0;i<29;i++)
    {
        if (num[i]==0)
        break;
    }

}

最後腳本輸出例如如下：

Token.txt:

Symbol_table.txt:

此次實驗就算作完了。

儘管完畢的很是水，但是正確狀況下的輸出仍是使人愜意的，詞法分析完過後，等着句法分析調用它的輸出結果吧。對了，至於符號表在整個編譯階段的做用，龍書初版2.4節和6.7節有介紹，請讀者查閱。

寫的很是水很是水，若有不足，歡迎指正。