Home >  > 如何求语词的相关性(PMI)

如何求语词的相关性(PMI)

0

待处理文本:

郭富城方发声明否认炒作,真的是这样吗?我发布了头条文章:《罗生门:六小龄童上春晚究竟谁炒作?从这开8并怀念当年那美好的西游》

除夕还剩几天时间,这几天“猴哥”#六小龄童#突然成为网络热搜词,朋友圈全是“帮猴哥上春晚”的帖子。在充当“自来水”的同时,有人开始反思,“大圣”热闹上春晚话题是不是一种“情怀绑架”?有网友说“这年头,似乎打上情怀的标签,都能把一个看似毫无逻辑的事,套上隆重的仪式感,煽动起某种情绪。

​呼吁六小龄童老师上春晚是好事,是人人盼望之事,但是不要被人牵着鼻子走,像央视张立,借着六小龄童老师炒作自己,中伤他人,这对章老师本人也不是益事,更不会促进什么期望实现。

发布了头条文章:《六小龄童,你真的不配上春晚!》 这两天大家都在为六小龄童鸣不平,咱说句公道话,六小龄童,你不配上春晚!别急着拍砖,听我细细道来!〔希望能静下来看完内容再评论〕[微笑]

凡事物极必反,现在给央视的舆论压力足够大了,甚至有点过了,而且我觉得现在最大的受害者还是六小龄童老师,不管最后上不上春晚,六小龄童老师现在估计都十分尴尬。大家可以稍微缓一缓,毕竟春晚的不可预测性是有太多先例的,04年那次六小龄童老师还是腊月28才接到通知临时进组的呢。[doge]

#热点#【六小龄童:如果上不了我就在家看春晚】“其实我完全没有料到,观众会对我上春晚的事这么轰动。我很感动,很感谢。除夕如果上(春晚),那就去给大家拜年,让大家高兴。如果没有上,我会和我的家人一起看春晚,继续支持央视春节联欢晚会。”昨日,@六小龄童 接受采访时说。

​#支付宝集福#怎么了?全国人都在找敬业福,支付宝红包口令?还是希望看到#六小龄童上春晚#

如何看待有的人因为六小龄童老师没上春晚就骂tfboys

相关代码:

#coding=utf-8
import jieba
import jieba.analyse
import jieba.posseg as pseg
import re

class Extractkeys:
 
    def removeEmoji(self, sentence):
        return re.sub('\[.*?\]', '', sentence)

    def CutWithPartOfSpeech(self, sentence):
        sentence = self.removeEmoji(sentence)
        words =jieba.cut(sentence)
        wordlist=[]
        for word in words:
            wordlist.append(word)
        return wordlist

    def ExtractWord(self,wordlist):
        sentence = ','.join(wordlist)
        words = jieba.analyse.extract_tags(sentence,5)
        wordlist = []
        for w in words:
            wordlist.append(w)
        return wordlist

    def RemoveStopWord(self,wordlist):
        stopWords = self.GetStopWords()
        keywords = []
        for word in wordlist:
            if word not in stopWords:
                keywords.append(word)
        return keywords

def extract(text):
    ek = Extractkeys()
    wordlist = ek.CutWithPartOfSpeech(text)
    extractwords = ek.ExtractWord(wordlist)
    return extractwords
# coding=utf-8

class PMI:
    def __init__(self, document):
        self.document = document
        self.pmi = {}
        self.miniprobability = float(1.0) / document.__len__()
        self.minitogether = float(0)/ document.__len__()
        self.set_word = self.getset_word()

    def calcularprobability(self, document, wordlist):

        """
        :param document:
        :param wordlist:
        :function : 计算单词的document frequency
        :return: document frequency
        """

        total = document.__len__()
        number = 0
        for doc in document:
            if set(wordlist).issubset(doc):
                number += 1
        percent = float(number)/total
        return percent

    def togetherprobablity(self, document, wordlist1, wordlist2):

        """
        :param document:
        :param wordlist1:
        :param wordlist2:
        :function: 计算单词的共现概率
        :return:共现概率
        """

        joinwordlist = wordlist1 + wordlist2
        percent = self.calcularprobability(document, joinwordlist)
        return percent

    def getset_word(self):

        """
        :function: 得到document中的词语词典
        :return: 词语词典
        """
        list_word = []
        for doc in self.document:
            list_word = list_word + list(doc)
        set_word = []
        for w in list_word:
            if set_word.count(w) == 0:
                set_word.append(w)
        return set_word

    def get_dict_frq_word(self):

        """
        :function: 对词典进行剪枝,剪去出现频率较少的单词
        :return: 剪枝后的词典
        """
        dict_frq_word = {}
        for i in range(0, self.set_word.__len__(), 1):
            list_word=[]
            list_word.append(self.set_word[i])
            probability = self.calcularprobability(self.document, list_word)
            if probability > self.miniprobability:
                dict_frq_word[self.set_word[i]] = probability
        return dict_frq_word

    def calculate_nmi(self, joinpercent, wordpercent1, wordpercent2):
        """
        function: 计算词语共现的nmi值
        :param joinpercent:
        :param wordpercent1:
        :param wordpercent2:
        :return:nmi
        """
        return (joinpercent)/(wordpercent1*wordpercent2)

    def get_pmi(self):
        """
        function:返回符合阈值的pmi列表
        :return:pmi列表
        """
        dict_pmi = {}
        dict_frq_word = self.get_dict_frq_word()
        print dict_frq_word
        for word1 in dict_frq_word:
            wordpercent1 = dict_frq_word[word1]
            for word2 in dict_frq_word:
                if word1 == word2:
                    continue
                wordpercent2 = dict_frq_word[word2]
                list_together=[]
                list_together.append(word1)
                list_together.append(word2)
                together_probability = self.calcularprobability(self.document, list_together)
                if together_probability > self.minitogether:
                    string = word1 + ',' + word2
                    dict_pmi[string] = self.calculate_nmi(together_probability, wordpercent1, wordpercent2)
        return dict_pmi

最后一个:

#coding=utf-8
__author__ = 'root'
from PMI import *
import os
from extract import extract

if __name__ == '__main__':
    documents = []
    testfile = 'data.txt'
    f = open(testfile, 'r')
    data = f.readlines()
    if data is not None:
        for sentences in data:
            extractwords = []
            words = extract(sentences)
            for word in words:
                extractwords.append(word)
            documents.append(set(extractwords))
    pm = PMI(documents)
    pmi = pm.get_pmi()
    for p in pmi:
        if pmi[p] > 1.5:
            print p, pmi[p], '\n'

执行结果:

来源:https://blog.csdn.net/qq_30843221/article/details/50767590

本文暂无标签

发表评论

*

*