转录因子结合位点预测

今天想要分析转录因子结合位点的数据，使用http://gene-regulation.com/cgi-bin/pub/programs/patch/bin/patch.cgi网站进行预测，但由于该网站不提供文件上传的功能，同时每次进行处理的数据量有上限，想到我又得点击几百次鼠标的操作就头大；于是参考别人的博客写下一个爬虫

脚本下载地址（https://github.com/zpliu1126/booknote/raw/master/Script/patch.tar.gz)

用法

依赖python3，需要安装tqdm第三方库用于显示进度条

//第一步使用login.py获取cookie文件
python3 login.py >cookie.txt
//第二步使用patch.py文件进行POSt请求爬取数据
python patch.py 基因fasta序列文件 输出结果文件

​ :warning:注意的是patch.py文件中需要指定cookie.txt文件的路径

  • :jack_o_lantern: 首先使用python库urllib、http.cookiejar来登录网站

    import urllib.error, urllib.request, urllib.parse
import http.cookiejar

LOGIN_URL = 'http://gene-regulation.com/login'
values = {'user':'账号','password':'密码'}
postdata = urllib.parse.urlencode(values).encode()
user_agent = r'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36' \
             r' (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36'
headers = {'User-Agent':user_agent, 'Connection':'keep-alive'}
#将cookie保存在本地，并命名为cookie.txt
cookie_filename = 'cookie.txt'
cookie_aff = http.cookiejar.MozillaCookieJar(cookie_filename)
handler = urllib.request.HTTPCookieProcessor(cookie_aff)
opener = urllib.request.build_opener(handler)

request = urllib.request.Request(LOGIN_URL, postdata, headers)
try:
    response = opener.open(request)
except urllib.error.URLError as e:
    print(e.reason)

cookie_aff.save(ignore_discard=True, ignore_expires=True) 
# 保存信息到cookie中

  • :checkered_flag:使用cookie文件信息进行登录

    #get_url为使用cookie所登陆的网址，该网址必须先登录才可
get_url = 'http://gene-regulation.com/cgi-bin/pub/programs/patch/bin/patch.cgi'
# 使用cookie文件进行登录
cookie_filename = 'cookie.txt'
cookie_aff = http.cookiejar.MozillaCookieJar(cookie_filename)
cookie_aff.load(cookie_filename,ignore_discard=True,ignore_expires=True)
handler = urllib.request.HTTPCookieProcessor(cookie_aff)
opener = urllib.request.build_opener(handler)
#构造请求头，伪装成浏览器
user_agent = r'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36' \
             r' (KHTML, like Gecko) Chrome/61.0.3163.79 Safari/537.36'
headers = {'User-Agent':user_agent, 'Connection':'keep-alive'}
# 构造post请求post表单
searchvalue={"Status": "First",
            'searchName':'default',
                'usr_seq':'default',
                "seqStat": "DEL",
                'sequenceName':'default.seq',
                "site_opt": "OUR",
                'group':'plants',
                'minLen': 8,
                'mismatch':1,
                'penalty':100,
                'boundary':87.5}
#初始化序列信息
searchvalue['theSequence']=''

  • :carousel_horse:关于post请求如何查看键值对

  • :diamond_shape_with_a_dot_inside: ​开始从文件中读取数据进行post请求

    #读取基因序列信息，赋值给字典                
genelist=fastaread(sys.argv[1])
#创建输出文件句柄
patchout=open(sys.argv[2],'w')
# 用于记录序列数目
flag=0
# 循环添加序列信息
for gene in tqdm(genelist.keys(),desc="request is doing"):
    flag+=1
    #拼接字符串操作
    searchvalue['theSequence']='%s%s%s%s%s%s' % (searchvalue['theSequence'],">",gene," \n",genelist[gene],"\n")
    # 每隔200个序列发起一次请求，但是最后还会剩下不能够整除200的一些序列
    if(flag%200==0):
        # 对post内容进行编码
        searchtdata = urllib.parse.urlencode(searchvalue).encode()
        #使用cookie登陆get_url
        get_request = urllib.request.Request(get_url,searchtdata,headers=headers)
        get_response = opener.open(get_request)
        # 创建 BeautifulSoup对象
        soup=BeautifulSoup(get_response.read().decode(),features="html.parser")
        #BeautifulSoup 解析说明文档https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh
        for form in  soup.find_all("form",action="/cgi-bin/pub/programs/patch/bin/files.cgi"):
            #查找所有的提交表单
            if(form.previous_sibling.previous_sibling.previous_sibling.string!=None):
                patchout.write(">"+form.previous_sibling.previous_sibling.previous_sibling.string+"\n")
            else:
                print("当前的"+flag+"存在没有的结果！")
            # 根据兄弟节点获得每个序列id和结果信息
            pre=form.next_sibling.next_sibling
            # 如果没有结果，它的string就为no result find
            if(pre.string!=None):
                patchout.write(pre.string+"\n")
            else:
                for string in pre.strings:
                    patchout.write(string)
                patchout.write("\n")
        searchvalue['theSequence']=''
        #推迟1s，发送请求
        time.sleep(1);
        # 当只剩下最后89不能够满足整除的条件时
    elif(flag==len(genelist)):
        # 对post内容进行编码
        searchtdata = urllib.parse.urlencode(searchvalue).encode()
        #使用cookie登陆get_url
        get_request = urllib.request.Request(get_url,searchtdata,headers=headers)
        get_response = opener.open(get_request)
        # 创建 BeautifulSoup对象
        soup=BeautifulSoup(get_response.read().decode(),features="html.parser")
        #BeautifulSoup 解析说明文档https://www.crummy.com/software/BeautifulSoup/bs4/doc.zh

  • :bride_with_veil:对HTML文件进行解析

    for form in  soup.find_all("form",action="/cgi-bin/pub/programs/patch/bin/files.cgi"):
            #查找所有的提交表单
            patchout.write(">"+form.previous_sibling.previous_sibling.previous_sibling.string+"\n")
            # 根据兄弟节点获得每个序列id和结果信息
            pre=form.next_sibling.next_sibling
            # 如果没有结果，它的string就为no result find
            if(pre.string!=None):
                patchout.write(pre.string+"\n")
            else:
                for string in pre.strings:
                    patchout.write(string)
                patchout.write("\n")
        searchvalue['theSequence']=''
        #推迟1s，发送请求
        time.sleep(1);
    else:
        continue
patchout.close()

  • 将解析得到的网络文件，转化为一定的格式

    sed 's/Scanning sequence: //g'  Ga_gene.txt| awk -F "  " '{print $1}'|awk 'NR==1{flag=substr($1,2)}'
'{a[NR]=$1}'
'END{
    print flag;
    for(i=2;i<=NR-1;i++){
    if(a[i]~/^>/){
        flag=substr(a[i],2);
        print flag;}
    else if(a[i]==""){
            print}
    else{
    print flag
        }}}'|paste  - Ga_gene.txt -d "\t"|sed 's/\(\s\s\)\+/\t/g'

  • 转换格式后的文件

    转换格式后

PreviousScriptNextBinomTest

Last updated