shell bypass 403
3
B;�W�~ � ) @ s� d dl mZmZmZ d dlmZmZ d dlmZm Z d dl
Z
d dl Z d dl Z ddl
mZmZmZmZ ddl
mZ ddlmZ d d lmZ yd d
lmZ W n ek
r� eZY nX ed d � eD ��Zed
d � eD ��Zedd � eD ��Z eeddg�B Z dZ ej �r$e j e ddE� e!d� d �Z"n
e j e �Z"e#ddddddddd d d d d d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3g �Z$e j d4�Z%i Z&G d5d6� d6e'�Z(d7d8� Z)G d9d:� d:e'�Z*G d;d<� d<e*�Z+G d=d>� d>e,�Z-G d?d@� d@e'�Z.G dAdB� dBe'�Z/dCdD� Z0dS )F� )�absolute_import�division�unicode_literals)� text_type� binary_type)� http_client�urllibN� )�EOF�spaceCharacters� asciiLetters�asciiUppercase)�ReparseException)�_utils)�StringIO)�BytesIOc C s g | ]}|j d ��qS )�ascii)�encode)�.0�item� r �"/usr/lib/python3.6/_inputstream.py�
<listcomp> s r c C s g | ]}|j d ��qS )r )r )r r r r r r s c C s g | ]}|j d ��qS )r )r )r r r r r r s � >� <u� [- - --]z"\uD800-\uDFFF"�]i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i�� i��
i��
i�� i�� i�� i�� i��
i��
i�� i�� i�� i�� i�� i�� z[ -
-/:-@[-`{-~]c @ sH e Zd ZdZdd� Zdd� Zdd� Zdd � Zd
d � Zd d
� Z dd� Z
dS )�BufferedStreamz�Buffering for streams that do not have buffering of their own
The buffer is implemented as a list of chunks on the assumption that
joining many strings will be slow since it is O(n**2)
c C s || _ g | _ddg| _d S )Nr r ���)�stream�buffer�position)�selfr r r r �__init__@ s zBufferedStream.__init__c C s@ d}x(| j d | jd � D ]}|t|�7 }qW || jd 7 }|S )Nr r )r r �len)r! �pos�chunkr r r �tellE s
zBufferedStream.tellc C sH |}d}x0t | j| �|k r8|t | j| �8 }|d7 }q
W ||g| _d S )Nr r )r# r r )r! r$ �offset�ir r r �seekL s zBufferedStream.seekc C sT | j s| j|�S | jd t| j �krF| jd t| j d �krF| j|�S | j|�S d S )Nr r r )r � _readStreamr r# �_readFromBuffer)r! �bytesr r r �readU s
zBufferedStream.readc C s t dd� | jD ��S )Nc S s g | ] }t |��qS r )r# )r r r r r r _ s z1BufferedStream._bufferedBytes.<locals>.<listcomp>)�sumr )r! r r r �_bufferedBytes^ s z BufferedStream._bufferedBytesc C s< | j j|�}| jj|� | jd d7 < t|�| jd< |S )Nr r )r r- r �appendr r# )r! r, �datar r r r* a s
zBufferedStream._readStreamc C s� |}g }| j d }| j d }x�|t| j�k r�|dkr�| j| }|t|�| krb|}||| g| _ n"t|�| }|t|�g| _ |d7 }|j|||| � � ||8 }d}q W |r�|j| j|�� dj|�S )Nr r � )r r# r r0 r* �join)r! r, ZremainingBytes�rvZ bufferIndexZ bufferOffsetZ bufferedDataZ bytesToReadr r r r+ h s$
z BufferedStream._readFromBufferN) �__name__�
__module__� __qualname__�__doc__r" r& r) r- r/ r* r+ r r r r r 9 s r c K s� t | tj�s(t | tjj�r.t | jtj�r.d}n&t| d�rJt | jd�t �}n
t | t �}|r�dd� |D �}|rvt
d| ��t | f|�S t | f|�S d S )NFr- r c S s g | ]}|j d �r|�qS )Z _encoding)�endswith)r �xr r r r � s z#HTMLInputStream.<locals>.<listcomp>z3Cannot set an encoding with a unicode input, set %r)
�
isinstancer Z HTTPResponser ZresponseZaddbase�fp�hasattrr- r � TypeError�HTMLUnicodeInputStream�HTMLBinaryInputStream)�source�kwargsZ isUnicodeZ encodingsr r r �HTMLInputStream� s
rC c @ sp e Zd ZdZdZdd� Zdd� Zdd� Zd d
� Zd d � Z d
d� Z
ddd�Z dd� Z dd� Z
d dd�Zdd� ZdS ) r? z�Provides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
i ( c C sZ t jsd| _n td�dkr$| j| _n| j| _dg| _td�df| _| j |�| _
| j � dS )a� Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
Nu r r zutf-8�certain) r �supports_lone_surrogates�reportCharacterErrorsr# �characterErrorsUCS4�characterErrorsUCS2ZnewLines�lookupEncoding� charEncoding�
openStream�
dataStream�reset)r! rA r r r r" � s
z HTMLUnicodeInputStream.__init__c C s. d| _ d| _d| _g | _d| _d| _d | _d S )N� r )r% � chunkSize� chunkOffset�errors� prevNumLines� prevNumCols�_bufferedCharacter)r! r r r rM � s z HTMLUnicodeInputStream.resetc C s t |d�r|}nt|�}|S )zvProduces a file object from source.
source can be either a file object, local filename or a string.
r- )r= r )r! rA r r r r rK � s
z!HTMLUnicodeInputStream.openStreamc C sT | j }|jdd|�}| j| }|jdd|�}|dkr@| j| }n ||d }||fS )N�
r r r )r% �countrR �rfindrS )r! r' r% ZnLinesZ positionLineZ lastLinePosZpositionColumnr r r � _position� s
z HTMLUnicodeInputStream._positionc C s | j | j�\}}|d |fS )z:Returns (line, col) of the current position in the stream.r )rX rP )r! �line�colr r r r � s z HTMLUnicodeInputStream.positionc C s6 | j | jkr| j� stS | j }| j| }|d | _ |S )zo Read one character from the stream or queue if available. Return
EOF when EOF is reached.
r )rP rO � readChunkr
r% )r! rP �charr r r r\ � s
zHTMLUnicodeInputStream.charNc C s� |d kr| j }| j| j�\| _| _d| _d| _d| _| jj|�}| j rX| j | }d | _ n|s`dS t
|�dkr�t |d �}|dks�d| ko�dkn r�|d
| _ |d d� }| j r�| j |� |j
dd �}|j
d
d �}|| _t
|�| _d S )NrN r Fr �
i � i�� z
rU �
Tr r r )�_defaultChunkSizerX rO rR rS r% rP rL r- rT r# �ordrF �replace)r! rO r1 Zlastvr r r r[ � s0
z HTMLUnicodeInputStream.readChunkc C s, x&t ttj|���D ]}| jjd� qW d S )Nzinvalid-codepoint)�ranger# �invalid_unicode_re�findallrQ r0 )r! r1 �_r r r rG % s z*HTMLUnicodeInputStream.characterErrorsUCS4c C s� d}x�t j|�D ]�}|rqt|j� �}|j� }tj|||d � �rttj|||d � �}|tkrn| j j
d� d}q|dkr�|dkr�|t |�d kr�| j j
d� qd}| j j
d� qW d S )NF� zinvalid-codepointTi � i�� r ) rc �finditerr` �group�startr ZisSurrogatePairZsurrogatePairToCodepoint�non_bmp_invalid_codepointsrQ r0 r# )r! r1 �skip�matchZ codepointr$ Zchar_valr r r rH ) s z*HTMLUnicodeInputStream.characterErrorsUCS2Fc C s� yt ||f }W nN tk
r^ djdd� |D ��}|s@d| }tjd| � }t ||f< Y nX g }x||j| j| j�}|dkr�| j| jkr�P n0|j � }|| jkr�|j
| j| j|� � || _P |j
| j| jd� � | j � sfP qfW dj|�}|S )z� Returns a string of characters from the stream up to but not
including any character in 'characters' or EOF. 'characters' must be
a container that supports the 'in' method and iteration over its
characters.
rN c S s g | ]}d t |� �qS )z\x%02x)r` )r �cr r r r N s z5HTMLUnicodeInputStream.charsUntil.<locals>.<listcomp>z^%sz[%s]+N) �charsUntilRegEx�KeyErrorr3 �re�compilerl r% rP rO �endr0 r[ ) r! Z
charactersZopposite�charsZregexr4 �mrr �rr r r �
charsUntil@ s.
z!HTMLUnicodeInputStream.charsUntilc C s@ |d k r<| j dkr.|| j | _| jd7 _n| j d8 _ d S )Nr r )rP r% rO )r! r\ r r r �ungeto s
z HTMLUnicodeInputStream.unget)N)F)r5 r6 r7 r8 r_ r" rM rK rX r r\ r[ rG rH rv rw r r r r r? � s
&
/r? c @ sL e Zd ZdZddd�Zdd� Zd d
� Zdd d �Zd
d� Zdd� Z dd� Z
dS )r@ z�Provides a unicode stream of characters to the HTMLTokenizer.
This class takes care of character encoding and removing or replacing
incorrect byte-sequences and also provides column and line tracking.
N� windows-1252Tc C s\ | j |�| _tj| | j� d| _d| _|| _|| _|| _|| _ || _
| j |�| _ | j
� dS )a� Initialises the HTMLInputStream.
HTMLInputStream(source, [encoding]) -> Normalized stream from source
for use by html5lib.
source can be either a file-object, local filename or a string.
The optional encoding parameter must be a string that indicates
the encoding. If specified, that encoding will be used,
regardless of any BOM or later declaration (such as in a meta
element)
i �d N)rK � rawStreamr? r" � numBytesMeta�numBytesChardet�override_encoding�transport_encoding�same_origin_parent_encoding�likely_encoding�default_encoding�determineEncodingrJ rM )r! rA r} r~ r r� r� Z
useChardetr r r r" � s z HTMLBinaryInputStream.__init__c C s& | j d jj| jd�| _tj| � d S )Nr ra )rJ Z
codec_info� streamreaderrz rL r? rM )r! r r r rM � s zHTMLBinaryInputStream.resetc C sD t |d�r|}nt|�}y|j|j� � W n t|�}Y nX |S )zvProduces a file object from source.
source can be either a file object, local filename or a string.
r- )r= r r) r& r )r! rA r r r r rK � s
z HTMLBinaryInputStream.openStreamc C s� | j � df}|d d k r |S t| j�df}|d d k r:|S t| j�df}|d d k rX|S | j� df}|d d k rt|S t| j�df}|d d k r�|d jjd� r�|S t| j�df}|d d k r�|S |�rdyddl m
} W n t k
r� Y nxX g }|� }x6|j �s.| j
j| j�}|�sP |j|� |j|� q�W |j� t|jd �}| j
jd� |d k �rd|dfS t| j�df}|d d k �r�|S td�dfS )NrD r Z tentativezutf-16)�UniversalDetector�encodingz windows-1252)� detectBOMrI r} r~ �detectEncodingMetar �name�
startswithr� Zchardet.universaldetectorr� � ImportError�donerz r- r| r0 Zfeed�close�resultr) r� )r! ZchardetrJ r� ZbuffersZdetectorr r� r r r r� � sP
z'HTMLBinaryInputStream.determineEncodingc C s� t |�}|d krd S |jdkr(t d�}nT|| jd krH| jd df| _n4| jjd� |df| _| j� td| jd |f ��d S )N�utf-16be�utf-16lezutf-8r rD z Encoding changed from %s to %s)r� r� )rI r� rJ rz r) rM r )r! Z newEncodingr r r �changeEncoding s
z$HTMLBinaryInputStream.changeEncodingc
C s� t jdt jdt jdt jdt jdi}| jjd�}|j|dd� �}d}|sp|j|�}d}|sp|j|dd � �}d }|r�| jj |� t
|�S | jj d
� dS dS ) z�Attempts to detect at BOM at the start of the stream. If
an encoding can be determined from the BOM return the name of the
encoding otherwise return Nonezutf-8zutf-16lezutf-16bezutf-32lezutf-32be� N� rf r ) �codecs�BOM_UTF8� BOM_UTF16_LE� BOM_UTF16_BE� BOM_UTF32_LE� BOM_UTF32_BErz r- �getr) rI )r! ZbomDict�stringr� r) r r r r� s"
z HTMLBinaryInputStream.detectBOMc C sH | j j| j�}t|�}| j jd� |j� }|dk rD|jdkrDtd�}|S )z9Report the encoding declared by the meta element
r N�utf-16be�utf-16lezutf-8)r� r� )rz r- r{ �EncodingParserr) � getEncodingr� rI )r! r �parserr� r r r r� 9 s z(HTMLBinaryInputStream.detectEncodingMeta)NNNNrx T)T) r5 r6 r7 r8 r" rM rK r� r� r� r� r r r r r@ � s
(
>"r@ c @ s� e Zd ZdZdd� Zdd� Zdd� Zdd � Zd
d � Zd d
� Z dd� Z
dd� Z e e e
�Z
dd� Ze e�Zefdd�Zdd� Zdd� Zdd� Zd S ) �
EncodingBytesz�String-like object with an associated position and various extra methods
If the position is ever greater than the string length then an exception is
raisedc C s t j| |j� �S )N)r, �__new__�lower)r! �valuer r r r� L s zEncodingBytes.__new__c C s
d| _ d S )Nr r )rX )r! r� r r r r" P s zEncodingBytes.__init__c C s | S )Nr )r! r r r �__iter__T s zEncodingBytes.__iter__c C s> | j d }| _ |t| �kr"t�n |dk r.t�| ||d � S )Nr r )rX r# �
StopIterationr> )r! �pr r r �__next__W s zEncodingBytes.__next__c C s | j � S )N)r� )r! r r r �next_ s zEncodingBytes.nextc C sB | j }|t| �krt�n |dk r$t�|d | _ }| ||d � S )Nr r )rX r# r� r> )r! r� r r r �previousc s zEncodingBytes.previousc C s | j t| �krt�|| _ d S )N)rX r# r� )r! r r r r � setPositionl s zEncodingBytes.setPositionc C s* | j t| �krt�| j dkr"| j S d S d S )Nr )rX r# r� )r! r r r � getPositionq s
zEncodingBytes.getPositionc C s | | j | j d � S )Nr )r )r! r r r �getCurrentByte{ s z EncodingBytes.getCurrentBytec C sL | j }x:|t| �k r@| ||d � }||kr6|| _|S |d7 }qW || _dS )z Skip past a list of charactersr N)r r# rX )r! rs r� rm r r r rk � s zEncodingBytes.skipc C sL | j }x:|t| �k r@| ||d � }||kr6|| _|S |d7 }qW || _d S )Nr )r r# rX )r! rs r� rm r r r � skipUntil� s zEncodingBytes.skipUntilc C s> | j }| ||t|� � }|j|�}|r:| j t|�7 _ |S )z�Look for a sequence of bytes at the start of a string. If the bytes
are found return True and advance the position to the byte after the
match. Otherwise return False and leave the position alone)r r# r� )r! r, r� r1 r4 r r r �
matchBytes� s
zEncodingBytes.matchBytesc C sR | | j d� j|�}|dkrJ| jdkr,d| _| j|t|� d 7 _dS t�dS )z�Look for the next sequence of bytes matching a given sequence. If
a match is found advance the position to the last byte of the matchNr r Tr r )r �findrX r# r� )r! r, Z newPositionr r r �jumpTo� s
zEncodingBytes.jumpToN)r5 r6 r7 r8 r� r" r� r� r� r� r� r� �propertyr r� � currentByte�spaceCharactersBytesrk r� r� r� r r r r r� H s
r� c @ sX e Zd ZdZdd� Zdd� Zdd� Zdd � Zd
d � Zd d
� Z dd� Z
dd� Z dd� Z dS )r� z?Mini parser for detecting character encoding from meta elementsc C s t |�| _d| _dS )z3string - the data to work on for encoding detectionN)r� r1 r� )r! r1 r r r r" � s
zEncodingParser.__init__c
C s� d| j fd| jfd| jfd| jfd| jfd| jff}x^| jD ]T}d}xD|D ]<\}}| jj|�rJy |� }P W qJ tk
r� d}P Y qJX qJW |s<P q<W | jS ) Ns <!--s <metas </s <!s <?r TF) �
handleComment�
handleMeta�handlePossibleEndTag� handleOther�handlePossibleStartTagr1 r� r� r� )r! ZmethodDispatchre Z keepParsing�key�methodr r r r� � s& zEncodingParser.getEncodingc C s | j jd�S )zSkip over commentss -->)r1 r� )r! r r r r� � s z EncodingParser.handleCommentc C s� | j jtkrdS d}d }x�| j� }|d kr.dS |d dkr^|d dk}|r�|d k r�|| _dS q|d dkr�|d }t|�}|d k r�|| _dS q|d dkrtt|d ��}|j� }|d k rt|�}|d k r|r�|| _dS |}qW d S ) NTFr s
http-equivr s content-types charsets content) r1 r� r� � getAttributer� rI �ContentAttrParserr� �parse)r! Z hasPragmaZpendingEncoding�attrZtentativeEncoding�codecZ
contentParserr r r r� � s: zEncodingParser.handleMetac C s
| j d�S )NF)�handlePossibleTag)r! r r r r� � s z%EncodingParser.handlePossibleStartTagc C s t | j� | jd�S )NT)r� r1 r� )r! r r r r� � s
z#EncodingParser.handlePossibleEndTagc C sf | j }|jtkr(|r$|j� | j� dS |jt�}|dkrD|j� n | j� }x|d k r`| j� }qNW dS )NTr )r1 r� �asciiLettersBytesr� r� r� �spacesAngleBracketsr� )r! ZendTagr1 rm r� r r r r� � s
z EncodingParser.handlePossibleTagc C s | j jd�S )Nr )r1 r� )r! r r r r� s zEncodingParser.handleOtherc C s� | j }|jttdg�B �}|dkr&dS g }g }xt|dkr@|r@P nX|tkrT|j� }P nD|d krjdj|�dfS |tkr�|j|j� � n|dkr�dS |j|� t|�}q0W |dkr�|j � dj|�dfS t|� |j� }|d
k�r:|}x�t|�}||k�rt|� dj|�dj|�fS |tk�r*|j|j� � q�|j|� q�W nJ|dk�rRdj|�dfS |tk�rl|j|j� � n|dk�rzdS |j|� x^t|�}|t
k�r�dj|�dj|�fS |tk�r�|j|j� � n|dk�r�dS |j|� �q�W dS ) z_Return a name,value pair for the next attribute in the stream,
if one is found, or None� /r N� =r2 � '� ")r N)r� r )r� r� ) r1 rk r� � frozensetr3 �asciiUppercaseBytesr0 r� r� r� r� )r! r1 rm ZattrNameZ attrValueZ quoteCharr r r r� sf
zEncodingParser.getAttributeN)
r5 r6 r7 r8 r" r� r� r� r� r� r� r� r� r r r r r� � s $r� c @ s e Zd Zdd� Zdd� ZdS )r� c C s
|| _ d S )N)r1 )r! r1 r r r r" f s zContentAttrParser.__init__c C s y�| j jd� | j jd7 _| j j� | j jdks8d S | j jd7 _| j j� | j jdkr�| j j}| j jd7 _| j j}| j j|�r�| j || j j� S d S nF| j j}y | j jt� | j || j j� S tk
r� | j |d � S X W n tk
�r d S X d S )Ns charsetr r� r� r� )r� r� )r1 r� r rk r� r� r� r� )r! Z quoteMarkZ oldPositionr r r r� j s.
zContentAttrParser.parseN)r5 r6 r7 r" r� r r r r r� e s r� c C s` t | t�r.y| jd�} W n tk
r, dS X | dk rXy
tj| �S tk
rT dS X ndS dS )z{Return the python codec name corresponding to an encoding or None if the
string doesn't correspond to a valid encoding.r N)r; r �decode�UnicodeDecodeError� webencodings�lookup�AttributeError)r� r r r rI � s
rI r )1Z
__future__r r r Zsixr r Z six.movesr r r� rp r� Z constantsr
r r r
r rN r �ior r r� r� r� r� r� r� Z invalid_unicode_no_surrogaterE rq �evalrc �setrj Zascii_punctuation_rern �objectr rC r? r@ r, r� r� r� rI r r r r �<module> sV
J g Ih 6'