Monday, September 20, 2010
Sunday, September 19, 2010
7 Must-Have Technologies
Excerptions of Tommy Peterson's article
1. Managed Antispam/Antivirus
Example: Trend Micro. eTrustsoftware;
2. Smartphones
使用智能手机收发email，和接入互联网：远程管理，系统监控: Blackberry。
3. Remote Access Software
远程办公。(SonicWallVPN & CitrixSecure)
进一步考虑用移动设备进行接入 (CitrixXenApp; NetMotion; VNC)
4. Imaging Software
使用镜像软件创建系统快照，可以立即在多处pc/workstation部署。(Norton Ghost, VMWare, Win PE; Acronis)
5. Server Backup
主服务器->磁带->异地存储器
Veritas Backup Exec; Quantum Autoloader; Backup Exec; Barracuda's Yosemite; IBM’s Tivoli Storage Manager)
6. Storage: NAS and SAN
(4TB Netgear ReadyNAS; EMCSAN)
7. UPS Devices
(APC; Tripp Lite UPS)
1. Managed Antispam/Antivirus
Example: Trend Micro. eTrustsoftware;
2. Smartphones
使用智能手机收发email，和接入互联网：远程管理，系统监控: Blackberry。
3. Remote Access Software
远程办公。(SonicWallVPN & CitrixSecure)
进一步考虑用移动设备进行接入 (CitrixXenApp; NetMotion; VNC)
4. Imaging Software
使用镜像软件创建系统快照，可以立即在多处pc/workstation部署。(Norton Ghost, VMWare, Win PE; Acronis)
5. Server Backup
主服务器->磁带->异地存储器
Veritas Backup Exec; Quantum Autoloader; Backup Exec; Barracuda's Yosemite; IBM’s Tivoli Storage Manager)
6. Storage: NAS and SAN
(4TB Netgear ReadyNAS; EMCSAN)
7. UPS Devices
(APC; Tripp Lite UPS)
Saturday, September 18, 2010
二项分布与泊松分布 （binomial distribution and poisson distribution)
二项分布和Poisson分布均是常见的离散型分布，在分类资料的统计推断中有非常广泛的应用。
一、二项分布的概念及应用条件
1. 二项分布的概念:
如某实验中小白鼠染毒后死亡概率P为0.8,则生存概率为=1-P=0.2，故
对一只小白鼠进行实验的结果为：死（概率为P）或生（概率为1-P）
对二只小白鼠（甲乙）进行实验的结果为：甲乙均死（概率为P2）、甲死乙生[概率为P(1-P)]、乙死甲生[概率为(1-P）P]或甲乙均生[概率为(1-P）2]，概率相加得P2+P(1-P)+(1-P）P+(1-P）2=[P+(1-P)]2
依此类推，对n只小白鼠进行实验，所有可能结果的概率相加得Pn+cn1P(1-P)n-1+...+cnxPx(1-P)n-x+...+(1-P)x=[P+(1-P)]n 其中n为样本含量,即事件发生总数，x为某事件出现次数,cnxPx(1-P)n-x为二项式通式，cnx=n!/x!(n-x)!, P为总体率。
因此，二项分布是说明结果只有两种情况的n次实验中发生某种结果为x次的概率分布。其概率密度为：
P(x)=cnxPx(1-P)n-x, x=0,1,...n。
2. 二项分布的应用条件:
医学领域有许多二分类记数资料都符合二项分布(传染病和遗传病除外)，但应用时仍应注意考察是否满足以下应用条件：(1) 每次实验只有两类对立的结果；(2) n次事件相互独立；(3) 每次实验某类结果的发生的概率是一个常数。
3. 二项分布的累计概率
二项分布下最多发生k例阳性的概率为发生0例阳性、1例阳性、...、直至k例阳性的概率之和。至少发生k例阳性的概率为发生k例阳性、k+1例阳性、...、直至n例阳性的概率之和。
4. 二项分布的图形
二项分布的图形有如下特征：(1)二项分布图形的形状取决于P 和n 的大小；(2) 当P=0.5时，无论n的大小，均为对称分布；(3) 当P<>0.5 ,n较小时为偏态分布,n较大时逼近正态分布。
5. 二项分布的均数和标准差
二项分布的均数µ=np，当用率表示时µ=p
二项分布的标准差为np(1-p)的算术平方根，当用率表示时为p(1-p)的算术平方根。
二、二项分布的应用
二项分布主要用于符合二项分布分类资料的率的区间估计和假设检验。当P=0.5或n较大，nP及n(1-P)均大于等于5时，可用(p-u0.05sp,p+u0.05sp)对总体率进行95%的区间估计。当总体率P接近0.5，阳性数x较小时，可直接计算二项分布的累计概率进行单侧的假设检验。当P=0.5或n较大，nP及n(1-P)均大于等于5时，可用正态近似法进行样本率与总体率，两个样本率比较的u检验。
三、Poisson分布的概念及应用条件
1. Poisson分布的概念:
Poisson分布是二项分布n很大而P很小时的特殊形式，是两分类资料在n次实验中发生x次某种结果的概率分布。其概率密度函数为：P(x)=e-µ*µx/x! x=0,1,2...n，其中e为自然对数的底，µ为总体均数，x为事件发生的阳性数。
2. Poisson分布的应用条件:
医学领域中有很多稀有疾病(如肿瘤,交通事故等）资料都符合Poisson分布，但应用中仍应注意要满足以下条件：（1) 两类结果要相互对立；（2) n次试验相互独立；（3) n应很大, P应很小。
3. Poisson分布的概率
Poisson分布的概率利用以下递推公式很容易求得：
P(0)=e-µ
P(x+1)=P(x)*µ/x+1, x=0,1,2,...
4. Poisson分布的性质:
(1) Poisson分布均数与方差相等；
(2) Poisson分布均数µ较小时呈偏态,µ>=20时近似正态；
(3) n很大, P很小，nP=µ为常数时二项分布趋近于Poisson分布；
(4) n个独立的Poisson分布相加仍符合Poisson分布
四、Poisson分布的应用
Poisson分布也主要用于符合Poisson分布分类资料率的区间估计和假设检验。当µ>=20时，根据正态近似的原理，可用（x-u0.05*x的算术平方根，x+u0.05*x的算术平方根）对总体均数进行95%的区间估计。同样，也可通过直接计算Poisson分布的累计概率进行单侧的假设检验，在符合正态近似条件时，也可用u检验进行样本率与总体率，两个样本率比较的假设检验
一、二项分布的概念及应用条件
1. 二项分布的概念:
如某实验中小白鼠染毒后死亡概率P为0.8,则生存概率为=1-P=0.2，故
对一只小白鼠进行实验的结果为：死（概率为P）或生（概率为1-P）
对二只小白鼠（甲乙）进行实验的结果为：甲乙均死（概率为P2）、甲死乙生[概率为P(1-P)]、乙死甲生[概率为(1-P）P]或甲乙均生[概率为(1-P）2]，概率相加得P2+P(1-P)+(1-P）P+(1-P）2=[P+(1-P)]2
依此类推，对n只小白鼠进行实验，所有可能结果的概率相加得Pn+cn1P(1-P)n-1+...+cnxPx(1-P)n-x+...+(1-P)x=[P+(1-P)]n 其中n为样本含量,即事件发生总数，x为某事件出现次数,cnxPx(1-P)n-x为二项式通式，cnx=n!/x!(n-x)!, P为总体率。
因此，二项分布是说明结果只有两种情况的n次实验中发生某种结果为x次的概率分布。其概率密度为：
P(x)=cnxPx(1-P)n-x, x=0,1,...n。
2. 二项分布的应用条件:
医学领域有许多二分类记数资料都符合二项分布(传染病和遗传病除外)，但应用时仍应注意考察是否满足以下应用条件：(1) 每次实验只有两类对立的结果；(2) n次事件相互独立；(3) 每次实验某类结果的发生的概率是一个常数。
3. 二项分布的累计概率
二项分布下最多发生k例阳性的概率为发生0例阳性、1例阳性、...、直至k例阳性的概率之和。至少发生k例阳性的概率为发生k例阳性、k+1例阳性、...、直至n例阳性的概率之和。
4. 二项分布的图形
二项分布的图形有如下特征：(1)二项分布图形的形状取决于P 和n 的大小；(2) 当P=0.5时，无论n的大小，均为对称分布；(3) 当P<>0.5 ,n较小时为偏态分布,n较大时逼近正态分布。
5. 二项分布的均数和标准差
二项分布的均数µ=np，当用率表示时µ=p
二项分布的标准差为np(1-p)的算术平方根，当用率表示时为p(1-p)的算术平方根。
二、二项分布的应用
二项分布主要用于符合二项分布分类资料的率的区间估计和假设检验。当P=0.5或n较大，nP及n(1-P)均大于等于5时，可用(p-u0.05sp,p+u0.05sp)对总体率进行95%的区间估计。当总体率P接近0.5，阳性数x较小时，可直接计算二项分布的累计概率进行单侧的假设检验。当P=0.5或n较大，nP及n(1-P)均大于等于5时，可用正态近似法进行样本率与总体率，两个样本率比较的u检验。
三、Poisson分布的概念及应用条件
1. Poisson分布的概念:
Poisson分布是二项分布n很大而P很小时的特殊形式，是两分类资料在n次实验中发生x次某种结果的概率分布。其概率密度函数为：P(x)=e-µ*µx/x! x=0,1,2...n，其中e为自然对数的底，µ为总体均数，x为事件发生的阳性数。
2. Poisson分布的应用条件:
医学领域中有很多稀有疾病(如肿瘤,交通事故等）资料都符合Poisson分布，但应用中仍应注意要满足以下条件：（1) 两类结果要相互对立；（2) n次试验相互独立；（3) n应很大, P应很小。
3. Poisson分布的概率
Poisson分布的概率利用以下递推公式很容易求得：
P(0)=e-µ
P(x+1)=P(x)*µ/x+1, x=0,1,2,...
4. Poisson分布的性质:
(1) Poisson分布均数与方差相等；
(2) Poisson分布均数µ较小时呈偏态,µ>=20时近似正态；
(3) n很大, P很小，nP=µ为常数时二项分布趋近于Poisson分布；
(4) n个独立的Poisson分布相加仍符合Poisson分布
四、Poisson分布的应用
Poisson分布也主要用于符合Poisson分布分类资料率的区间估计和假设检验。当µ>=20时，根据正态近似的原理，可用（x-u0.05*x的算术平方根，x+u0.05*x的算术平方根）对总体均数进行95%的区间估计。同样，也可通过直接计算Poisson分布的累计概率进行单侧的假设检验，在符合正态近似条件时，也可用u检验进行样本率与总体率，两个样本率比较的假设检验
Thursday, September 16, 2010
Format of a Transport Stream Packet
Each MPEG-2 TS packet carries 184 B of payload data prefixed by a 4 B (32 bit) header.
The header has the following fields:
The header starts with a well-known Synchronisation Byte (8 bits). This has the bit pattern 0x47 (0100 0111).
Two options are possible for inserting PES data into the TS packet payload:
The header has the following fields:
The header starts with a well-known Synchronisation Byte (8 bits). This has the bit pattern 0x47 (0100 0111).
- A set of three flag bits are used to indicate how the payload should be processed.
- The first flag indicates a transport error.
- The second flag indicates the start of a payload (payload_unit_start_indicator)
- The third flag indicates transport priority bit.
- The flags are followed by a 13 bit Packet Identifier (PID). This is used to uniquely identify the stream to which the packet belongs (e.g. PES packets corresponding to an ES) generated by the multiplexer. The PID allows the receiver to differentiate the stream to which each received packet belongs. Some PID values are predefined and are used to indicate various streams of control information. A packet with an unknown PID, or one with a PID which is not required by the receiver, is silently discarded. The particular PID value of 0x1FFF is reserved to indicate that the packet is a null packet (and is to be ignored by the receiver).
- two scrambling control bits, used by conditional access procedures to encrypted the payload of some TS packets.
- Two adaption field control bits, which may take four values:
- 01 – no adaptation field, payload only
- 10 – adaptation field only, no payload
- 11 – adaptation field followed by payload
- 00 - RESERVED for future use Finally there is a half byte Continuity Counter (4 bits)
Two options are possible for inserting PES data into the TS packet payload:
- The simplest option, from both the encoder and receiver viewpoints, is to send only one PES (or a part of single PES) in a TS packet. This allows the TS packet header to indicate the start of the PES, but since a PES packet may have an arbitrary length, also requires the remainder of the TS packet to be padded, ensuring correct alignment of the next PES to the start of a TS packet. In MPEG-2 the padding value is the hexadecimal byte 0xFF.
- In general a given PES packet spans several TS packets so that the majority of TS packets contain continuation data in their payloads. When a PES packet is starting, however, the payload_unit_start_indicator bit is set to ‘1’ which means the first byte of the TS payload contains the first byte of the PES packet header. Only one PES packet can start in any single TS packet. The TS header also contains the PID so that the receiver can accept or reject PES packets at a high level without burdening the receiver with too much processing. This has an impact on short PES packets
video transmission (mpeg2)
(excerpted from http://www.abdn.ac.uk/erg/research/future-net/digital-video/mpeg2-trans.html)
The MPEG-2 standards define how to format the various component parts of a multimedia programme (which may consist of: MPEG-2 compressed video, compressed audio, control data and/or user data). It also defines how these components are combined into a single synchronous transmission bit stream. The process of combining the steams is known as multiplexing.
The multiplexed stream may be transmitted over a variety of links, standards / products are (or will soon be) available for :
Building the MPEG Bit Stream
To understand how the component parts of the bit stream are multiplexed, we need to first look at each component part. The most basic component is known as an Elementary Stream (ES) in MPEG. A programme (perhaps most easily thought of as a television programme, or a DVD track) contains a combination of elementary streams (typically one for video, one or more for audio, control data, subtitles, etc).
Each Elementary Stream (ES) output by an MPEG audio, video and (some) data encoders contain a single type of (usually compressed) signal. There are various forms of ES, including:
For video and audio, the data is organised into access units, each representing a fundamental unit of encoding. For example, in video, an access unit will usually be a complete encoded video frame.
Each ES is input to an MPEG-2 processor (e.g. a video compressor or data formatted) which accumulates the data into a stream of Packetised Elementary Stream (PES) packets. A PES packet may be a fixed (or variable) sized block, with up to 65536 bytes per block and includes a 6 byte protocol header. A PES is usually organised to contain an integral number of ES access units.
The PES header starts with a 3 byte start code, followed by a one byte stream ID and a 2 byte length field.
PES Indicators provide additional information about the stream to assist the decoder at the receiver. The following indicators are defined:
The PES packet payload includes the ES data. The information in the PES header is, in general, independent of the transmission method used.
MPEG-2 Multiplexing
The MPEG-2 standard allows two forms of multiplexing:
MPEG Program Stream
A group of tightly coupled PES packets referenced to the same time base. Such streams are suited for transmission in a relatively error-free environment and enable easy software processing of the received data. This form of multiplexing is used for video playback and for some network applications.
MPEG Transport Stream Each PES packet is broken into fixed-sized transport packets forming a general purpose way of combining one or more streams, possibly with independent time bases. This is suited for transmission in which there may be potential packet loss or corruption by noise, or / and where there is a need to send more than one programme at a time.
MPEG Transport Streams
A transport stream consists of a sequence of fixed sized transport packet of 188 B. Each packet comprises 184 B of payload and a 4 B header. One of the items in this 4 B header is the 13 bit Packet Identifier (PID) which plays a key role in the operation of the Transport Stream.
The format of the transport stream is described with an example: two elementary streams sent in the same MPEG-2 transport multiplex. Each packet is associated with a PES through the setting of the PID value in the packet header (the values of 64 and 51 respectively). The audio packets have been assigned PID 64, and the video packets PID 51 (these are arbitrary, but different values). As is usual, there are more video than audio packets, but you may also note that the two types of packets are not evenly spaced in time. The MPEG-TS is not a time division multiplex, packets with any PID may be inserted into the TS at any time by the TS multiplexor. If no packets are available at the multiplexor, it inserts null packets (denoted by a PID value of 0x1FFF) to retain the specified TS bit rate. The multiplexor also does not synchronise the two PESs, indeed the encoding and decoding delay for each PES may (and usually is different). A separate process is therefore require to synchronise the two streams.
Single and Multiple Program Transport Streams
A TS may correspond to a single TV programme, or multimedia stream (e.g. with two a video PES and an audio PES). This type of TS is normally called a Single Programme Transport Stream (SPTS).
An SPTS contains all the information requires to reproduce the encoded TV channel or multimedia stream. It may contain only an audio and video PESs, but in practice there will be other types of PES as well. Each PES shares a common timebase. Although some equipments output and use SPTS, this is not the normal form transmitted over a DVB link.
In most cases one or more SPTS streams are combined to form a Multiple Programme Transport Stream (MPTS). This larger aggregate also contains all the control information (Program Specific Information (PSI)) required to co-ordinate the DVB system, and any other data which is to be sent.
Signalling Tables
For a user to receive a particular transport stream, the user must first determine the PID being used, and then filter packets which have a matching PID value. To help the user identify which PID corresponds to which programme, a special set of streams, known as Signalling Tables, are transmitted with a description of each program carried within the MPEG-2 Transport Stream. Signalling tables are sent separately to PES, and are not synchronised with the elementary streams (i.e they are an independent control channel).
The tables (called Program Specific Information (PSI) in MPEG-2) consist of a description of the elementary streams which need to be combined to build programmes, and a description of the programmes. Each PSI table is carried in a sequence of PSI Sections, which may be of variable length (but are usually small, c.f. PES packets). Each section is protected by a CRC (checksum) to verify the integrity of the table being carried. The length of a section allows a decoder to identify the next section in a packet. A PSI section may also be used for down-loading data to a remote site. Tables are sent periodically by including them in the transmitted transport multiplex.
The MPEG-2 standards define how to format the various component parts of a multimedia programme (which may consist of: MPEG-2 compressed video, compressed audio, control data and/or user data). It also defines how these components are combined into a single synchronous transmission bit stream. The process of combining the steams is known as multiplexing.
The multiplexed stream may be transmitted over a variety of links, standards / products are (or will soon be) available for :
- Radio Frequency Links (UHF/VHF)
- Digital Broadcast Satellite Links
- Cable TV Networks
- Standard Terrestrial Communication Links (PDH, SDH)
- Microwave Line of Sight (LoS) Links (wireless)
- Digital Subscriber Links (ADSL family)
- Packet / Cell Links (ATM, IP, IPv6, Ethernet)
Building the MPEG Bit Stream
To understand how the component parts of the bit stream are multiplexed, we need to first look at each component part. The most basic component is known as an Elementary Stream (ES) in MPEG. A programme (perhaps most easily thought of as a television programme, or a DVD track) contains a combination of elementary streams (typically one for video, one or more for audio, control data, subtitles, etc).
Each Elementary Stream (ES) output by an MPEG audio, video and (some) data encoders contain a single type of (usually compressed) signal. There are various forms of ES, including:
- Digital Control Data
- Digital Audio (sampled and compressed)
- Digital Video (sampled and compressed)
- Digital Data (synchronous, or asynchronous)
For video and audio, the data is organised into access units, each representing a fundamental unit of encoding. For example, in video, an access unit will usually be a complete encoded video frame.
Each ES is input to an MPEG-2 processor (e.g. a video compressor or data formatted) which accumulates the data into a stream of Packetised Elementary Stream (PES) packets. A PES packet may be a fixed (or variable) sized block, with up to 65536 bytes per block and includes a 6 byte protocol header. A PES is usually organised to contain an integral number of ES access units.
The PES header starts with a 3 byte start code, followed by a one byte stream ID and a 2 byte length field.
PES Indicators provide additional information about the stream to assist the decoder at the receiver. The following indicators are defined:
- PES_Scrambling_Control - Defines whether scrambling is used, and the chosen scrambling method.
- PES_Priority - Indicates priority of the current PES packet.
- data_alignment_indicator - Indicates if the payload starts with a video or audio start code.
- copyright information - Indicates if the payload is copyright protected.
- original_or_copy - Indicates if this is the original ES.A one byte flags field completes the PES header. This defines the following optional fields, which if present, are inserted before the start of the PES payload.
- Presentation Time Stamp (PTS) and possibly a Decode Time Stamp (DTS) - For audio / video streams these time stamps which may be used to synchronise a set of elementary streams and control the rate at which they are replayed by the receiver.
- Elementary Stream Clock Reference (ESCR)
- Elementary Stream rate - Rate at which the ES was encoded.
- Trick Mode - indicates the video/audio is not the normal ES, e.g. after DSM-CC has signalled a replay.
- Copyright Information - set to 1 to indicated a copyright ES.
- CRC - this may be used to monitor errors in the previous PES packet
- PES Extension Information - may be used to support MPEG-1 streams.
The PES packet payload includes the ES data. The information in the PES header is, in general, independent of the transmission method used.
MPEG-2 Multiplexing
The MPEG-2 standard allows two forms of multiplexing:
MPEG Program Stream
A group of tightly coupled PES packets referenced to the same time base. Such streams are suited for transmission in a relatively error-free environment and enable easy software processing of the received data. This form of multiplexing is used for video playback and for some network applications.
MPEG Transport Stream Each PES packet is broken into fixed-sized transport packets forming a general purpose way of combining one or more streams, possibly with independent time bases. This is suited for transmission in which there may be potential packet loss or corruption by noise, or / and where there is a need to send more than one programme at a time.
MPEG Transport Streams
A transport stream consists of a sequence of fixed sized transport packet of 188 B. Each packet comprises 184 B of payload and a 4 B header. One of the items in this 4 B header is the 13 bit Packet Identifier (PID) which plays a key role in the operation of the Transport Stream.
The format of the transport stream is described with an example: two elementary streams sent in the same MPEG-2 transport multiplex. Each packet is associated with a PES through the setting of the PID value in the packet header (the values of 64 and 51 respectively). The audio packets have been assigned PID 64, and the video packets PID 51 (these are arbitrary, but different values). As is usual, there are more video than audio packets, but you may also note that the two types of packets are not evenly spaced in time. The MPEG-TS is not a time division multiplex, packets with any PID may be inserted into the TS at any time by the TS multiplexor. If no packets are available at the multiplexor, it inserts null packets (denoted by a PID value of 0x1FFF) to retain the specified TS bit rate. The multiplexor also does not synchronise the two PESs, indeed the encoding and decoding delay for each PES may (and usually is different). A separate process is therefore require to synchronise the two streams.
Single and Multiple Program Transport Streams
A TS may correspond to a single TV programme, or multimedia stream (e.g. with two a video PES and an audio PES). This type of TS is normally called a Single Programme Transport Stream (SPTS).
An SPTS contains all the information requires to reproduce the encoded TV channel or multimedia stream. It may contain only an audio and video PESs, but in practice there will be other types of PES as well. Each PES shares a common timebase. Although some equipments output and use SPTS, this is not the normal form transmitted over a DVB link.
In most cases one or more SPTS streams are combined to form a Multiple Programme Transport Stream (MPTS). This larger aggregate also contains all the control information (Program Specific Information (PSI)) required to co-ordinate the DVB system, and any other data which is to be sent.
Signalling Tables
For a user to receive a particular transport stream, the user must first determine the PID being used, and then filter packets which have a matching PID value. To help the user identify which PID corresponds to which programme, a special set of streams, known as Signalling Tables, are transmitted with a description of each program carried within the MPEG-2 Transport Stream. Signalling tables are sent separately to PES, and are not synchronised with the elementary streams (i.e they are an independent control channel).
The tables (called Program Specific Information (PSI) in MPEG-2) consist of a description of the elementary streams which need to be combined to build programmes, and a description of the programmes. Each PSI table is carried in a sequence of PSI Sections, which may be of variable length (but are usually small, c.f. PES packets). Each section is protected by a CRC (checksum) to verify the integrity of the table being carried. The length of a section allows a decoder to identify the next section in a packet. A PSI section may also be used for down-loading data to a remote site. Tables are sent periodically by including them in the transmitted transport multiplex.
Subscribe to:
Posts (Atom)