【转载】VisualSVN Server 增加自助修改密码页面(支持2.1-3.6最新版)

==============================================================

原文链接:http://oicu.cc.blog.163.com/blog/static/1230394712016102312546504/

请访问原文链接以获取最新版本,感谢原文作者的无私共享,本站仅转载作为镜像保存!

==============================================================

注:本方法实测支持 VisualSVN 2.1~3.6.x,不过2还需多改Apache的配置,老版本应该抛弃,懒得写了。
转载请注明来源,不为别的,就是怕那些无良网站为了流量到处转帖,转丢失内容或者是老版本的内容。
如果不用VisualSVN客户端的话,VisualSVN Server只能在服务器端修改密码,对管理来说很不方便。
 
网上大部分给 VisualSVN 增加自助修改密码的补丁都是基于 2.5.x 版本的,也有几个用于 3.5.x/3.6.x 版本,多数还是用 cgi 方式,而且要么像 csdn 那样藏着掖着,要么也没个详细的说明。
 
VisualSVN Server 帐号及密码保存在 htpasswd 文件里,除了 cgi 有以下几种修改方式:
 
1、使用 VisualSVN Server Manager 管理工具重置密码。
 
2、通过 WMI 里用 PowerShell 脚本更改。示例如下:
$svnuser = Get-WmiObject -Namespace Root\VisualSVN `
-ComputerName svn.hostname.com `
-query “select * from VisualSVN_User where name = ‘username'”
$svnuser.SetPassword(‘123456’)
3、使用 Windows 版 Apache 的 htpasswd.exe 命令更改。
该方法是使用 php 页面来调用 htpasswd.exe 修改密码,方便用户通过网页修改,下面讲解配置要点。

 

我使用的操作系统是 Windows Server 2008R2 x64,安装了 VisualSVN Server 3.5.6 x64 带 Apache 2.2.31 x64 的版本,默认安装路径。
 
从 Apache 官方网站下载完整的 Apache 2.2.31 x64 版本,从里面提取一个文件 htpasswd.exe 放到
C:\Program Files\VisualSVN Server\bin\htpasswd.exe
Apache 2.2.x 要以 handler 方式加载 php 模块,只有 php 5.2-5.4 的 Thread Safe 版本才带 php5apache2_2.dll 文件,php 5.5 及之后的版本只能和 Apache 2.4.x 搭配了,所以选定 php 5.4 版本。
 
特别注意,如果用 VisualSVN Server x64 就必须找 x64 的 php!
否则 Apache 加载 php 模块会提示错误 Cannot load php5apache2_2.dll into server
因为 Apache x64 无法使用 php x86。
 
使用 32 位版本的 VisualSVN Server 比较简单,因为 php 官网都是 x86 版本:
 
使用 64 位版本的 VisualSVN Server 就得找第三方编译的 php x64 版本了,提供两个下载:
 
我下载的是 php-5.4.36-Win32-VC9-x64.zip,下载后解压到 C:\Program Files\VisualSVN Server\php 文件夹。把 php.ini-production 文件重命名为 php.ini 即可,其他不用配置。
 
修改空文件 C:\Program Files\VisualSVN Server\conf\httpd-custom.conf
内容如下:
LoadModule php5_module “php/php5apache2_2.dll”
<IfModule php5_module>
AddType application/x-httpd-php .php
DirectoryIndex index.html index.php
</IfModule>
# 配置 php.ini 的路径
PHPIniDir “php”
新建一个 php 文件放到 C:\Program Files\VisualSVN Server\htdocs\pw\index.php
内容如下(做了一点小修改,增加返回及跳转页面的处理):
<?php
$username = $_SERVER[“PHP_AUTH_USER”]; //经过 AuthType Basic 认证的用户名
$authed_pass = $_SERVER[“PHP_AUTH_PW”]; //经过 AuthType Basic 认证的密码
$input_oldpass = (isset($_REQUEST[“oldpass”]) ? $_REQUEST[“oldpass”] : “”); //从界面上输入的原密码
$newpass = (isset($_REQUEST[“newpass”]) ? $_REQUEST[“newpass”] : “”); //界面上输入的新密码
$repeatpass = (isset($_REQUEST[“repeatpass”]) ? $_REQUEST[“repeatpass”] : “”); //界面上输入的重复密码
$action = (isset($_REQUEST[“action”]) ? $_REQUEST[“action”] : “”); //以hide方式提交到服务器的action
if ($action!=”modify”) {
$action = “view”;
} else if ($authed_pass!=$input_oldpass) {
$action = “oldpasswrong”;
} else if (empty($newpass)) {
$action = “passempty”;
} else if ($newpass!=$repeatpass) {
$action = “passnotsame”;
} else{
$action = “modify”;
}
?>
<html>
<head>
<meta http-equiv=”Content-Type” content=”text/html; charset=GBK”>
<title>Subversion 在线自助密码修改</title>
</head>
<body>

<?php
//action=view 显示普通的输入信息
if ($action == “view”) {
?>
<script language = “javaScript”>
<!–
function loginIn(myform) {
var newpass=myform.newpass.value;
var repeatpass=myform.repeatpass.value;

    if (newpass==””) {
alert(“请输入密码!”);
return false;
}

    if (repeatpass==””) {
alert(“请重复输入密码!”);
return false;
}

    if (newpass!=repeatpass) {
alert(“两次输入密码不一致,请重新输入!”);
return false;
}
return true;
}
//–>
</script>

<style type=”text/css”>
<!–
table {
border: 1px solid #CCCCCC;
background-color: #f9f9f9;
text-align: center;
vertical-align: middle;
font-size: 9pt;
line-height: 15px;
}
th {
font-weight: bold;
line-height: 20px;
border-top-width: 1px;
border-right-width: 1px;
border-bottom-width: 1px;
border-left-width: 1px;
border-bottom-style: solid;
color: #333333;
background-color: f6f6f6;
}
input{
height: 18px;
}
.button {
height: 20px;
}
–>
</style>

<br><br><br>
<form method=”post”>
<input type=”hidden” name=”action” value=”modify”/>
<table width=”220″ cellpadding=”3″ cellspacing=”8″ align=”center”>
<tr>
<th colspan=2>Subversion 密码修改</th>
</tr>
<tr>
<td>用 户 名:</td>
<td align=”left”> <?php echo $username?></td>
</tr>
<tr>
<td>原 密 码:</td>
<td><input type=password size=12 name=oldpass></td>
</tr>
<tr>
<td>用户密码:</td>
<td><input type=password size=12 name=newpass></td>
</tr>
<tr>
<td>确认密码:</td>
<td><input type=password size=12 name=repeatpass></td>
</tr>
<tr>
<td colspan=2>
<input onclick=”return loginIn(this.form)” class=”button” type=submit value=”修 改”>
<input name=”reset” type=reset class=”button” value=”取 消”>
<input onclick=”window.location.href=’/'” class=”button” type=”button” value=”返 回”>
</td>
</tr>
</table>
</form>
<?php
} else if ($action == “oldpasswrong”) {
$msg=”原密码错误”;
} else if ($action == “passempty”) {
$msg=”请输入新密码”;
} else if ($action == “passnotsame”) {
$msg=”两次输入密码不一致,请重新输入”;
} else {
$passwdfile=”
D:\Repositories\htpasswd“;
$command='”
C:\Program Files\VisualSVN Server\bin\htpasswd.exe” -b ‘.$passwdfile.” “.$username.’ “‘.$newpass.'”‘;
system($command, $result);
if ($result==0) {
$msg_succ=”用户[“.$username.”]密码修改成功,请用新密码登陆”;
} else {
$msg=”用户[“.$username.”]密码修改失败,返回值为”.$result.”,请和管理员联系”;
}
}

if (isset($msg_succ)) {
?>
<script language=”javaScript”>
<!–
alert(“<?php echo $msg_succ?>”);
window.location.href=”/”
//–>
</script>
<?php
} else if (isset($msg)) {
?>
<script language=”javaScript”>
<!–
alert(“<?php echo $msg?>”);
window.location.href=”<?php echo $_SERVER[“PHP_SELF”]?>”
//–>
</script>
<?php
}
?>
</body>
</html>

 
在header增加超链接麻烦,所以我就在页脚增加修改密码的链接,
修改文件 C:\Program Files\VisualSVN Server\WebUI\index.html
内容如下:
<footer>
Powered by <a href=”
http://www.visualsvn.com/server/”>VisualSVN Server</a>. &copy; 2005-2016 VisualSVN Limited.
<br /><br /><a href=”/pw”>自助修改密码</a>
</footer>
完工。

 

效果图如下:

VisualSVN Server 3.5.x增加自助修改密码页面 - oicu - Oh! I see you!

 

VisualSVN Server 3.5.x增加自助修改密码页面 - oicu - Oh! I see you!
==============================================================
本站说明:如果启动http服务报错,则需要安装VC9.0的运行库
==============================================================

eGalax Touchscreen Configuration Ubuntu 14.04LTS

In order to configure the eGalax Touchscreen in Ubuntu 14.04LTS you need to perform the following steps:

the newest driver has a setup.sh file. just run it and it does all the work for you. For Ubuntu 16.04 see Page 6 below.

Ensure the Kernel Modules start on boot

Modify the file /etc/modules to add the modules usbtouchscreen and usbhid. These must come after lp and before rtc.

=== /etc/modules ===

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with “#” are ignored.

loop
lp
usbtouchscreen
usbhid
rtc

=================


Load the Kernel Modules (only the 1st time)

Then load the modules by hand the 1st time (or reboot)

sudo modprobe usbtouchscreen usbhid

Download the Files

http://www.eeti.com.tw/LinuxDriverDownload.html

Unzip the Files to your Downloads

gunzip /home/$USER/Downloads/eGalax*.zip

Move the Daemon and Calibration Files

Move al the files in the folder /home/user/Downloads/eGalax/eGTouch_v2.5.4330.L-x/eGTouch64/eGTouch64withX to

/usr/local/bin/  

*Note the following command is one line

sudo rsync -avz /home/$USER/Downloads/eGalax/eGTouch_v2.5.4330.L-x/eGTouch64/eGTouch64withX /usr/local/bin/

        


Place Upstart Job

place the following Upstart job in /etc/init

=== /etc/init/egalaxtouch.conf ===

#!/bin/bash

description        “Start the egalax touchscreen daemon on boot”

author                “globeone  Damiön la Bagh”

start on runlevel 2

exec /usr/local/bin/eGTouchD

=========================

Reload the Upstart configuration files

initctl reload

Start the Touchscreen Daemon

Start the touchscreen daemon with the following command

        sudo service egalaxtouch start

Calibration

Calibrate the Touchscreen with eGTouchU

        sudo /usr/local/bin/eGTouchU

For Dual Screen Configurations:

Follow the onscreen prompts of the config program for dual screen setup choose Left or Right mode!

Then calibrate the screen

l

Enable the Touchscreen in Lightdm (the login screen)

  1. copy ~/.config/monitors.xml to /var/lib/lightdm/.config

Ubuntu 16.04

Ubuntu 16.04 has moved from Upstart to SystemD.

This changes how to install the eGalax touchscreens so they keep running even if the service crashes.

Setup.sh

Run the setup.sh file provided by eGalax

SystemD Setup

sudo systemctl edit –full eGTouch.service

Overwrite the file with the code here below:

############################################

 # eGalax Touchscreen service file

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/usr/bin/eGTouchD
DefaultDependencies=no
Before=sysinit.target
After=apparmor.service

[Service]
Type=forking
Restart=always

RestartSec=5
StartLimitInterval=60s
StartLimitBurst=3
TimeoutSec=0
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
User=root
Group=root
ExecStart=
ExecStart=/usr/bin/eGTouchD start
ExecReload=/usr/bin/eGTouchD restart
ExecStop=/usr/bin/eGTouchD stop

[Install]
WantedBy=multi-user.target

########################################

Remove the SystemV init script from /etc/init.d/eGTouch.sh

sudo rm /etc/init.d/eGTouch.sh

Enable the new SystemD service

sudo systemctl enable eGTouch.service

sudo systemctl daemon-reload

sudo systemctl restart eGTouch.service

Check the status of the service

sudo systemctl status eGTouch.service

user@hostname:~$ sudo systemctl status eGTouch.service
 eGTouch.service
Loaded: loaded (/usr/bin/eGTouchD; enabled; vendor preset: enabled)
Active: 
active (running) since zo 2017-03-26 13:55:23 CEST; 14min ago
Docs: man:systemd-sysv-generator(8)
CGroup: /system.slice/eGTouch.service
└─906 /usr/bin/eGTouchD start

mrt 26 13:55:23 hostname systemd[1]: Starting eGTouch.service…
mrt 26 13:55:23 hostname eGTouchD[895]: rm: kan ‘/tmp/eGTouch_*’ niet verwijderen: Bestand of map bestaat niet
mrt 26 13:55:23 hostname systemd[1]: Started eGTouch.service.


If you kill the service

sudo kill 906

You will see the touchscreen restarting.

【原文链接】

https://docs.google.com/document/d/1G4oD6Y8vlyNHW6wJT89pxcjWHoETLLT-SEoAIW6_7Xc/pub

Windows KMS Key

Windows安装时使用,安装完成后可以使用KMS工具激活。

 

Operating system edition KMS Client Setup Key
Windows Server 2016 Datacenter CB7KF-BWN84-R7R2Y-793K2-8XDDG
Windows Server 2016 Standard WC2BQ-8NRM3-FDDYY-2BFGV-KHKQY
Windows Server 2016 Essentials JCKRF-N37P4-C2D82-9YXRT-4M63B
Operating system edition KMS Client Setup Key
Windows 10 Professional W269N-WFGWX-YVC9B-4J6C9-T83GX
Windows 10 Professional N MH37W-N47XK-V7XM9-C7227-GCQG9
Windows 10 Enterprise NPPR9-FWDCX-D2C8J-H872K-2YT43
Windows 10 Enterprise N DPH2V-TTNVB-4X9Q3-TJR4H-KHJW4
Windows 10 Education NW6C2-QMPVW-D7KKK-3GKT6-VCFB2
Windows 10 Education N 2WH4N-8QGBV-H22JP-CT43Q-MDWWJ
Windows 10 Enterprise 2015 LTSB WNMTR-4C88C-JK8YV-HQ7T2-76DF9
Windows 10 Enterprise 2015 LTSB N 2F77B-TNFGY-69QQF-B8YKP-D69TJ
Windows 10 Enterprise 2016 LTSB DCPHK-NFMTC-H88MJ-PFHPY-QJ4BJ
Windows 10 Enterprise 2016 LTSB N QFFDN-GRT3P-VKWWX-X7T3R-8B639
Operating system edition KMS Client Setup Key
Windows 8.1 Professional GCRJD-8NW9H-F2CDX-CCM8D-9D6T9
Windows 8.1 Professional N HMCNV-VVBFX-7HMBH-CTY9B-B4FXY
Windows 8.1 Enterprise MHF9N-XY6XB-WVXMC-BTDCT-MKKG7
Windows 8.1 Enterprise N TT4HM-HN7YT-62K67-RGRQJ-JFFXW
Windows Server 2012 R2 Server Standard D2N9P-3P6X9-2R39C-7RTCD-MDVJX
Windows Server 2012 R2 Datacenter W3GGN-FT8W3-Y4M27-J84CP-Q3VJ9
Windows Server 2012 R2 Essentials KNC87-3J2TX-XB4WP-VCPJV-M4FWM
Operating system edition KMS Client Setup Key
Windows 8 Professional NG4HW-VH26C-733KW-K6F98-J8CK4
Windows 8 Professional N XCVCF-2NXM9-723PB-MHCB7-2RYQQ
Windows 8 Enterprise 32JNW-9KQ84-P47T8-D8GGY-CWCK7
Windows 8 Enterprise N JMNMF-RHW7P-DMY6X-RF3DR-X2BQT
Windows Server 2012 BN3D2-R7TKB-3YPBD-8DRP2-27GG4
Windows Server 2012 N 8N2M2-HWPGY-7PGT9-HGDD8-GVGGY
Windows Server 2012 Single Language 2WN2H-YGCQR-KFX6K-CD6TF-84YXQ
Windows Server 2012 Country Specific 4K36P-JN4VD-GDC6V-KDT89-DYFKP
Windows Server 2012 Server Standard XC9B7-NBPP2-83J2H-RHMBY-92BT4
Windows Server 2012 MultiPoint Standard HM7DN-YVMH3-46JC3-XYTG7-CYQJJ
Windows Server 2012 MultiPoint Premium XNH6W-2V9GX-RGJ4K-Y8X6F-QGJ2G
Windows Server 2012 Datacenter 48HP8-DN98B-MYWDG-T2DCC-8W83P
Operating system edition KMS Client Setup Key
Windows 7 Professional FJ82H-XT6CR-J8D7P-XQJJ2-GPDD4
Windows 7 Professional N MRPKT-YTG23-K7D7T-X2JMM-QY7MG
Windows 7 Professional E W82YF-2Q76Y-63HXB-FGJG9-GF7QX
Windows 7 Enterprise 33PXH-7Y6KF-2VJC9-XBBR8-HVTHH
Windows 7 Enterprise N YDRBP-3D83W-TY26F-D46B2-XCKRJ
Windows 7 Enterprise E C29WB-22CC8-VJ326-GHFJW-H9DH4
Windows Server 2008 R2 Web 6TPJF-RBVHG-WBW2R-86QPH-6RTM4
Windows Server 2008 R2 HPC edition TT8MH-CG224-D3D7Q-498W2-9QCTX
Windows Server 2008 R2 Standard YC6KT-GKW9T-YTKYR-T4X34-R7VHC
Windows Server 2008 R2 Enterprise 489J6-VHDMP-X63PK-3K798-CPX3Y
Windows Server 2008 R2 Datacenter 74YFP-3QFB3-KQT8W-PMXWJ-7M648
Windows Server 2008 R2 for Itanium-based Systems GT63C-RJFQ3-4GMB6-BRFB9-CB83V
Operating system edition KMS Client Setup Key
Windows Vista Business YFKBB-PQJJV-G996G-VWGXY-2V3X8
Windows Vista Business N HMBQG-8H2RH-C77VX-27R82-VMQBT
Windows Vista Enterprise VKK3X-68KWM-X2YGT-QR4M6-4BWMV
Windows Vista Enterprise N VTC42-BM838-43QHV-84HX6-XJXKV
Windows Web Server 2008 WYR28-R7TFJ-3X2YQ-YCY4H-M249D
Windows Server 2008 Standard TM24T-X9RMF-VWXK6-X8JC9-BFGM2
Windows Server 2008 Standard without Hyper-V W7VD6-7JFBR-RX26B-YKQ3Y-6FFFJ
Windows Server 2008 Enterprise YQGMW-MPWTJ-34KDK-48M3W-X4Q6V
Windows Server 2008 Enterprise without Hyper-V 39BXF-X8Q23-P2WWT-38T2F-G3FPG
Windows Server 2008 HPC RCTX3-KWVHP-BR6TB-RB6DM-6X7HP
Windows Server 2008 Datacenter 7M67G-PC374-GR742-YH8V4-TCBY3
Windows Server 2008 Datacenter without Hyper-V 22XQ2-VRXRG-P8D42-K34TD-G3QQC
Windows Server 2008 for Itanium-Based Systems 4DWFP-JF3DJ-B7DTH-78FJB-PDRHK

使用dd pv gzip进行硬盘的克隆

使用dd可以对硬盘(操作系统)进行克隆,以便批量恢复。gzip可以对克隆后的文件进行压缩。pv可以实时显示克隆任务的进度。以下简述一下步骤,仅供参考:

(一)获取克隆母盘的实际大小

使用parted命令,支持GPT分区;设置unit为Byte,打印硬盘信息,如下所示:

$ sudo parted /dev/sda
GNU Parted 2.3
Using /dev/sda
Welcome to GNU Parted! Type ‘help’ to view a list of commands.
(parted) unit B
(parted) print
Model: ATA WDC WDS120G1G0A- (scsi)
Disk /dev/sda: 120034123776B
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
1 1048576B 537919487B 536870912B fat32 EFI System Partition boot
2 537919488B 793772031B 255852544B ext2
3 793772032B 120033640447B 119239868416B lvm

驱动器大小为红色字体位置。

(二)备份压缩硬盘到镜像

dd if=/dev/sda | pv -s 120034123776 | gzip –fast > /media/usbdisk/backup.img

pv -s后的参数是步骤(一)中获取的硬盘大小;绿色字体部分为克隆文件的保存路径,使用了gzip快速压缩,备份过程结束。

(三)还原硬盘镜像

gzip -dc /media/usbdisk/backup.img | pv -s 120034123776 | dd of=/dev/sda

同理,pv -s后的参数是步骤(一)中获取的硬盘大小。

至此,使用dd配合gzip和pv对硬盘进行克隆完成。

参考链接:http://allgood38.io/a-snapshot-of-your-computer-with-dd-pv-and-gzip-part-1.html

解决Ubuntu使用dd克隆后网卡从eth1开始的问题

ubuntu下使用dd可以对相同配置的电脑进行克隆,在实际使用中会遇到一个问题,主机的网卡是eth0开始,而dd克隆的克隆机网卡却从eth1开始。经过查询资料,这是udev的特性,默认会在规则中记录网卡物理地址(MAC地址)和ethX的对应关系。因此,在备份之前从主机中删除(或还原后从克隆机中删除)

$ sudo rm /etc/udev/rules.d/70-persistent-net.rules

即可解决网卡从eth1开始的问题

Ubuntu解决设置静态IP后开机未连接网线导致开机速度缓慢的问题

通过/etc/network/interface设置了静态IP地址:

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
address 192.168.2.182
netmask 255.255.255.0
gateway 192.168.2.1
dns-nameservers 192.168.2.1

在不插网线的情况下,开机时候会显示waiting for network configuration,大约会卡住2分钟左右。通过网络搜索发现在/etc/init/failsafe.conf发现了问题所在根源,大致和upstart工作机制有关。

为解决这个问题,采取了一个简单粗暴的方法,即注释有关提示和sleep语句,加快系统启动速度,修改后的failsafe.conf如下:

# failsafe

description “Failsafe Boot Delay”
author “Clint Byrum <clint@ubuntu.com>”

start on filesystem and net-device-up IFACE=lo
stop on static-network-up or starting rc-sysinit

emits failsafe-boot

console output

script
# Determine if plymouth is available
if [ -x /bin/plymouth ] && /bin/plymouth –ping ; then
PLYMOUTH=/bin/plymouth
else
PLYMOUTH=”:”
fi

# The point here is to wait for 2 minutes before forcibly booting
# the system. Anything that is in an “or” condition with ‘started
# failsafe’ in rc-sysinit deserves consideration for mentioning in
# these messages. currently only static-network-up counts for that.

# sleep 20

# Plymouth errors should not stop the script because we *must* reach
# the end of this script to avoid letting the system spin forever
# waiting on it to start.
# $PLYMOUTH message –text=”Waiting for network configuration…” || :
# sleep 40

# $PLYMOUTH message –text=”Waiting up to 60 more seconds for network configuration…” || :
# sleep 59
# $PLYMOUTH message –text=”Booting system without full network configuration…” || :

# give user 1 second to see this message since plymouth will go
# away as soon as failsafe starts.
# sleep 1
exec initctl emit –no-wait failsafe-boot
end script

post-start exec logger -t ‘failsafe’ -p daemon.warning “Failsafe of 120 seconds reached.”

Ubuntu禁用USB设备插入提示和自动运行

方法一:在终端中运行以下命令

$ sudo gsettings set org.gnome.desktop.media-handling autorun-never true

$ sudo gsettings set org.gnome.desktop.media-handling automount-open false

需要注意的是通过telnet或openssh远程登录执行无效。

方法二:使用dconf editor进行设置

进入org->gnome->desktop->media-handling,选中automount和autorun-never,取消选中automount-open

为什么TCP不可靠 why is my tcp not reliable

【转自https://blog.netherlabs.nl/articles/2009/01/18/the-ultimate-so_linger-page-or-why-is-my-tcp-not-reliable】

This post is about an obscure corner of TCP network programming, a corner where almost everybody doesn’t quite get what is going on. I used to think I understood it, but found out last week that I didn’t.

So I decided to trawl the web and consult the experts, promising them to write up their wisdom once and for all, in hopes that this subject can be put to rest.

The experts (H. Willstrand, Evgeniy Polyakov, Bill Fink, Ilpo Jarvinen, and Herbert Xu) responded, and here is my write-up.

Even though I refer a lot to the Linux TCP implementation, the issue described is not Linux-specific, and can occur on any operating system.

What is the issue?

Sometimes, we have to send an unknown amount of data from one location to another. TCP, the reliable Transmission Control Protocol, sounds like it is exactly what we need. From the Linux tcp(7) manpage:

“TCP provides a reliable, stream-oriented, full-duplex connection between two sockets on top of ip(7), for both v4 and v6 versions. TCP guarantees that the data arrives in order and retransmits lost packets. It generates and checks a per-packet checksum to catch transmission errors.”

However, when we naively use TCP to just send the data we need to transmit, it often fails to do what we want – with the final kilobytes or sometimes megabytes of data transmitted never arriving.

Let’s say we run the following two programs on two POSIX compliant operating systems, with the intention of sending 1 million bytes from program A to program B (programs can be found here):

A:

   
      sock = socket(AF_INET, SOCK_STREAM, 0);  
      connect(sock, &remote, sizeof(remote));
      write(sock, buffer, 1000000);             // returns 1000000
      close(sock);

B:

    int sock = socket(AF_INET, SOCK_STREAM, 0);
    bind(sock, &local, sizeof(local));
    listen(sock, 128);
    int client=accept(sock, &local, locallen);
    write(client, "220 Welcome\r\n", 13);

    int bytesRead=0, res;
    for(;;) {
        res = read(client, buffer, 4096);
        if(res < 0)  {
            perror("read");
            exit(1);
        }
        if(!res)
            break;
        bytesRead += res;
    }
    printf("%d\n", bytesRead);

Quiz question – what will program B print on completion?

A) 1000000
B) something less than 1000000
C) it will exit reporting an error
D) could be any of the above

The right answer, sadly, is ‘D’. But how could this happen? Program A reported that all data had been sent correctly!

What is going on

Sending data over a TCP socket really does not offer the same ‘it hit the disk’ semantics as writing to a normal file does (if you remember to call fsync()).

In fact, all a successful write() in the TCP world means is that the kernel has accepted your data, and will now try to transmit it in its own sweet time. Even when the kernel feels that the packets carrying your data have been sent, in reality, they’ve only been handed off to the network adapter, which might actually even send the packets when it feels like it.

From that point on, the data will traverse many such adapters and queues over the network, until it arrives at the remote host. The kernel there will acknowledge the data on receipt, and if the process that owns the socket is actually paying attention and trying to read from it, the data will finally have arrived at the application, and in filesystem speak, ‘hit the disk’.

Note that the acknowledgment sent out only means the kernel saw the data – it does not mean the application did!

OK, I get all that, but why didn’t all data arrive in the example above?

When we issue a close() on a TCP/IP socket, depending on the circumstances, the kernel may do exactly that: close down the socket, and with it the TCP/IP connection that goes with it.

And this does in fact happen – even though some of your data was still waiting to be sent, or had been sent but not acknowledged: the kernel can close the whole connection.

This issue has led to a large number of postings on mailing lists, Usenet and fora, and these all quickly zero in on the SO_LINGER socket option, which appears to have been written with just this issue in mind:

“When enabled, a close(2) or shutdown(2) will not return until all queued messages for the socket have been successfully sent or the linger timeout has been reached. Otherwise, the call returns immediately and the closing is done in the background. When the socket is closed as part of exit(2), it always lingers in the background.”

So, we set this option, rerun our program. And it still does not work, not all our million bytes arrive.

How come?

It turns out that in this case, section 4.2.2.13 of RFC 1122 tells us that a close() with any pending readable data could lead to an immediate reset being sent.

“A host MAY implement a ‘half-duplex’ TCP close sequence, so that an application that has called CLOSE cannot continue to read data from the connection. If such a host issues a CLOSE call while received data is still pending in TCP, or if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost.”

And in our case, we have such data pending: the “220 Welcome\r\n” we transmitted in program B, but never read in program A!

If that line has not been sent by program B, it is most likely that all our data would have arrived correctly.

So, if we read that data first, and LINGER, are we good to go?

Not really. The close() call really does not convey what we are trying to tell the kernel: please close the connection after sending all the data I submitted through write().

Luckily, the system call shutdown() is available, which tells the kernel exactly this. However, it alone is not enough. When shutdown() returns, we still have no indication that everything was received by program B.

What we can do however is issue a shutdown(), which will lead to a FIN packet being sent to program B. Program B in turn will close down its socket, and we can detect this from program A: a subsequent read() will return 0.

Program A now becomes:

    sock = socket(AF_INET, SOCK_STREAM, 0);  
    connect(sock, &remote, sizeof(remote));
    write(sock, buffer, 1000000);             // returns 1000000
    shutdown(sock, SHUT_WR);
    for(;;) {
        res=read(sock, buffer, 4000);
        if(res < 0) {
            perror("reading");
            exit(1);
        }
        if(!res)
            break;
    }
    close(sock);

So is this perfection?

Well.. If we look at the HTTP protocol, there data is usually sent with length information included, either at the beginning of an HTTP response, or in the course of transmitting information (so called ‘chunked’ mode).

And they do this for a reason. Only in this way can the receiving end be sure it received all information that it was sent.

Using the shutdown() technique above really only tells us that the remote closed the connection. It does not actually guarantee that all data was received correctly by program B.

The best advice is to send length information, and to have the remote program actively acknowledge that all data was received.

This only works if you have the ability to choose your own protocol, of course.

What else can be done?

If you need to deliver streaming data to a ‘stupid TCP/IP hole in the wall’, as I’ve had to do a number of times, it may be impossible to follow the sage advice above about sending length information, and getting acknowledgments.

In such cases, it may not be good enough to accept the closing of the receiving side of the socket as an indication that everything arrived.

Luckily, it turns out that Linux keeps track of the amount of unacknowledged data, which can be queried using the SIOCOUTQ ioctl(). Once we see this number hit 0, we can be reasonably sure our data reached at least the remote operating system.

Unlike the shutdown() technique described above, SIOCOUTQ appears to be Linux-specific. Updates for other operating systems are welcome.

The sample code contains an example of how to use SIOCOUTQ.

But how come it ‘just worked’ lots of times!

As long as you have no unread pending data, the star and moon are aligned correctly, your operating system is of a certain version, you may remain blissfully unimpacted by the story above, and things will quite often ‘just work’. But don’t count on it.

Some notes on non-blocking sockets

Volumes of communications have been devoted the the intricacies of SO_LINGER versus non-blocking (O_NONBLOCK) sockets. From what I can tell, the final word is: don’t do it. Rely on the shutdown()-followed-by-read()-eof technique instead. Using the appropriate calls to poll/epoll/select(), of course.

A few words on the Linux sendfile() and splice() system calls

It should also be noted that the Linux system calls sendfile() and splice() hit a spot in between – these usually manage to deliver the contents of the file to be sent, even if you immediately call close() after they return.

This has to do with the fact that splice() (on which sendfile() is based) can only safely return after all packets have hit the TCP stack since it is zero copy, and can’t very well change its behaviour if you modify a file after the call returns!

Please note that the functions do not wait until all the data has been acknowledged, it only waits until it has been sent.

【文中提到的源码:tcp-programs.tar

一句话总结:使用ioctl(sockfd, SIOCOUTQ, &value),通过value可获取当前TCP发送缓冲区待发送数据的大小,从而通过一条一条的发送数据来从协议层控制数据包成功发送至对端。