go get proxy with gitlab

2015-12-29 17:07:11 #tool

1.Fuck

工欲善其事，必先利其器。最近在处理gitlab的各种问题，接入公司LDAP、启用HTTPS、邮件通知、参数调优，当然也有个最棘手的问题，那就是支持golang 的go get命令。之前老东家有个牛人做过，但是具体也不知道是怎么做的，于是抽时间把这个东西搞下，不能用go get还是很蛋疼的。

我不相信这么严重的问题 Gitlab 公司会不关心，于是翻了翻gitlab在github的issues，果然找到了相应的issue：

gitlab & golang's "go get" (or default to .git repo extension)
https://github.com/gitlabhq/gitlabhq/issues/5769

另外也找到了响应的PR：

Added support for Go's repository retrieval. #5958
https://github.com/gitlabhq/gitlabhq/pull/5958

2.why

既然官方有了PR应该已经解决了啊，看了下时间也是去年merged的，又看了下自己线上的gitlab版本，也是修复的。奇怪了，只能重头查了

我执行如下命令,加上-v查看详情：

go get -v git.test.com/golang/ilog

返回结果：

Fetching https://git.test.com/golang/ilog?go-get=1
Parsing meta tags from https://git.test.com/golang/ilog?go-get=1 (status code 200)
import "git.test.com/golang/ilog": parse https://git.test.com/golang/ilog?go-get=1: no go-import meta tags
package git.test.com/golang/ilog: unrecognized import path "git.test.com/golang/ilog"

很明显，提示no go-import meta tags，这是什么东西，翻阅官方资料：

<meta name="go-import" content="import-prefix vcs repo-root">

https://golang.org/cmd/go/#hdr-Remote_import_paths

The import-prefix is the import path corresponding to the repository root. It must be a prefix or an exact match of the package being fetched with “go get”. If it’s not an exact match, another http request is made at the prefix to verify the tags match.

The meta tag should appear as early in the file as possible. In particular, it should appear before any raw JavaScript or CSS, to avoid confusing the go command’s restricted parser.

哦，原来是会去读一个自定义的meta信息，看了下之前gitlab官方修复的那个PR：


if controller_name == 'projects' && action_name == 'show'
     %meta{name: "go-import", content: "#{@project.web_url_without_protocol} git #{@project.web_url}.git"}

没看出来什么问题啊，也支持了，继续回到开始，执行如下命令：

curl https://git.test.com/golang/ilog?go-get=1

返回信息：

<html><body>You are being <a href="https://git.test.com/users/sign_in">redirected</a>.</body></html>

哦，原来TM的跳转到登录页面了，因为我get的是自己的私有库。肯定要涉及到认证，看了上面官方的修复，发现 @project.web_url 这个应该是走的http或是https，但是查了golang的官方文档，go get支持的协议很多，而且大家经常用ssh走自己的密钥对，所以这里最好改成ssh协议，不想在gitlab上直接改，于是打算在gitlab和nginx中间搞个proxy。

3.Done

问题现在很清楚了，于是动手开干，还是用go来做这件事，其实很简单，就是根据用户go get的东西重新生成一个meta信息，然后go get 会调用git clone去执行命令，我们要做的就是保证它走ssh就好了。直接上代码：


package main

import (
	"fmt"
	logger "log"
	"net/http"
	"os"
	"path/filepath"
	"strings"

	"github.com/go-martini/martini"
)

const (
	LOGPATH = "/tmp/goget-proxy.log"
	LISTEN  = "127.0.0.1:9999"
	RESULT  = "<!DOCTYPE html><html><head><meta content='%s%s git ssh://git@%s%s' name='go-import'></head></html>"
)

func main() {
	m := martini.Classic()

	logfile := filepath.Join(LOGPATH)
	f, err := os.OpenFile(logfile, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
	if err != nil {
		fmt.Println("error opening file: %v", err)
	}
	defer f.Close()
	m.Map(logger.New(f, "", logger.Ldate|logger.Ltime))

	// https://github.com/go-martini/martini/issues/34
	m.Get("/:namespace/:repo", proxyHandler)

	m.RunOnAddr(LISTEN)
}

func proxyHandler(w http.ResponseWriter, r *http.Request, params martini.Params) {
	ns := strings.ToLower(params["namespace"])
	repo := strings.ToLower(params["repo"])
	if ns == "" || repo == "" {
		w.WriteHeader(http.StatusBadRequest)
		fmt.Fprintf(w, "%s", string("params error"))
	}
	w.Header().Set("Content-Type", "application/json; charset=utf-8")
	w.WriteHeader(http.StatusOK)
	//RequestURI
	uriarr := strings.SplitN(r.RequestURI, "?", 2)
	res := fmt.Sprintf(RESULT, r.Host, uriarr[0], r.Host, uriarr[0])
	w.Write([]byte(res))
}

简单写了个demo，还有很多地方需要改进，继续测试下：

哦对了还有 nginx 的配置文件需要添加配置：

    proxy_set_header   Host    $http_host;
    if ($args ~* "^go-get=1") {
      proxy_pass http://127.0.0.1:9999;
    }

启动服务：

/usr/local/goget-proxy/goget-proxy > /dev/null &

测试结果：


go get -v git.test.com/golang/ilog

Fetching https://git.test.com/golang/ilog?go-get=1
Parsing meta tags from https://git.test.com/golang/ilog?go-get=1 (status code 200)
get "git.test.com/golang/ilog": found meta tag main.metaImport{Prefix:"git.test.com/golang/ilog", VCS:"git", RepoRoot:"ssh://[email protected]/golang/ilog"} at https://git.test.com/golang/ilog?go-get=1
git.test.com/golang/ilog (download)
git.test.com/golang/ilog

OK,搞定。

对了，最后给一国外的哥们儿给出的办法，自己没有试过，自己改下nginx配置就行了：

 
if ($args ~* "^go-get=1") {
  set $condition goget;
}
if ($uri ~ ^/([a-zA-Z0-9_-]+)/([a-zA-Z0-9_-]+)/.*$) {
  set $condition "${condition}path";   
}
if ($condition = gogetpath) {
  return 200 "<!DOCTYPE html><html><head><meta content='your.domain.com/$1/$2 git http://your.domain.com/$1/$2.git' name='go-import'></head></html>";
}

Change the FQDN to fit your setup.