Cnpm,官方解釋爲Company npm。html
因爲團隊需求,如今須要搭建一個npm私服,用來更方便地管理團隊的組件庫,而且更快速更穩定地提供服務,我踏上了搭建npm私服的道路。node
git clone https://github.com/cnpm/cnpmjs.org.git
mysql
下載完代碼後,我們先來大概瞄一眼項目目錄react
+-- bin/ ---一些命令腳本
| --- nodejsctl ---npm start啓動的腳本
| --- ...
+-- common/ ---公共目錄,存放日誌配置、郵件配置等
+-- config/
| --- index.js ---主要配置文件
+-- controllers/
| --- registry/ ---7001端口的controller層
| --- web/ ---7002端口的controller層
| --- sync_module_worker.js ---sync的主進程文件
| --- ...
+-- docs/
| --- db.sql ---數據庫建表sql
| --- ...
+-- lib/
+-- middleware/
+-- models/ ---數據庫操做目錄
+-- public/
+-- routes/
| --- registry.js ---7001端口的路由文件
| --- web.js ---7002端口的路由文件
| --- ...
+-- servers/
| --- registry.js ---7001端口的服務器入口文件
| --- web.js ---7002端口的服務器入口文件
| --- ...
+-- services/
+-- sync/
| --- sync_all.js ---sync模式選擇all時執行的文件
| --- sync_exist.js ---sync模式選擇exist時執行的文件
| --- ...
+-- test/
+-- tools/
+-- view/
--- dispatch.js ---啓動npm服務的主要文件,bin/nodejsctl中執行的就是這個文件
--- package.json
複製代碼
咱們能夠發現,cnpm使用的是koa框架,結構是經典的route->controller->services->modelgit
同步模塊的具體流程是在controllers/sync_module_worker.js文件中的github
1. 根據設置的sync模式,從上游源中下載模塊到一個臨時路徑/root/.cnpmjs.org/downloads/xxxxx.tgz
2. 調用nfs.upload方法將臨時路徑存儲的tgz上傳到指定存儲位置
3. 不管是否上傳成功,都刪除剛剛下載的臨時文件
複製代碼
看到這裏,喜歡思考的同窗或許會說了,我到底應該怎麼搭建本身的npm私服?你說了半天,我仍是啥都不知道,好比吧:web
var config = {
version: version,
dataDir: dataDir,
/**
* Cluster mode
*/
enableCluster: true,
numCPUs: os.cpus().length,
/*
* server configure
*/
registryPort: 7001,
webPort: 7002,
bindingHost: '0.0.0.0', // only binding on 127.0.0.1 for local access
// debug mode
// if in debug mode, some middleware like limit wont load
// logger module will print to stdout
debug: process.env.NODE_ENV === 'development',
// page mode, enable on development env
pagemock: process.env.NODE_ENV === 'development',
// session secret
sessionSecret: 'cnpmjs.org test session secret',
// max request json body size
jsonLimit: '10mb',
// log dir name
logdir: path.join(dataDir, 'logs'),
// update file template dir
uploadDir: path.join(dataDir, 'downloads'),
// web page viewCache
viewCache: false,
// config for koa-limit middleware
// for limit download rates
limit: {
enable: false,
token: 'koa-limit:download',
limit: 1000,
interval: 1000 * 60 * 60 * 24,
whiteList: [],
blackList: [],
message: 'request frequency limited, any question, please contact fengmk2@gmail.com',
},
enableCompress: true, // enable gzip response or not
// default system admins
admins: {
// name: email
sunxiuguo: 'sunxiuguo@my.com',
},
// email notification for errors
// check https://github.com/andris9/Nodemailer for more informations
mail: {
enable: false,
appname: 'cnpmjs.org',
from: 'cnpmjs.org mail sender <adderss@gmail.com>',
service: 'gmail',
auth: {
user: 'address@gmail.com',
pass: 'your password'
}
},
logoURL: 'https://os.alipayobjects.com/rmsportal/oygxuIUkkrRccUz.jpg', // cnpm logo image url
adBanner: '',
customReadmeFile: '', // you can use your custom readme file instead the cnpm one
customFooter: '', // you can add copyright and site total script html here
npmClientName: 'cnpm', // use `${name} install package`
packagePageContributorSearch: true, // package page contributor link to search, default is true
// max handle number of package.json `dependencies` property
maxDependencies: 200,
// backup filepath prefix
backupFilePrefix: '/cnpm/backup/',
/**
* database config
*/
database: {
db: '******', // 庫名
username: '*********', // 數據庫用戶名
password: '************', // 數據庫密碼
// the sql dialect of the database
// - currently supported: 'mysql', 'sqlite', 'postgres', 'mariadb'
dialect: 'mysql',
// the Docker container network hostname defined at docker-compose.yml
host: '**************', // 數據庫域名
// custom port; default: 3306
port: 3318, // 數據庫端口號
// use pooling in order to reduce db connection overload and to increase speed
// currently only for mysql and postgresql (since v1.5.0)
pool: {
maxConnections: 10,
minConnections: 0,
maxIdleTime: 30000
},
dialectOptions: {
// if your server run on full cpu load, please set trace to false
trace: true,
},
// the storage engine for 'sqlite'
// default store into ~/.cnpmjs.org/data.sqlite
storage: path.join(dataDir, 'data.sqlite'),
logging: !!process.env.SQL_DEBUG,
},
// package tarball store in local filesystem by default
nfs: aws.create({
accessKeyId: '*************', // s3 accessKeyId
secretAccessKey: '****************', // s3 secretAccessKey
// change to your endpoint
endpoint: '*****************', // https://docs.aws.amazon.com/zh_cn/general/latest/gr/rande.html
bucket: 'npm-online', // s3 bucket名稱
signatureVersion: 'v4', // s3 api版本
mode: 'private', // public: 經過url下載tar包; private: 經過key下載tar包
}),
// if set true, will 302 redirect to `nfs.url(dist.key)`
downloadRedirectToNFS: false,
// registry url name
registryHost: 'registry.npm.my.com',
/**
* registry mode config
*/
// enable private mode or not
// private mode: only admins can publish, other users just can sync package from source npm
// public mode: all users can publish
enablePrivate: false,
// registry scopes, if dont set, means do not support scopes
scopes: [ '@cnpm', '@sunxiuguo', '@companyName' ],
// some registry already have some private packages in global scope
// but we want to treat them as scoped private packages,
// so you can use this white list.
privatePackages: [],
/**
* sync configs
*/
// the official npm registry
// cnpm wont directly sync from this one
// but sometimes will request it for some package infomations
// please dont change it if not necessary
officialNpmRegistry: 'https://registry.npmjs.com',
officialNpmReplicate: 'https://replicate.npmjs.com',
// sync source, upstream registry
// If you want to directly sync from official npm registry
// please drop them an email first
sourceNpmRegistry: 'https://registry.npm.taobao.org',
sourceNpmWeb: 'https://npm.taobao.org',
// upstream registry is base on cnpm/cnpmjs.org or not
// if your upstream is official npm registry, please turn it off
sourceNpmRegistryIsCNpm: true,
// if install return 404, try to sync from source registry
syncByInstall: true,
// sync mode select
// none: do not sync any module, proxy all public modules from sourceNpmRegistry
// exist: only sync exist modules
// all: sync all modules
syncModel: 'exist', // 'none', 'all', 'exist'
syncConcurrency: 1,
// sync interval, default is 10 minutes
syncInterval: '10m',
// sync polular modules, default to false
// because cnpm can not auto sync tag change for now
// so we want to sync popular modules to ensure their tags
syncPopular: false,
syncPopularInterval: '1h',
// top 100
topPopular: 100,
// sync devDependencies or not, default is false
syncDevDependencies: false,
// try to remove all deleted versions from original registry
syncDeletedVersions: true,
// changes streaming sync
syncChangesStream: false,
handleSyncRegistry: 'http://127.0.0.1:7001',
// default badge subject
badgeSubject: 'cnpm',
// defautl use https://badgen.net/
badgeService: {
url: function(subject, status, options) {
options = options || {};
let url = `https://badgen.net/badge/${utility.encodeURIComponent(subject)}/${utility.encodeURIComponent(status)}`;
if (options.color) {
url += `/${utility.encodeURIComponent(options.color)}`;
}
if (options.icon) {
url += `?icon=${utility.encodeURIComponent(options.icon)}`;
}
return url;
},
},
packagephobiaURL: 'https://packagephobia.now.sh',
packagephobiaSupportPrivatePackage: false,
// custom user service, @see https://github.com/cnpm/cnpmjs.org/wiki/Use-Your-Own-User-Authorization
// when you not intend to ingegrate with your company user system, then use null, it would
// use the default cnpm user system
userService: null,
// always-auth https://docs.npmjs.com/misc/config#always-auth
// Force npm to always require authentication when accessing the registry, even for GET requests.
alwaysAuth: false,
// if you are behind firewall, need to request through http proxy, please set this
// e.g.: `httpProxy: 'http://proxy.mycompany.com:8080'`
// httpProxy: 'http://gfw.guazi-corp.com',
httpProxy: null,
// snyk.io root url
snykUrl: 'https://snyk.io',
// https://github.com/cnpm/cnpmjs.org/issues/1149
// if enable this option, must create module_abbreviated and package_readme table in database
enableAbbreviatedMetadata: true,
// global hook function: function* (envelope) {}
// envelope format please see https://github.com/npm/registry/blob/master/docs/hooks/hooks-payload.md#payload
globalHook: null,
opensearch: {
host: '',
},
};
複製代碼
是否是成功打開文件了~恭喜你!你成功的邁出了第二步!
什麼?你問第一步是什麼?第一步是clone代碼啊sql
syncModel屬性控制sync模式,分爲none,exist,all三種狀況。docker
上游源就是你同步包的地址,好比你的上游源是淘寶源,那麼你的npm私服就會從淘寶源進行包的同步。shell
uploadDir屬性設置同步的模塊存放的臨時路徑,默認爲path.join(dataDir, 'downloads'),即root/.cnpmjs.org/downloads
nfs屬性控制包存儲,包括上傳,下載等等。nfs的意思是network file system
registryPort屬性默認爲7001,webPort屬性默認爲7002.
registry服務主要是用來提供給用戶源相關操做,好比設置npm源 web服務主要是提供給用戶的一個圖形化管理界面,好比在界面上查詢某個模塊
由於咱們的npm私服是放在docker裏,包文件不可能使用fs-cnpm存儲在docker裏,因此咱們接入了amazon s3的對象存儲服務。
官方提供了接入npm的協議NFS-Guide
Can download the uploaded file through http request. like qn-cnpm.
uploadBuffer: use options.key to customize the filename, then callback {url: 'http://test.com/xxx.tgz'}.
url: accept a key and respond download url.
remove: remove file by key
Can not download by http request. like sfs-client or oss-cnpm.
uploadBuffer: upload the file, and must callback {key: 'xxx'}, so cnpmjs.org can record the key, and use this key to download or remove.
download: need provide download api to download the file by key.
createDownloadStream: streaming download file by key
remove: remove file by key
複製代碼
若是存儲系統支持經過http請求下載包文件,就提供uploadBuffer,url,remove方法 若是存儲系統不支持經過http請求下載包文件,就須要提供uploadBuffer,download,createDownloadStream,remove方法。
而且全部方法都須要是async的,或者是generatord的。
由於咱們使用的bucket,首先要提供一個create的方法來實例化一個s3對象。
exports.create = function (options) {
return new AwsWrapper(options);
};
function AwsWrapper(options) {
this.client = new S3(options);
this.mode = options.mode;
this.bucket = options.bucket;
var params = {
Bucket: options.bucket,
CreateBucketConfiguration: {
LocationConstraint: ":npm"//桶所在服務區
}
};
this.client.createBucket(params, function (err, data) {
if (err) {
// an error occurred
logger.syncInfo(err);
} else {
// successful response
console.log(data.Location);
}
});
}
複製代碼
而後按照協議提供對應的方法
調用路徑在controllers/registry/package/save.js,當publish包時會進入這個方法,入參爲fileBuffer和options;
這個方法很簡單,只需調用對應存儲系統提供的api,把buffer上傳便可。
const key = trimKey(options.key);
logger.syncInfo(`enter aws->uploadBuffer key=${key}`);
let result = {
key,
};
let uploadParams = {
Bucket: this.bucket,
Key: key,
Body: fileBuffer
};
this.client.upload (uploadParams, function (err, data) {
if (err) {
logger.syncInfo(err);
}
});
複製代碼
調用路徑在controllers/sync_module_worker.js,當從上游同步包的時候會進入這個方法,入參爲filePath和options
upload和uploadBuffer不一樣的是,upload是讀取傳入的filePath的文件做爲body上傳,uploadBuffer是直接把傳入的buffer對象做爲body上傳。
const key = trimKey(options.key);
logger.syncInfo(`進入aws->upload key=${key} filePath=${filePath}`);
let result = {
key,
};
let fileStream = fs.createReadStream(filePath);
fileStream.on('error', function(err) {
logger.syncInfo(err);
});
let uploadParams = {
Bucket: this.bucket,
Key: key,
Body: fileStream
};
await this.client.upload (uploadParams, function (err, data) {
if (err) {
logger.syncInfo(err);
}
});
return result;
複製代碼
調用路徑在controllers/registry/package/download.js,當下載包的時候會進入這個方法,入參爲key和options,用於獲取包的存放的url地址
const params = { Bucket: this.bucket, Key: trimKey(key) };
logger.syncInfo(`進入aws->url key=${key} trimKey=${trimKey(key)}`);
return this.client.getSignedUrl('getObject', params);
複製代碼
調用路徑在controllers/registry/package/remove.js controllers/registry/package/remove_version.js 和 controllers/sync_module_worker.js,當刪除包或者刪除版本的時候會進入這個方法,入參爲key和options
const params = { Bucket: this.bucket, Key: trimKey(key) };
logger.syncInfo(`進入aws->remove key=${key} trimKey=${trimKey(key)}`);
await this.client.deleteObject(params);
複製代碼
調用路徑在controllers/utils.js,當下載包的時候會進入這個方法,入參爲key和options,把可讀流做爲用戶下載請求的response的body
utils.js中是惟一調用download和createDownloadStream的地方,然而咱們仔細看源碼,能夠發現若是定義了createDownloadStream方法,就會直接返回createDownloadStream的結果,而不會繼續進行下面的download操做。
也就是說,咱們只須要定義createDownloadStream方法便可。
const params = { Bucket: this.bucket, Key: trimKey(key) };
logger.syncInfo(`進入aws->createDownloadStream key=${key} trimKey=${trimKey(key)}`);
return this.client.getObject(params).createReadStream();
複製代碼
設置npm源爲剛搭建的私有源
npm config set registry http://registry.npm.my.com
查看當前的registry地址
npm get registry
清理npm緩存
npm cache clean --force
隨便選一個項目 刪除node_modules包
rm -rf node_modules
安裝
npm install
手動同步一個包,好比react(能夠在web界面上的/sync/路徑下輸入包名進行同步)
npm sync react
複製代碼
只是手動安裝一個項目的依賴包可能沒法說明什麼,咱們來寫一個簡單的自動測試腳本
require('shelljs/global')
const logger = require('./log').logger;
const fs = require('fs');
const MODULE_DIR = '/node_modules';
const PARENT_PATH = '/Users/sunxiuguo/project/';
const projectName = [
'test1',
'test2',
'test3',
'test4',
]
const absolutePath = projectName.map(item => {
return {
modulesPath: PARENT_PATH + item + MODULE_DIR,
parentPath: PARENT_PATH + item,
}
});
const startTime = new Date('2018/11/06 21:00:000').getTime();
const endTime = new Date('2018/11/08 10:00:000').getTime();
/**
* 讀取路徑
* @param path
*/
function getStat(path){
if (exec(`cd ${path}`).code == 0) {
return true;
}
return false;
}
async function npmCachecleanAndInstall(projectPath) {
cd(projectPath);
logger.info(`cd ${projectPath}`);
exec('pwd');
await execAndLogAsync(`npm cache clean --force`);
await execAndLogAsync(`npm install --registry=http://registry.npm.my.com`)
}
async function execAndLogAsync(command) {
logger.info(command);
let result = await exec(command);
if (result.stderr) {
logger.error(result.stderr);
}
}
async function install(path) {
logger.info(`install: path = ${JSON.stringify(path)}`);
let isExists = getStat(path.modulesPath);
if (!isExists) {
// 若是不存在 npm install
logger.info(`install: 不存在${path.modulesPath}目錄,開始npm install`)
await npmCachecleanAndInstall(path.parentPath);
} else {
// 若是存在,刪除 && npm install
logger.info(`install: 存在${path.modulesPath}目錄,開始刪除`)
await execAndLogAsync(`rm -rf ${path.modulesPath}`);
logger.info(`install: 刪除${path.modulesPath}成功,開始npm install`)
await npmCachecleanAndInstall(path.parentPath);
}
}
logger.info('beginning!')
if (new Date().getTime() < startTime) {
logger.info(`未到開始時間, 開始時間爲2018/11/06 22:00:000`)
exit(1);
}
for(let path of absolutePath) {
(async function(){
while (new Date().getTime() < endTime) {
await install(path);
}
})()
}
複製代碼
首先添加一個用戶,添加後會默認以這個用戶登陸
npm adduser
username:sunxiuguo
password:sunxiuguo
email:sunxiuguo1@qq.com
進入要發佈的目錄
npm publish
查看剛纔發佈的包信息(也能夠在web界面上查詢)
npm view moduleName
這時若是其餘小夥伴也要發佈這個包,就會報錯了,由於其餘小夥伴不是這個包的maintainer
我們來查看一下這個包的owner都有誰
npm owner ls moduleName
而後添加wangwang爲這個包的owner
npm owner add wangwang moduleName
什麼?!!又報錯了?!
不要慌,那是由於根本沒有wangwang這個用戶,須要執行npm adduser添加一下
npm adduser
username:wangwang
password:1231131313
email:wangwang@guazi.com
再次添加owner
npm owner add wangwang moduleName
成功了!今後wangwang也能夠發佈這個包了
之後若是想登陸,直接Login便可
npm login
username:wangwang
password:1231131313
email:wangwang@guazi.com
複製代碼
強調一下,撤銷發佈包是很危險的一件事情,若是有其餘同窗用了你的包,而後你心血澎湃地把這個包撤銷了??其餘同窗確定一臉問號
npm unpublish moduleName
複製代碼
- 根據規範,只有在發包的24小時內才容許撤銷發佈的包( unpublish is only allowed with versions published in the last 24 hours)
- 即便你撤銷了發佈的包,發包的時候也不能再和被撤銷的包的名稱和版本重複了(即不能名稱相同,版本相同,由於這二者構成的惟一標識已經被「佔用」了)
若是你再也不維護你發佈的moduleA了,可使用下面這個命令
這個命令並不會撤銷已發佈的包,只是會在其餘人用的你的包時收到警告
npm deprecate moduleA
複製代碼
版本格式:主版號.次版號.修訂號,版號遞增規則以下:
主版號:當你作了不相容的API 修改,
次版號:當你作了向下相容的功能性新增,
修訂號:當你作了向下相容的問題修正,好比修復了一個bug。
改變當前package的版本號,update_type爲patch, minor, or major其中之一,分別表示修訂號,次版號,主版號
npm version <update_type>
好比當前版本號爲0.1.0
npm version patch
0.1.1
npm version minor
0.2.0
npm version major
1.0.0
複製代碼
Error: could not get uid/gid
[ 'nobody', 0 ]
at /usr/lib/node_modules/npm/node_modules/uid-number/uid-number.js:37:16
at ChildProcess.exithandler (child_process.js:205:5)
at emitTwo (events.js:106:13)
at ChildProcess.emit (events.js:191:7)
at maybeClose (internal/child_process.js:891:16)
at Socket.<anonymous> (internal/child_process.js:342:11)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
at Pipe._handle.close [as _onclose] (net.js:497:12)
在全局安裝前執行下面這條命令便可
npm config set unsafe-perm true
複製代碼
清一下緩存
npm cache clean --force
複製代碼
這個問題我本身的狀況是,在controllers/utils.js裏,調用nfs.download方法,writeStream尚未寫完,就開始了readStream而且清理了臨時路徑,致使文件被截斷了,因此必定要注意異步的問題,而且調試的時候儘可能寫好try catch和日誌,方便之後定位問題。
固然也能夠直接定義一個createDownloadStream方法,直接返回可讀流給body。
複製代碼
期間還踩過好多好多坑,遺憾的是忘記記錄下來了....
以上是在下關於npm私服搭建的一點拙見,若有不足,望諸位客官多多指正。